Independent Survey of BSS Examiners March 2021

Examiner Survey 2021

Background

In March 2021, the BSS commissioned an independent consultant to survey all examiners to understand more about what they like about the role and what they would like see change.  It was also important have opinions from Examiners with a good geographic spread and of all experiences from decades in the role to those with a handful of years and those who are very busy and those were less so.

The survey is an key tool to help the BSS design and steer Examiner development strategies. So it is also important that the survey consultant follows the Market Research Society’s Professional Code of Conduct, including that the anonymity of respondents is protected at all times and only known to the consultant. The BSS is only provided anonymised data.

The consultant was commissioned to provide a report and this follows. In the last section the consultant provides recommendations based on the survey results. The BSS has provided a comment on the recommendations from the consultant. These indicate either progress or future developments to meet the key findings and recommendations.

Headline findings

  • The response rate was over 70% of examiners contributed to the survey
  • Many examiners, just under half had over 16 years’ experience in 2021
  • Most responders considered that there had not been much progress since 2018
  • Almost nine out of ten examiners are proud to be part of the BSS and eight of ten believe in the role of identifying risks before they become an issue.
  • A third of examiners claim the BSS revenue is a significant source of income for them
  • Over half believe it is easy to find work in their area, albeit more work would be welcome
  • Just under two-thirds are keen on a more-frequent examination cycle.
  • Around the same number would welcome a move to fixed price examinations (especially true of less active examiners)
  • Only 56% believe the processes and procedures were fit for purpose
  • Eight in ten Examiners had a good sense of clarity with the Examination Checking Procedures (ECP) although enhancing the ECPs was cited as the ‘one change’ that would make the most significant improvement for Examiners
  • Around half of the examiners thought an app to replace paper would be helpful
  • Also around half are satisfied in the way the old examiner portal to the database works

A full copy of the report and the finding is available on this [Link]

Recommendations to the BSS from Morale Solutions

Support for examiners, Examination standards and Examiner Performance

> Actioning ECP improvements: sharpening and tackle any looseness, ambiguities or inexactness

BSS Comment:

    1. A programme involving BSS Technical Committee (BSSTC) members, BSS Advice Committee (BSSAC) members and the BSS Management Committee with the support of the BSS Office Team and the Scheme’s technical consultants and competent advisors was started in 2019 and culminated in the early days of 2023, to review the ECP support BSS Examiners consistent application of the Checks by BSS Examiners;
    2. Ensuring that BSS Examiner training can be effective to deliver an individual’s understanding of BSS Checking actions, Requirements and Applicabilities, whether the training is for existing Examiners or new-entrants.

The first stage of the review published in September 2021 introduced 540 adjustments. Around three-quarters the changes were editorial improvements or process amendments that will help everyone’s understanding and application of the BSS Checks, whether they are BSS Examiners, boat owners or members of the marine industry.

The second stage review is in its final agreement phase 49 changes of which 35 deal with improvement of accuracy, consistency and clarity. The remainder were updates for the reasons covered below.

The BSS Office has detailed every alteration in both iterations in a document called – ‘Every Change Explained Iterations 1 and 2 Jan 2023.’ Note: A very minimal third and final set of improvements are planned for January 2023. A controlled version of the ECPs will be distributed around the end of the financial year.

However, there were three additional and radical enhancements to the Appendices supporting the ECPs.

Firstly, existing Appendices were re-written and represented with new images or flow charts to aid understanding and clarity. They were reviewed for clarity, consistency, accuracy, and outdatedness.

The second development is that each ECP Part now has its own supporting Appendix and all but the shorter ones, have two sections:

    • Section 1 contains essential material helping the application of specific Checks
    • Section 2 contains additional information to support BSS Examiners’ understanding of the BSS Requirements.

The third augmentation is the provision specialist ‘one-subject’ Appendices have also been developed to support deeper detailed knowledge and consistent examination.

These enhanced tools should also be seen in the context of universal training with improved techniques and enhanced knowledge writing as well as improved support for individual examiners through new desktop and field-based quality assurance programmes. More on these below.

 

> Updating ECP

BSS Comment: Updates are process based and go through the committee led change process above. In the last two iterations (Sep 21 and Jan 23), there were three catalysts for change

    • Updates were introduced based on information, generally from the marine industry that a specific Checks may be in part or wholly in conflict with standards mentioned in Recreational Craft Regulations/Directive or supporting ISO standards;
    • amendments were introduced recognising post-Brexit changes introducing UKCA-marking alongside CE marking.
    • Consolidating technical appeals, updates and new product information into checks

 

> How is reviewing the ECP a living breathing process

BSS Comment: The process to review and change checks and procedures draws on a diverse range of potential drivers, including sources such as: -

    • BSS Technical Committee meetings
    • Reviews of technical queries raised with the BSS by examiners,
    • collated points arising during BSS Examiner Field and Desk Top Assessments
    • points that develop from the new entrant training programme design and delivery
    • comments made by Examiners arising during formal investigations application of BSS Checks and procedures

Proposals to change, amend, withdraw or introduce checks drawn up by BSS Office team, consultants, and committee members are taken forward for review by BSSTC on a technical risk-based process and passed through to BSSAC to accept or refine the proposals, which it does for each proposal against these terms of reference:

    • Does each proposed change make sense?
    • Does each proposed change support the aim of the project (which has been previously agreed)?
    • Can each proposed change be applied consistently in the field?
    • Are there any unforeseen consequences associated with each proposed change?

Examiners can influence this process individually by asking support queries and/or by raising suggestions or making comments and ideas through examiner group representatives.

 

> Is a mobile device app viable and practical for reporting BSS Examination processes

BSS Comment: Unless and until mobile field recording is demonstrably 99.99% reliable, paper-based recording will be available as an option.

However, our vision is that direct input through mobile devices will save time by reducing duplication and provide greater levels of field support. It will only be adopted if field trials prove it to achieve these predicted benefits.

We will be seeking best value and the process will be run through the Canal & River Trust procurement process and guided by the Trust’s IT team.

Reduced duplication of recording activity combined with improved on screen support in applying checks should lead to more consistency and reducing unproductive time consumption.

It is hoped that it will help reduce the amount of paper documents needing retention for 6 years by

It will help protection an examiner, having that recording of their examinations and findings

 

> What is the decision-making process, the involvement of examiners

BSS Comment: Examiners and committee members will be fully involved in the development of the apps from early stages to beta versions and roll out plans

 

Support from the BSS Office

> How can speed of response from the Office be improved?

BSS Comment: As a direct consequence of the Examiner Survey, the Scheme was able to justify the recruitment of additional staff  which was a Delivery Executive in November 2021 and a new role of Customer Service Officer (CSO) in December 2021.

It has resulted in an improved and updated case handling system which has been fully implemented and is now adopted by the BSS team. It is subject to an improvement programme as further potential enhancements are identified, and this is managed by the CSO.

We have compared examiner queries handled in September 2021 with September 2022 and the direct impact of taking on the new staff, we have halved the case handling times.

The case system has also allowed for better tracking and case knowledge sharing.

We have also instituted two critical regular working groups in the context of supporting Examiners, a technical team group and a customer service team group. These are where cases with more complex origins or more complex solutions are channelled for resolution.

The combination of the technology and new working practices have made a material difference to improving the time and quality of responses to queries.

 

> What is preventing a 7 day/week support function

BSS Comment: Providing human support for 7 days per week 50 weeks per year would have considerable impacts on direct and indirect costs.

However, we are exploring routes to enhancing self-service systems, potentially in the first half of 2023 and this will be followed by investigating the feasibility of artificial intelligence (AI) options that already being offered to the Scheme.

We understand neither of these may meet the most desired outcome, but they are likely to deliver an affordable and effective alternative meeting needs without costly impacts on examiners or their customers.

 

> What are the benefits for the BSS from building a better support structure for Examiners

BSS Comment: We have made a considerable investment in customer service because we believe there are tangle benefits for BSS Examiners and their customers as well as the Scheme and our customers.

It will improve the timeliness of Examiners providing Examination results to their customers and help facilitate good customer service to boat owners when they have need of technical information or need issues resolving.

The smooth and efficient engagement on Examiner queries, especially the most difficult subjects, is also likely to have a positive impact on the well-being of examiners as we understand, and also have first-hand experience of dealing with challenging and complex issues.

Resolving technical compliance related queries or administration related queries in a calm and efficient way can also mean that the customers of navigation authorities are able to licence or register their boats in a timely way, while the authorities’ duties of care are supported.

 

> BSS costing and delivering an ongoing training programme – how effective can it be

BSS Comment: The BSS training programme fundamental approach is a level playing field including that of knowledge, competency and quality of training.

Our aim is to reduce ambiguity and increase transparency in the examination process.

The Scheme has made a huge investment in upgrading the quality of training including in reimagining the content of the modules in the training programme in terms of both breadth and depth.

We have ensured the modules have a consistent structure in knowledge modules, a consistent approach to assessment techniques and consistent writing style.

There has been an investment in providing clearer diagrams and videos and more is to follow. The structure has for the first time in some cases, explained the background to the Checks and Requirements.

We learnt that Examiners need to understand the reasons for, and importance of, applying the checks accurately and diligently, as well as knowing precisely how those checks should be applied.

The feedback from new entrants and existing examiners to the quality of training has been very encouraging.

One examiner representative in a BSS Committee said it has reduced subjective profession judgement and increased transparency

While we appreciate the intensity of the core training programme has been quite challenging for Examiners, likewise the BSS Team and the Scheme’s consultants, we know that it would be deeply unfair for any examiner (and his customers) who did not have access to that same knowledge and understanding.

Navigation authorities, boat owners, the marine trade and insurers all welcome the drive to a more consistent approach and well supported examiners

 

> Cultivating and Examiner community and sharing good practice

BSS Comment: This is likely to be encouraged through new tools in the Examiner Community which are being developed in the first half of 2023 including a new community supported help centre that will replace the examiner pages on the website

Further measures are being introduced such as the new BSS Technical Newsletter that launched in December 2022 and the Community Appliance knowledge database which is seeing new data about previously unsupported appliances, models and manufacturers being loaded on a monthly basis.

Desk Top Assessments are also proving to be a good means of sharing good practice.

Helping examiners to understand more and identifying any support needs and clarifications at an early stage to provide support and training when needed, through the feedback, advice, tips and PiPs from the assessors.

 

> Addressing issues of malpractice and policing poor performance

BSS Comment: The BSS approach is predicated on reducing malpractice, poor performance or less than consistent application of BSS Examination and delivery of customer service through enhanced training, more support tools and improved monitoring.

The aim is that by the next financial year nearly every examiner will have had the opportunity to engage in either a desk top assessment or/and a new style field assessment and benefit from the one-to-one support it offers.

Also there has been investment in the BSS team and supporting consultants to launch and deliver process-based investigations where evidence or reports of potential deliberate poor performance arises. New measurement and analysis tools of our investigations are in development in January 2023 and these will be used to determine causes and report on improvement measures to our Committees and line management.

The Conditions of Registration are under review by a working group of the BSSAC and it is expected to report back in 2024. The group will be led by data and evidence.

 

Examiner Business Models

> How would the BSS approach a review of the 4-year lifespan of BSS certifications

BSS Comment: While it is recognised that the navigation authorities need a sustainable corps of BSS Examiners to provide adequate cover to support demand for BSS Certifications across their waterways, the lifespan of BSS certifications will be driven by risk and that will be assessed through holistic data provided including by all the Examination reports.

 

> Is a fixed priced model of BSS Examination feasible or desirable

BSS Comment: It is unlikely to be feasible because of the breadth of variation in the presentation of boats and their systems from the smallest to the largest, the simplest to most complex, the accessible to the inaccessible, one part of Britain to another, to have a set menu or scale of prices is unlikely to be fair or realistic.

When the Scheme was set up, it was purposely devised to not be involved in determining the price of BSS Examinations.

 

> What route to decision making is possible

BSS Comment: A working group under the BSSAC has been set up and following a scoping and ratification process, it has been agreed that it should run an evidence led programme considering these two wide-based topics.

A formal review of the existing BSS Examiner Conditions of Registration and the Service Level Agreement between the BSS Office and the BSS Examiner Bodies. There should be particular reference to the inclusion of terms of reference to be agreed, in writing, between Examiner and boat owner prior to the commencement of any BSS Examination.

The timing and frequency of BSS Examinations. This should be wide ranging and should include, but not be limited to the following:

    • The current four yearly BSS Certification regime
    • New boats, including part builds and ‘sailaways’
    • Modifications and technical changes
    • Change of ownership
    • Whether the current categories fully reflect the changing use of boats and should the use or purpose of a vessel determine the frequency of BSS Examinations

The working group expects to report back in 2024. There is a full breadth of stakeholders on the group including two examiner representatives on the working group and it is expected that examiners should be fully engaged in the process through these reps.

While we apologise for the time it has taken to publish the survey, but the passage of time has provided the opportunity to present evidence of a significant body of work and is demonstrable and is ongoing and we welcome any suggestions ideas and contributions.