- PDF Version (184K) - Print Friendly - Help

Summary of Submissions on the Proposal for NZFSA to Manage the Evaluation of Risk Based Management Plans

NZFSA Summary of Submissions No 03/08

March 2008

Online Print

ISBN number 978-0-478-31185-3 ISBN Number 978-0-478-31184-6
ISSN number 1177-7478 ISSN Number 1174-961X

Important Disclaimer

Every effort has been made to ensure the information in this report is accurate.

NZFSA does not accept any responsibility or liability whatsoever for any error of fact, omission, interpretation or opinion that may be present, however it may have occurred.

Further copies

Requests for further copies should be directed to:
New Zealand Food Safety Authority
P O Box 2835
WELLINGTON
Telephone : (04) 894 2500
Fax : (04) 894 2583

Website
A copy of this document can be found at www.nzfsa.govt.nz

Table of Contents

1 Introduction 4

2 Executive summary 5

2.1 Overall judgement 5

2.2 Assessment of the existing situation 5

2.3 The NZFSA Proposal 6

2.4 Alternative Options to the NZFSA Proposal 7

3 Background 8

4 The consultation process 10

5 Overall judgment of the proposal 11

6 Existing situation and issues with evaluation 13

6.1 Introduction 13

6.2 Numbers and availability evaluators 13

6.2.1 Is evaluation a largely one off function? 14

6.2.2 Are evaluators experiencing problems with collecting

payment? 15

6.2.3 Geographic variation in the availability of evaluators 15

6.3 Quality and consistency of evaluation and the performance of

evaluators 16

6.4 Other issues identified with evaluation 17

7 The NZFSA proposal 19

7.1 Introduction 19

7.2 Legislation 19

7.3 The role of NZFSA 20

7.4 Third party evaluators and the tender process 22

7.5 Existing agencies and individuals recognised to evaluate 24

7.5.1 Impact on business flows 24

7.5.2 Impact on relationships between evaluators and their

industry clients 25

7.6 Allocation of evaluation work 26

7.7 Competencies for evaluators 27

7.7.1 ISO 17020 28

7.8 Fees 29

7.9 Timeframe 30

8 Alternative options to the NZFSA proposal 31

8.1 Eliminate the role of the evaluator 31

8.2 Eliminate independent or third party evaluators 31

8.3 A combined system 31

8.4 A modified version of the NZFSA proposal 32

8.5 Improvements to the current framework for evaluation 32

8.5.1 Improvements relating to quality, consistency and performance monitoring 32

8.5.2 Evaluators and the regulator to work more closely 34

8.5.3 Build the capacity of the evaluator workforce 34

8.5.4 Other suggestions for improvements to the current

framework 35

9 Other matters raised 37

Appendix 1: List of Submitters 38

1 Introduction

This document summarises submissions made in response to the NZFSA Public Discussion Document No 11/07: Proposal for NZFSA to Manage the Evaluation of Risk Based Management Plans (the Discussion Paper).

This document is comprised of:

• an executive summary of the key issues raised by submitters; and

• a detailed outline of the content of submissions, organised into sections based on the predominant themes apparent in the submissions.

This document collates the opinions, concerns and queries raised by submitters. It does not seek to provide an NZFSA comment or response on these matters. NZFSA will give consideration to the matters raised by submitters and will, in due course, provide a statement on the future evaluation of risk based management plans.

2 Executive summary

The purpose of this document is to summarise and collate the comments made in the submissions on NZFSA Public Discussion Paper No 11/07: Proposal for NZFSA to manage the evaluation of risk based management plans. This document does not provide an NZFSA response to the submissions; this will follow in due course.

Twenty seven written submissions were received. Submitters came from a representative cross-section of those parties with an interest in evaluation and included: existing evaluators and auditors; groups representing the beekeeping, cold storage, goat, meat, poultry and seafood sectors; industry operators of a variety of sizes; and government bodies active in the health sector. Many submissions were detailed and engaged in robust discussion on the pros and cons of the proposal for NZFSA to manage evaluation.

2.1 Overall judgement

Twenty one submitters provided a clear overall judgement or conclusion about the proposal. Of these, one submitter fully supported the proposal. Nine other submitters indicated opposition to most, or all, aspects of the proposal. The remaining eleven submitters also disagreed with the proposal proceeding, but did concur either with some of the justification for the NZFSA proposal and/or some aspects of the proposed changes to the framework for evaluation.

2.2 Assessment of the existing situation

The Discussion Paper gave an assessment of the existing system for evaluation and identified associated problems. This assessment attracted comment from many submitters.

Opinion was mixed about whether NZFSA was correct in identifying a shortage of evaluators. This opinion often depended on the sectors in which submitters were operating. For example, those submitters who work under the framework of the Animal Products Act 1999 and deal with meat or seafood products were most likely to disagree that there was any problem with the supply of evaluators. Those working in other sectors, such as the domestic food or dairy sectors, were more likely to be open to the NZFSA assessment.

Fourteen submitters commented on the NZFSA assessment of the quality and performance of existing evaluation services. Most agreed that variation in the quality and performance of evaluation services can sometimes pose problems. However, it did not necessarily follow that these submitters thought that NZFSA needed to manage the evaluation process in order to resolve these issues. Instead, many submitters offered alternative suggestions for addressing quality and performance problems within the existing framework for evaluation.

2.3 The NZFSA Proposal

The roles envisaged for NZFSA under the proposal attracted considerable scrutiny. Most submitters who commented on these roles were comfortable with NZFSA taking the lead in setting the scope for evaluation, developing a list of suppliers of evaluation (although not necessarily via a tender process), setting competencies for evaluators and monitoring the delivery and quality of evaluation. These activities were seen as being consistent with the roles that NZFSA has traditionally assumed under the current system for evaluation.

However, approximately two thirds of submitters disagreed with NZFSA taking on the additional roles envisaged by the proposal, such as allocating evaluation work, collecting payment from industry and reimbursing evaluators. Some submitters expressed concerns about the transparency and fairness of the processes by which NZFSA would select successful tenderers and allocate evaluation assignments. Others thought the proposed new roles were outside the scope of the activities that should be undertaken by a government agency. Some doubted whether NZFSA would solve the problems with the supply of evaluators by becoming more involved in evaluation. It was also suggested that the expansion in NZFSA’s roles would necessitate NZFSA devoting greater resources to evaluation and having to then recover higher costs from industry.

A few submitters also commented on the role which the NZFSA Verification Agency (VA) would have under the proposal. Submitters noted the importance of effectively managing any potential for conflict of interest to arise, if and when NZFSA was to allocate work to the VA rather than to independent evaluators. These submitters thought it was valuable to have the option of using third party evaluators and recommended that the VA be used only as an evaluator of last resort.

Another aspect of the proposal that drew significant comment related to its potential impact on existing evaluators. Concern was expressed that some evaluators could lose business or be forced out of the market, thereby worsening the shortage of evaluators. Others noted that that all evaluators would face additional paperwork as a result of the change to a tender process. Over two thirds of submitters highlighted the possible impact of the proposal on established relationships between evaluators and their industry clients. Some submitters would welcome industry operators having access to a more streamlined process for finding an evaluator. However many others disagreed with industry operators no longer being able to choose evaluators and to negotiate directly with them. This was especially in situations where industry operators have developed long term relationships with evaluators who are familiar with their processes and premises.

A majority of submitters supported the need for evaluators to adhere to a minimum set of competencies and for evaluators to be subject to effective performance expectations and scrutiny. However, of these submitters, thirteen argued that these competencies and performance expectations could be set and managed through the existing framework for evaluation, and did not require wholesale changes.

Submitters requested further details on how some aspects of the proposal would operate. This included confirmation of the criteria that would be used to choose successful tenderers, the factors which NZFSA would take into account when allocating work to evaluators, and how the variation in the cost structures of different evaluators would be balanced when setting an hourly fee for evaluation.

Many submitters were supportive of the objectives underlying the NZFSA proposal, such as NZFSA wanting to ensure that evaluation is undertaken at a consistent level and that the obligations and compliance costs faced by industry are minimised. A number of submitters also identified benefits that would flow from specific aspects of the proposal, such as NZFSA developing a closer relationship with evaluators and evaluators being freed from debt collection issues.

2.4 Alternative Options to the NZFSA Proposal

Submitters were invited to identify any other feasible options for evaluation.

Five submitters suggested that, if the proposal for NZFSA to manage evaluation was to proceed, it should operate in conjunction with the current system of appointing evaluators. This would mean that industry operators would have a choice between either independently selecting and working with evaluators, or asking NZFSA to allocate an evaluator and manage the process.

Another option favoured by more than one submitter was for the adoption of a modified version of the NZFSA proposal. This would involve NZFSA providing operators with a shortlist of recommended evaluators and operators then selecting an evaluator from this list and negotiating with them direct.

The great majority of submitters preferred that the current framework for evaluation should be retained, but with improvements where necessary. Suggested improvements included adopting more stringent quality control mechanisms for evaluators. Ideas submitted included more frequent audits of evaluators by the Compliance and Investigation Group, the use of a formal complaints and appeals process and, if necessary, enhancing the legal powers which NZFSA has to sanction underperforming evaluators. Some submitters thought that NZFSA and evaluators should liaise more closely, for example to identify and fill gaps in the guidance material available to evaluators. Other submitters also considered that more could be done to build the capacity of the evaluator workforce, for example by expanding training for both existing evaluators and new evaluators, and by rethinking what is involved in evaluation with a view to conserving evaluator resources.

The above issues, and others, are discussed further in this document.

3 Background

The NZFSA Public Discussion Paper No 11/07: Proposal for NZFSA to manage the evaluation of risk based management plans was released in October 2007. It was made available on the internet (http://www.nzfsa.govt.nz/consultation/proposal-for-evaluation/index.htm) and in hard copy.

The Discussion Paper:

• summarised the purpose of evaluation and the types of risk based management plans¹ that require evaluation;

• identified problems with the existing system for the evaluation of risk based management plans. These problems included a shortage of evaluators, difficulties faced by some industry operators in locating evaluators, concerns about the quality and consistency of the work undertaken by some evaluators and difficulties experienced when addressing problems with non-performing evaluators; and

• outlined a proposal for NZFSA to manage the evaluation of risk based management plans. The proposal envisaged that, in future, when an operator has a plan that requires evaluation, this plan would be submitted to NZFSA (possibly through a Territorial Authority) for evaluation. NZFSA would then evaluate this plan either using external third party evaluators or, if necessary, staff in the NZFSA Verification Agency (VA). NZFSA would invite third parties to register their interest in being included on a preferred, or panel, list of suppliers of evaluation services and, if appropriate, to subsequently tender for inclusion on this list. NZFSA would set the competency criteria and performance expectations for evaluators, negotiate the terms of engagement with evaluators and allocate work to evaluators. NZFSA would also monitor the outcome of evaluation and take responsibility for recovering the costs of evaluation from industry operators and for reimbursing evaluators.

The closing date for submissions on the Discussion Paper was 17 December 2007.

Twenty-seven written submissions were received and these comprised:

• 10 submissions from currently recognised evaluators or approved auditors of risk based management plans;

• 7 submissions from sector or industry organisations;

• 5 submissions from industry operators who have registered various types of risk based management plans;

• 3 submissions from government bodies (including District Health Boards (DHBs)); and

• 2 submissions from members of the public who did not state whether they were employed or engaged by an industry operator, evaluator or government body.

Further details about submitters are provided in Appendix 1.

The large volume of submissions was encouraging and many submissions were detailed and comprehensive. NZFSA is appreciative of the time and resources which submitters have allocated to the preparation of their comments.

The NZFSA proposal has generated a significant level of robust debate and has resulted in the emergence of some useful comments and suggestions about both the current state of evaluation, and the way forward.

4 The consultation process

Eleven submitters commented on the consultation process for the Discussion Paper.

Seven submitters expressed appreciation of the opportunity to comment on the proposal, with several requesting ongoing involvement in either the future development of the proposal and/or alternative improvements recommended for evaluation. A further three submitters indicated general satisfaction or acceptance of the consultation process.

However, a number of comments were made about the need for greater clarity or detail to be supplied on some aspects of the proposal. Several submitters also considered that the problems with the current evaluation system identified by the paper were based on “questionable research” or “more perception than fact”.

One evaluator commented that it would have been preferable for the problems with evaluation to have been raised with evaluators prior to the release of the discussion paper. Another thought that all industry operators with risk based management plans should have received a hard copy of the Discussion Paper, rather than being alerted via email to the location of the document on the NZFSA website.

Two submitters appeared to be unaware that the proposal outlined in the Discussion Paper took an overarching approach and covered all types of risk based management plans administered by NZFSA, that is Risk Management Plans (RMPs) under the Animal Products Act 1999, Food Safety Plans under the Food Act 1981 (and later Food Control Plans under the Food Bill) and Wine Standards Management Plans under the Wine Act 2003. This wide coverage had been signalled in the Introduction section (page 3) of the Discussion Paper. This misunderstanding may have influenced the nature of some comments offered by these submitters.

5 Overall judgment of the proposal

Twenty one submitters offered a clear overall judgment about the proposal and/or an opinion about whether or not the proposal should be implemented.

One submitter expressed full support for the proposal.

Nine submitters expressed firm opposition to most, or all, aspects of the proposal, citing reasons including that:

• there is insufficient evidence of problems with the current system of evaluation to warrant making the changes proposed;

• the proposal would, in their view, have a negative impact on the numbers of existing evaluators, also on the financial viability of some evaluators;

• the proposed changes would add to the costs of evaluation which need to be met by industry operators;

• industry operators would lose control over which evaluators they could use and could be prevented from building long-term relationships with evaluators of their choice;

• the management of evaluation, and the allocation of evaluators to industry, is outside of the scope of the ‘public good’ activities in which NZFSA should be engaged;

• the proposal creates a risk that conflicts of interest could arise when NZFSA selects which evaluator/s to allocate to industry operators, and that this process would not be sufficiently transparent; and

• the proposal does not take into account the variations between the different sectors and processes which require evaluation.

A further eleven submitters expressed opposition to the proposal proceeding, but did support either some of the rationale underlying the NZFSA proposal and/or some aspects of the changes to the evaluation framework proposed in the Discussion Paper. Many of these submitters recommended that it would be both possible and more desirable to make improvements to the current framework for evaluation, without NZFSA needing to manage evaluation in its entirety. Some examples of the comments made by such submitters include:

“[our organisation] supports the proposal’s aims to improve industry access to evaluation services and to ensure greater consistency in the quality of evaluations. It does, however, have reservations as to whether the proposed system will provide the desired outcomes. [It] believes that the problems with the existing system are due not to market failure but to lack of workforce capacity and that taking measures to build capacity rather than interfering with the market would be more efficacious.”; and

“[We] agree with the elements of the proposal that NZFSA have the role of setting the competency criteria for evaluators, approving evaluators and monitoring the delivery and quality of the evaluators and the outcome of evaluation…However, we do not agree with the elements of the proposal that NZFSA manage the evaluation process and in particular we do not agree that NZFSA alone negotiate the terms of engagement with evaluators, allocate work to evaluators and take responsibility for payment of evaluators. These are commercial matters better handled by RMP [Risk Management Programme] operators….”

6 Existing situation and issues with evaluation

6.1 Introduction

Section 4 of the Discussion Paper identified a number of problems with the existing system for evaluating risk based management plans. These included :

• that only a low number of persons are recognised or approved to undertake evaluation. This limit on the number of evaluators was attributed to a range of factors, including New Zealand demographics; the trend towards using tools such as codes of practices which dispense with the need for evaluation; the largely one-off nature of evaluation; and the issues that some evaluators face with obtaining payment for their services;

• that some industry operators are experiencing difficulties locating evaluators;

• concerns about the quality and consistency of some evaluations; and

• some difficulties associated with managing non-performing evaluators within the current framework for evaluation.

6.2 Numbers and availability evaluators

Opinion was mixed amongst submitters about the validity of NZFSA’s assessment that the low number of persons recognised or approved to evaluate is causing difficulties. Often the views expressed by submitters were influenced by the sector in which they were engaged. For example, most submitters engaged in the seafood and meat sectors had not experienced issues with locating evaluators, but others (although not all) operating in the dairy or domestic food sectors were more likely to have come across instances of evaluators being difficult to source.

Eleven submitters agreed with the NZFSA assessment that there is a small number of evaluators and that this can impact on industry operators. Several of these submitters offered additional reasons to those given in the Discussion Paper for the shortage of evaluators, including:

“the stressful nature of the job and the lack of incentives to join the profession…the industry is still in a position where demands outstrips supply. Given that there are only 36 approved evaluators and a forecasted 1000 evaluations to be carried out in the next five years, this situation looks set to continue”; and (conversely)

“there is not enough demand [for evaluation]”; and

“NZFSA has helped create such a situation. The following points are relevant…the need for evaluators has not been advertised…VA ...see evaluation as their exclusive preserve…it appeared the list of evaluators was closed to new evaluators…workloads for evaluators have at times been horrendous…the process to get approved is far too complicated…the requirements placed on evaluators have never been that clear”; and

“recent changes to require ISO certification around evaluation practice has acted as a disincentive for sole-operator evaluators, as significant extra costs will be incurred.”

Eight submitters had not experienced any issues with a shortage of evaluators in their industry sector. These submitters were working with animal products (although two other submitters from the dairy industry and from the meat sector did record problems with sourcing evaluators). One submitter commented that, if other sectors were experiencing problems with sourcing evaluators, then “this should be addressed at their level for their industry”. Two recognised evaluators did not consider there were insufficient numbers of evaluators because they had never turned away clients, had not been contacted “out of the blue” by operators looking for an evaluator, and had increased or up skilled their evaluation capabilities.

Two submitters working as evaluators commented that the issue of evaluator availability may be clouded because some evaluators are more popular with industry operators than others and may have to turn work away, which can create the impression that all evaluators are in short supply. This was expressed as follows:

“some Evaluators are busy, whilst others are not fully engaged. I would suggest that the industry is exercising their ability to choose evaluators on performance and price, which seems to me a good thing.”

One submitter queried the number of recognised evaluators cited in Section 4 of the Discussion Paper. A search of the NZFSA website by the submitter had identified only 23 evaluators, rather than the 36 evaluators cited. However, to clarify, the number of recognised evaluators comprises those listed on both the animal products and the dairy products sections of the NZFSA website, and at the time the Discussion Paper was released this added to 36 evaluators.

6.2.1 Is evaluation a largely one off function?

Three submitters offered comments to support the NZFSA assessment that the largely one-off, non-repeatable nature of evaluation can limit both the numbers of persons interested in taking on evaluation functions and the value of the business which can be derived from evaluation. As one of these submitters expressed it:

“Evaluations in this sector [meat and seafood] have slowed markedly as all existing operations are evaluated, and only amendments and new businesses are now requiring evaluation…”

However, eleven other submitters disagreed that evaluators are likely to only work with industry operators on a one-off basis. For example:

“we have ongoing business relationships with clients that have endured since the Animal Products Act was introduced…”.

Comments about the value of such ongoing relationships between evaluators and industry operators, and the impact which the NZFSA proposal could have on these relationships, are summarised further in Section 7.5.2. of this document.

6.2.2 Are evaluators experiencing problems with collecting payment?

Three submitters, all of them evaluators, commented on whether debt collection issues are a problem for evaluators.

Two of these submitters had not personally experienced difficulties with non-payment for evaluation services. However, the third submitter had encountered problems and had several bad debts either in the course of being recovered or written off.

6.2.3 Geographic variation in the availability of evaluators

Five submitters disputed the assessment in the Discussion Paper that small to medium sized enterprises outside of the main centres may be particularly disadvantaged when trying to locate evaluators. These submitters all considered that evaluators regularly and willingly travelled throughout the country as required. It was also noted that site visits are not always necessary to complete evaluation of all plans.

However, one submitter had encountered problems with the geographical spread of evaluators, expressed as follows:

“We currently use XX [name of evaluator] to undertake RMP evaluations in both the North and South Islands and are satisfied with their performance. However they are based in the North Island [and] this does create logistical issues especially for South Island sites.”

6.3 Quality and consistency of evaluation and the performance of evaluators

Fourteen submitters commented on the assessment in the Discussion Paper that variation in the quality and consistency of evaluation can be a troubling issue, as can occasional poor performance by some evaluators.

Ten of these submitters shared some or all of the concerns expressed by NZFSA about the potential for issues to arise with evaluation quality, consistency and performance.

One submitter provided examples of the type of evaluator non-performance which can arise and the unnecessary costs which can flow as a result to operators:

“”Over-evaluation”, where evaluators place excessively and/or prescriptive requirements on operators, may not be an issue for NZFSA, but can place a high cost on the actual evaluation process as well as on-going costs for the operator to maintain unnecessarily high standards.

“Under-evaluation”, where operators may be able to get away with not fully meeting the requirements for their RMP, is of significant concern to NZFSA but is also costly for operators as subsequent verification finds them in non-compliance.”

Several submitters identified various factors which contribute to the problems with quality, consistency and performance. These included:

• inadequacies in the material and/or plans presented by industry operators for evaluation. This was linked to the increasing use by industry operators of tools such as templates when developing their plans because, in the view of the submitter, these mean that “industry no longer needs to understand its business from a food safety point of view”;

• generic evaluators (i.e. evaluators who do not specialise in particular sectors or industries) undertaking evaluation in sectors in which they have little or no experience, and/or without instruction from NZFSA to engage technical experts;

• that the maintenance of high standards by evaluators requires the commitment of considerable resources and funds. As one submitter expressed it:

“…some considerable resource can be allocated to ensure quality and consistency are maintained, for example, on-going training, peer review, management review, internal audit. There is a cost associated with this that that is easier to build into high-volume programmes.”’;

• insufficient initial training and on-going training being made available to evaluators by NZFSA;

• some evaluators charging at significantly higher rates than other evaluators; and

• occasional instances of evaluation taking longer and costing more than is justifiable, or anticipated by industry clients.

One submitter pointed out that the issue of evaluator non-performance could be self-addressing. This is because, if substandard evaluation results in the subsequent verification of a plan taking longer than anticipated, then the delays and higher costs incurred by the industry operator as a result will act as a disincentive to re-employing the non-performing evaluator.

Whilst many submitters agreed that some problems do exist with evaluation quality, consistency and non-performance, it did not necessarily follow that they supported the proposal for NZFSA to manage the evaluation process. Instead, many submitters offered alternative suggestions for dealing with these problems within the existing framework for evaluation, and these suggestions are outlined in Section 8.5.1. of this document.

Not all submitters concurred with the NZFSA view that quality or performance issues exist. One submitter was confident that there had been no problems with the quality and reliability of evaluation in their particular industry sector (the seafood sector). Another submitter commented that overseas markets had not identified any problems with New Zealand evaluators. Two other submitters were comfortable that the evaluators they employed were competent, although they did not have sufficient information to comment on the adequacy of other evaluators.

Two submitters were also concerned that the Discussion Paper, by publicising issues relating to the quality of evaluation, could have the effect of casting all evaluators in “a bad light”.

6.4 Other issues identified with evaluation

Submitters were invited to provide information about any other issues with evaluation not identified in the Discussion Paper. Issues which were highlighted as a result included:

• two submitters commented about problems relating to continual, ongoing changes to the evaluation process and/or framework (of which the proposal in the Discussion Paper would be another example if implemented), all of which impact on the attractiveness and viability of the provision of evaluation services as a business option:

“Over this time [since the introduction of the Animal Products Act 1999] we accredited evaluators have seen a number of changes to the system each time reducing the amount of available work from that initially anticipated while potentially increasing the requirements on accredited evaluators”;

• one submitter said that the rules and expectations regarding the content of risk based management plans and what should be evaluated sometimes change without adequate explanatory information being supplied by NZFSA to evaluators or industry operators; and

• three submitters identified a significant problem around the varying interpretations attributed to relevant standards and specifications by operators, evaluators and verifiers. These submitters recommended further efforts should go into promoting a consistent understanding of how standards and specifications should be interpreted.

7 The NZFSA proposal

7.1 Introduction

The Discussion Paper sought feedback on the objectives and content of the proposal for NZFSA to manage the evaluation of risk based management plans. To enable the proposal to receive serious consideration, the likely process of implementation was outlined, although it was emphasised that this process could be revised, dependent on the feedback received. The implementation details were presented under the following headings in the Discussion Paper:

• legislation;

• the role of NZFSA;

• third party evaluators;

• existing agencies and individuals recognised to evaluate;

• allocation of evaluation work;

• competencies;

• fees; and

• timeframe for introduction.

Submitters were also invited to comment on the assessment given in the Discussion Paper of the likely positive and negative impacts of the proposal. Many submitters merged their assessment of the impacts of the proposal with their general commentary on the implementation details, and the same approach has therefore been taken in this summary.

7.2 Legislation

Seven submitters commented on the legislative process that would be necessary to give effect to the proposed changes outlined in the Discussion Paper.

Four submitters acknowledged that, if the proposal is to proceed, then the legislative process would need to be as set out in the Discussion Paper, although one submitter added that the difficulties involved, and the time required, to give effect to these legislative changes may have been underestimated.

Two submitters did not think there was any need to amend either the Animal Products Act or the Wine Act to give effect to the changes.

One submitter commented that any new system, which could link the provisions covering evaluation in the various laws that NZFSA administers, would be beneficial and would simplify the system for all concerned.

7.3 The role of NZFSA

Most submitters commented on one or more aspects of the roles which NZFSA would assume under the proposal. These roles would include the Approvals Group managing the relationships with evaluators, the Standards Group setting the standards and competencies for evaluation and the VA undertaking evaluation in certain prescribed circumstances.

Many of the submitters who commented on these roles welcomed or accepted the responsibility which NZFSA would take for setting the parameters for the scope of evaluation, developing a list of suppliers of evaluation (although submitters did not necessarily agree that this list should be created through a tender process), and monitoring the delivery and quality of evaluation. Submitters generally considered these roles to be a continuation of the functions which NZFSA undertakes within the current framework for evaluation.

Some submitters also expressed support for NZFSA’s stated objectives in proposing changes to its involvement in evaluation, such as reducing compliance costs and obligations for operators and enhancing the overall evaluation framework:

“[Our organisation] supports NZFSA in any actions to reduce the obligations and compliance costs faced by industry when developing and implementing RMPs.”

However, the proposal for NZFSA to allocate evaluation work, collect payment and reimburse evaluators attracted criticism. Approximately two thirds of submitters raised objections to, or queries about, NZFSA assuming these roles for one or more of the following reasons:

• the costs involved in setting up a new or extended infrastructure within NZFSA to handle the proposed new roles and the additional charges that could be passed on to industry as a result. This was a particularly widespread concern; as one submitter put it:

“what indirect costs will be incurred by industry for the added level of bureaucracy incurred by NZFSA’s administration of this scheme?”;

• concerns about whether the process by which NZFSA would allocate work would be transparent and free from favouritism and “cronyism”. Some also considered that the proposal would be flawed because it would establish NZFSA as a monopoly provider:

“this [the proposal that evaluation be managed by NZFSA] establishes NZFSA as a monopoly provider. This would require any company requiring evaluation services, to purchase this service from a government department that is sheltered from the disciplines that competition brings.”

• these roles are outside the scope of the activities which NZFSA, as a government agency, should undertake. As one submitter put it:

“these are commercial matters better handled by RMP operators [and] are outside of NZFSA core accountability”.

• doubts about whether the new roles for NZFSA would resolve the problems identified with the current system of evaluation. For example, some suggested that by requiring evaluators to go through a tender process, NZFSA would cause some evaluators to exit the market for evaluation services;

• concerns that the proposed changes represent a step away from the regulatory model and the principle of encouraging contestability in service provision where possible:

“This appears to be stepping away from the optimal regulatory model and is not necessary to ensure that evaluation is competent and consistent.”

• some submitters considered that collecting payment for evaluation, whilst it can be a problem, is not a matter in which NZFSA should be involved; and

• a few submitters raised complaints about aspects of the past performance of NZFSA and doubted, as a result, whether NZFSA could effectively handle new roles.

Four submitters commented specifically about the future intended role of the NZFSA Verification Agency (VA) in evaluating plans². These submitters thought that when NZFSA allocated evaluation assignments to the VA, there could be potential for a funder/provider conflict of interest to arise. They considered it would be essential that the allocation process was transparent. Some also queried whether the VA might be called on to evaluate plans which it also verifies, with another resulting conflict of interest. Some submitters recommended that the VA should only be used as an evaluator of last resort:

“[Our organisation] does recognise that there may be occasions when it is necessary for NZFSA to become involved in the supply of evaluators… It is the view of the [organisation] that…NZFSA should only be involved with the management or provider of evaluation as last resort…If it is necessary to use the resources of VA, [our organisation] request that there is a transparent process to ensure evaluation activities are clearly separated from other costs recovered activities of VA”.

Notwithstanding these comments, four submitters did consider that benefits would flow if NZFSA was to develop a closer relationship with evaluators as a result of the proposal. They saw advantages in NZFSA defining the scope of evaluation work in more detail. They also supported NZFSA becoming more involved in developing skills and building the capacity of the evaluation workforce, especially if this capacity could simultaneously build up the resources available to verify plans:

“If …the scope of evaluation work will be better defined, possibly with standard industry checklists and guidelines, then this is seen as an advantage over current systems…Any involvement of NZFSA with evaluators to develop skills is seen as an advantage.”

However, some of these submitters added that NZFSA could as readily develop a closer relationship with evaluators within the current framework for evaluation, without making the changes proposed.

Three submitters also welcomed NZFSA taking responsibility for collecting payment for evaluations and reimbursing evaluators. One of these submitters said this would be especially beneficial when dealing with smaller, newer customers. Another submitter did stress, however, that evaluators would need rapid reimbursement by NZFSA as they do not necessarily have the resources to wait for several months to be paid.

7.4 Third party evaluators and the tender process

The Discussion Paper proposed that NZFSA would invite third party evaluators³ to register interest in being included on a preferred, or panel, list of suppliers of evaluation services. After registering interest, those parties with appropriate skills/experience and who meet the criteria and competencies set by evaluators would be invited to tender for inclusion on this list. The tender process would be conducted in line with mandatory government purchasing rules.

Eleven submitters commented on the tender process and the terms of engagement proposed for evaluators.

Three submitters expressed support for the tender process. As one submitter put it:

“It is understood that a tender process will need to be worked through by prospective evaluators. However, we do not see this as a negative process. We do understand that there will be a degree of uncertainty whilst this process is being carried out. However, this should be able to establish a list of preferred evaluators which are qualified, reliable and professional…an easy to obtain list, identifying competent evaluators in the various geographic and specialist areas.”

Another submitter based their support for the tender process on the benefits that could be derived from using third party evaluators, considering that value can be added by having access to diverse evaluators and to a different perspective from that offered by NZFSA (VA) staff.

However, seven other submitters expressed specific opposition to the tender process. Their reasons for doing so included concern that the tender process would be time consuming and increase the overall cost of evaluation, and doubts about whether the volume of evaluation work available would justify the investment of resources which evaluators would need to devote to the tendering process. In some cases they concluded that, as a result, the tender process could worsen any existing shortage of evaluators.

Submitters also had queries about how the tender process would work. These included:

• how controls would be put in place to ensure transparency in the development of the list of preferred suppliers;

• if NZFSA would negotiate with evaluators separately or collectively, and whether the same terms would be offered to all;

• whether any currently recognised evaluators subsequently excluded from the list of evaluators would have the right to appeal;

• what criteria, apart from price, would be used to decide the outcome of the tender process;

• how NZFSA would treat the significant variation in evaluation charges that could result from the different cost structures of, for example, large evaluation agencies versus single practitioner evaluators;

• whether evaluation agencies/persons would need to train people and obtain ISO 17020 recognition prior to tendering, and whether it would be worth their while doing so without a reasonable guarantee of the evaluation work that would be subsequently be made available; and

• the roles that Public Health Units, District Health Boards and Health Protection Officers would have in the proposed new framework.

7.5 Existing agencies and individuals recognised to evaluate

The impact which the proposal could have on the existing agencies and individuals recognised to evaluate attracted much attention from submitters, with more than two thirds of submitters providing related comments.

7.5.1 Impact on business flows

Fourteen submitters were concerned that the proposal could have a negative effect on the business flows of existing evaluators and their ability to, and/or interest in, staying in the evaluation market. Their reasons for concern were linked to opinion that:

• some evaluators could benefit from the proposal at the expense of others. One submitter thought:

“Those [evaluators] that will be most disadvantaged have developed strong business relationships and a correspondingly good record of evaluations, whereas those with lower evaluation activity, and possibly lower quality outputs, are likely to see an increase in business.”

Others thought that NZFSA and large third parties, such as AsureQuality New Zealand, would be able to remain in the market whilst other evaluators would be squeezed out or would experience a reduction in work volume;

• evaluators would experience uncertainty about the volume of work they would be allocated by NZFSA;

• evaluators would be required to devote more resources to administration, for example to the preparation of tenders;

• the payment that NZFSA would negotiate for evaluators could be at significant variance from the current charging policies of some existing evaluators; and

• the competencies which evaluators would be required to demonstrate, especially a commitment to becoming accredited to ISO 17020, could deter some current or prospective evaluators from remaining in, or entering the market.

One submitter did, however, comment that evaluators may be able to utilise their inclusion on the list of preferred suppliers as a tool when marketing other services to industry operators.

7.5.2 Impact on relationships between evaluators and their industry clients

Another issue that drew significant comment from more than two thirds of submitters related to the impact which the proposal would have on the existing relationships between evaluators and their industry clients.

Five submitters considered that at least some industry operators would benefit from a more streamlined process for locating an evaluator. One submitter thought there was particular justification for NZFSA to control the evaluation of “low level” domestic food control plans, also the evaluation of plans relating to industry sectors which had experienced significant variation in the prices charged for evaluation.

However, nineteen submitters considered that the proposed system of NZFSA allocating evaluators could have disadvantages in terms of the relationship between evaluators and their clients, including that:

• operators would not be able to choose for themselves who evaluates their plans. They would also be unable to negotiate directly with evaluators on factors such as price, timeliness and performance. One submitter described this as follows:

“The current model allows the operator to choose their evaluator. This gives them some feeling of control over the process. This may be valuable in terms of allowing them to better accept requirements placed upon them….Anything that can lead to improved buy-in from them and more openness must be good for the process.”

• Under the current system, many industry operators have developed long-term relationships with evaluators who they use repeatedly when they have multiple plans, or amendments to plans requiring evaluation. Both evaluator and industry operators submitted that these relationships can be based on evaluators’ knowledge of the specifics of operators’ systems, production processes and premises, and that this knowledge could offer benefits in terms of the costs and timeliness of the evaluation service. As one submitter expressed it:

“…the proposal does not allow for companies with multiple RMPs to choose one evaluator and develop a relationship so that the evaluator is familiar with company protocols, history and business principles across the companies total business. Similarly when significant amendments are made to existing RMPs… additional time and costs will be incurred by the company as the new evaluator will need to become familiar…”

One submitter queried the processes that would be used to manage the transfer, or sharing, of information between existing evaluators and any new evaluators appointed by NZFSA.

7.6 Allocation of evaluation work

Some of the comments offered by submitters about the proposed process for allocating evaluation work have already been summarised in Sections 7.3-7.5 of this document. However, other relevant observations are outlined below.

One submitter saw an advantage in NZFSA allocating work because this would alter the current practice of “generic evaluators” working in highly specialised industries:

“The NZFSA proposal will facilitate the matching of technical industry knowledge of evaluators with evaluation tasks at hand.”

Three submitters were uneasy about the geographic location of industry operators being taken into account in any “decision tree” used by NZFSA to allocate evaluation work. One of these submitters, who has an existing evaluation business, indicated that they would not want to evaluate industry operators located in close proximity to their business, instead preferring to offer consultancy services to these local companies. Another of these submitters explained that the ease of transport and the ability to combine evaluations in similar locations would overcome any obstacles related to evaluators being geographically distant from the operations that they evaluate. However, one other operator did consider it would be positive if NZFSA required evaluators to travel to service sectors/locations experiencing problems in sourcing evaluators.

Other submitters identified factors they considered NZFSA should take into account when determining the allocation of evaluation work. These included:

• that evaluators would require a “reasonable guarantee of evaluation work” in order to continue to be interested in offering evaluation services;

• that it is important that a variety of evaluators remain available to service industry operators;

• that industry should be consulted in the future development of the decision tree for allocating evaluation;

• recommendations that industry should still be given a role in the choice of evaluators. One submitter suggested making provision for operators to request evaluators, another suggested that NZFSA should consult with operators each time they allocated an evaluator, and a third submitter suggested that industry should have the ability to refuse to work with an evaluator allocated to them by NZFSA and to request an alternative when necessary. Another suggestion involved NZFSA preparing a short list of evaluators, and then providing for evaluators to select an evaluator from this list;

• that large operators with multiple plans may prefer to work with only one or two evaluators, in order to simplify the evaluation process and standardise the format and structure of their plans and systems;

• the need to protect the commercial sensitivity of some operator equipment and processes when assigning evaluation work;

• support for the complaints and appeal process signalled in the Discussion Paper; and

• the need for clear communications to industry about the time that needs to be allowed for NZFSA to allocate an evaluator and the importance of the NZFSA allocation process being timely, responsive and enabling industry to still make rapid changes to plans and/or to introduce new plans as required.

7.7 Competencies for evaluators

The Discussion Paper proposed that NZFSA would set competency criteria and performance expectations for evaluators on the list of suppliers. These competencies would be similar to those which recognised evaluators have to achieve under the existing framework. They would include accreditation (or evidence than an evaluator intends to work towards achieving accreditation) to ISO 17020 or to an equivalent NZFSA standard.

Approximately two-thirds of submitters contributed comments about the competencies and performance expectations for evaluators which would apply under the proposal.

The great majority of these submitters supported the need for a minimum set of competencies and for evaluators to be subject to clear and rigorous performance expectations and scrutiny. However, thirteen submitters considered that NZFSA should already be able to set competencies and deal with poorly performing evaluators through its powers under the current evaluation framework. References were made to using, within the current framework of accreditation to ISO 17020, the NZFSA Compliance and Investigation Group (CIG), and to NZFSA workshops for evaluators and other forms of liaison between NZFSA and evaluators. If necessary, submitters suggested strengthening existing scrutiny measures. For example, one submitter wrote:

“Under the current programme, CIG audits the evaluators. This should result in consistent standards and filter out poor performing evaluators. While the process could be improved we see no need for such an extensive overhaul of the system.”

Further details of the improvements which were suggested to the current framework for evaluation are provided in Section 8.5.1. below.

Another submitter doubted whether, if problems with quality and performance could not be resolved under the current framework, they would be able to be managed any more effectively under the proposed changed system for evaluation.

Four submitters asked for further detail to be released about the competencies that NZFSA would set for evaluators and the mechanisms which would be used to deal with incompetent evaluators. A further five submitters emphasised the importance of ensuring that any process for removing poorly performing third party evaluators from the list of preferred suppliers would be transparent and fair. Some support was voiced for the development of the complaints and appeals process signalled in the Discussion Paper.

Other comments made relating to competencies and performance included that:

• standard industry checklists and guidelines for evaluators should be developed; and

• the key to improving performance may lie in providing enhanced training to evaluators.
One submitter pointed out that suitability to evaluate is not necessarily limited only to those persons who already meet the set criteria. Instead it could be possible that persons with only some of the relevant interpersonal, analytical and technical skills could be readily trained to be effective evaluators.

7.7.1 ISO 17020

Eleven submitters made observations about the requirement, under the proposal, for evaluators to either be accredited to ISO 17020 or a similar standard, or to be working towards achieving this accreditation.

Eight submitters opposed the use of ISO 17020 for reasons including:

• that accreditation to the standard is, in the opinion of some submitters, expensive and difficult to achieve. It could act as a barrier for entry to the supply of evaluation services, could result in some evaluators not yet accredited to exit the market, and could increase the costs to industry operators;

• doubts that evaluators who have achieved accreditation to ISO 17020 are any more competent than those already working as evaluators without accreditation; and

• a concern that accreditation to ISO 17020 can discriminate against smaller, independent evaluation businesses, as opposed to evaluators working for larger organisations that hold the accreditation at an agency level. One submitter advocated the need for NZFSA to explore accreditation being achieved via an “’association’ / ‘recognised group’ concept”.

Three submitters acknowledged the value of evaluators becoming accredited to ISO 17020, despite the costs and resources involved, providing sufficient transition time is allowed to complete the accreditation process. These submitters considered that accreditation was an effective tool to address concerns about the quality and competency of evaluation work.

7.8 Fees

The Discussion Paper proposed that NZFSA would recover the costs of evaluation from industry operators via an hourly fee. NZFSA could either on-charge the exact hourly charges made by evaluators, or it could charge a representative flat hourly fee, struck at a rate that would reflect the range of parties being used to evaluate.

Eighteen submitters commented on one or more aspects of the proposed process for setting fees.

Five submitters took the opportunity to emphasise that the setting of fees should be a matter between industry operators and their evaluators and did not warrant involvement by NZFSA. As one submitter expressed it:

“The concept of NZFSA controlling the fee payments under regulation removes the right of operators to fairly negotiate cost with evaluators.”

Six submitters indicated support for an hourly fee being set for evaluation. Of those that stated a preference as to the type of hourly rate, three submitters preferred that exact hourly charges be passed on, whilst one submitter preferred that a flat hourly fee be struck.

One submitter was opposed to an hourly rate because of concerns that it could be too open-ended. Another submitter was concerned that, in the event that an hourly rate was to be set, NZFSA should not subsidise this fee. A third submitter was concerned that any hourly fee would be likely to reflect the prices charged by the large evaluating agencies, such as AsureQuality and the VA, but that smaller independent evaluators with lower overheads would normally charge less for their services. In such a case, the question was posed as to what NZFSA would do with any variation between the price that evaluators have tendered and the hourly rate recovered from industry.

Four submitters were concerned about the complexities involved in balancing out variations in the charges made by different evaluators. As one submitter explained:

“There are a number of variables that should be considered in developing a suitable fee structure. These include direct costs, corporate overheads, accreditation costs, and other compliance costs…This will vary between supplier organisations…NZFSA will need to be mindful of this in determining the hourly rate to be used for evaluation work.”

In the Discussion Paper, NZFSA indicated that it would be possible to publish a schedule of estimated evaluation costs for particular types of plans and to establish reasonable time frames⁴ for undertaking evaluation. Seven submitters indicated that they would welcome the release of this information and/or operators being provided with quotes prior to evaluation being undertaken. Some submitters thought this would be particularly important if an hourly rate was to be charged for evaluation, so as to ensure that operators would have access to an estimate of the likely total charges and total hours involved in evaluation.

One submitter suggested that, if the proposal was to proceed, one option could be for evaluators to set and collect fees themselves, but then provide a percentage of these fees to NZFSA to cover management and quality control costs.

Another submitter stressed that industry must be consulted prior to NZFSA introducing any pricing plan for evaluation, commenting:

“..how does a verification manager reduce costs when also acting as a broker?...Who controls cost when industry is removed from the negotiation table?”

7.9 Timeframe

The Discussion Paper suggested a possible start date for the proposed new system of evaluation of 1 July 2009 (whilst noting this could be dependant on the timing of the passage of the new Food Bill through Parliament).

Three submitters commented on this timeframe and all indicated agreement with it.

Another submitter suggested that existing evaluators should be required to submit to NZFSA details of plans on which they are working as at 30 June 2009, following which they should be given a three month transition time to complete this work before switching over to the new system for evaluation.

8 Alternative options to the NZFSA proposal

Submitters were invited to identify any alternative options to the NZFSA proposal which they considered could address the issues and problems with the current framework for evaluation. This prompted a wide range of suggestions, relating both to improvements that could be made to the status quo and to new options.

8.1 Eliminate the role of the evaluator

One submitter argued that there would be merit in eliminating the role of the evaluator. This was explained as follows:

“NZFSA will take over the issue of evaluators that was in principle the need of a “filter” in between the regulator and industry. Under this principle, why keeping [sic] the figure of the evaluator if NZFSA will manage the system? …If it [the evaluator role] causes problems for regulators and industry…then this is a feasible option”.

8.2 Eliminate independent or third party evaluators

Another submitter suggested that all evaluation (and verification) of plans should be undertaken either by a local territorial authority or by NZFSA, thus eliminating the need for any independent or third party evaluators. This was explained as follows:

“Third party involvement is not necessary and is not wanted by industry. People want to deal direct with the real knowledge decision makers…The idea of independent auditors is not required. This is an inefficient and costly process for the business sector…if approval was to come direct from an internal NZFSA office …it would be expected that applicants would receive courteous and helpful advice on a consistent basis.”

8.3 A combined system

Five submitters suggested that, if the proposal for NZFSA to manage evaluation does go ahead, then it should operate simultaneously with the current system for evaluation. Under this scenario, industry operators would have the option of either submitting their plans for evaluation directly to evaluators, or of using NZFSA to locate and secure an evaluator.

The advantages identified by submitters for this combined system were explained as follows:

“…you tackle some of the problems highlighted, but still give the opportunity for industry to manage their cost of compliance and maintain good long term relationships with the evaluators that they normally use, under the same set of competency criteria required to be an evaluator in the NZFSA managed team”; and

“This would also give a good indication of which service industry prefers”.

8.4 A modified version of the NZFSA proposal

Two submitters suggested a modified version of the NZFSA proposal, which would involve NZFSA providing industry operators with a shortlist of recommended evaluators and operators, then taking responsibility for choosing which evaluator to appoint from the NZFSA shortlist.

This option was advocated as follows:

“…there are a few significant variations [to the status quo], namely that the NZFSA will still select a pool of providers based on competency with the factor of cost being removed as a selection criteria, and that the provision of a shortlist will give industry operators a valuable steer towards making an informed and appropriate choice.”

8.5 Improvements to the current framework for evaluation

The great majority of submitters supported the maintenance of the current framework for evaluation, rather than the changes proposed in the Discussion Paper. However, many submitters agreed that problems did exist with the current system of evaluation and provided suggestions as to how these could be overcome.

8.5.1 Improvements relating to quality, consistency and performance monitoring

Thirteen submitters suggested that new initiatives should be taken to improve the controls relating to evaluation quality, consistency and performance. Many of these initiatives involved NZFSA enhancing the role that it plays in this regard. Specific initiatives suggested by one or more submitters included:

• a recommendation by one submitter that the current system of sanctioning evaluators be reviewed by NZFSA (especially by the CIG and the VA) and that an improved system should then be developed in consultation with industry and evaluators;

• that CIG should conduct regular (for example bi-annual) audits of all evaluators and base the frequency of these audits on performance;

• that NZFSA should provide more specific training in all aspects of evaluation, including hosting workshops on good evaluator practice. Underperforming evaluators could be required to undergo re-training, and provision made for supervision in the event of recurring non-performance. Alternatively, poorly performing evaluators could be restricted in the level or type of evaluation that they perform, or have their recognition or approval as an evaluator/auditor revoked. Some submitters also thought that NZFSA should help existing evaluators to expand their areas of expertise by providing specialist knowledge and training on a specific subject such as thermal processing, or more background on industry sectors with which evaluators may be unfamiliar;

• that NZFSA should develop/revise the criteria and performance expectations for evaluators;

• that greater emphasis should be given to limiting the industries/sectors in which evaluators can perform, perhaps by “activity endorsements” which would prevent evaluators being able to evaluate risk based management plans for any industry/sector regardless of their experience;

• that evaluators be required to undergo annual re-registration, and that this be subject to the completion of a satisfactory audit of their evaluation activities;

• that a formal evaluation complaints and appeal process be established;

• that a consultative forum be set up to enable all interested parties to raise issues relating to the quality and consistency of evaluation. It was suggested that the issues to be considered by this forum should include: queries regarding standards and specifications that arise during the evaluation process, the cost of evaluation, and whether any work currently included in the scope of evaluation would be better handled as part of the verification process;

• that a competency standard for evaluators be developed and that all evaluators be assessed against this standard. (However, the submitter who contributed this suggestion did not want this standard to be ISO 17020 because they considered that the costs of achieving accreditation to ISO 17020 were high);

• that NZFSA should release an indicative schedule of the time that should be taken to complete the evaluation of a “typical” risk based management plan, also of reasonable charges for this work; and

• that NZFSA’s powers to deal with non-performing evaluators should be strengthened, if necessary by amendments to food safety legislation.

One submitter offered a suggestion for how these expanded quality control responsibilities could be funded, namely that evaluators should pay a percentage of their evaluation fees to NZFSA.

8.5.2 Evaluators and the regulator to work more closely

Eleven submitters suggested that one key to improving the current framework for evaluation would be for NZFSA, as the regulator, to build a closer relationship with evaluators.

One submitter argued that the cost recovery environment, in which NZFSA is required to operate, has restricted the resources which NZFSA is able to allocate to fostering a relationship with the evaluation industry:

“The opportunity to work more closely with the Regulator does not require a change to the existing operational model. We believe that currently it is not encouraged by the Regulator. The primary reason we see as being the obligation for the Regulator, under the cost recovery requirements of the Legislation, to charge time. We suggest that the Evaluator/Regulator interaction should be encouraged and the time spent in such interaction seen as an overhead…to the strengthening and maintenance of the relationship with evaluators. Evaluators should be encouraged to work closely with the Regulator (Registration) during RMP evaluation….the process will become more collaborative…”

Suggestions were made about the form which a closer relationship between evaluators and NZFSA might take, including that:

• there should be more extensive communication between evaluators and those within NZFSA that review evaluation reports, especially whenever rules or expectations about risk based management plans change:

“At present feedback takes the form of identifying negative aspects (which should continue), but I suggest it could also look to more advisory issues. For example, what format of evaluation report would best assist NZFSA?”; and

• NZFSA should work with evaluators to develop templates for evaluation reports, as well as other guidance material or checklists to which evaluators could refer.

8.5.3 Build the capacity of the evaluator workforce

Eight submitters considered there was scope to improve the size and capacity of the evaluation workforce and in this way overcome supply issues within the current evaluation framework.

NZFSA was seen as having a role to play in this by providing enhanced training to current evaluators as indicated in Section 8.5.1 above. A few submitters also suggested that NZFSA should be more proactive in encouraging new entrants to the evaluation workforce, for example by identifying persons with some or all of the relevant technical competencies, personal attributes and experience to be evaluators and training them in specific sectors where evaluators are in short supply.

Other suggestions made for resolving the evaluation workforce issues included:

• evaluation being promoted as a career choice by training providers, with incentives being offered for potential evaluators to enter the workforce;

• rethinking what should be involved in evaluation, with a view to conserving evaluator resources:

“Much of the time spent on evaluation is focused on non technical issues (such as document control) rather than in aspects critical to ensuring product that is fit for purpose is produced. A re-think of the current requirements around evaluation is urgently required. It is recommended that consideration be given to getting the evaluator to review essential systems rather than the entire RMP. This type of an approach would have the advantage of limiting the time required to complete evaluations and as a consequence should free up additional evaluation resource…”;

• industry associations ensuring their members are aware of available templates and codes of practice which, if followed when developing risk based management plans, can dispense with the need for the plans to be evaluated; and

• increasing the volume of information which is made available to industry operators about the choice of evaluators currently available, perhaps through additional information on the NZFSA website or through a joint publicity undertaking with evaluators; and

• given that most evaluators are also equipped to undertake verification of risk based management plans, it was suggested that the market for verification services should be opened up wherever possible, so as to provide more an incentive for persons/agencies to enter the market to provide both verification and evaluation services.

8.5.4 Other suggestions for improvements to the current framework

Other suggestions for improvements to the current framework for evaluation included:

• that a review should be conducted into all aspects of NZFSA’s management of the RMP (Risk Management Plan) process, including NZFSA’s involvement in the registration of plans and the processing of amendments to plans. This review should occur (and any issues identified be resolved) prior to decisions being taken to increase the management responsibilities of NZFSA;

• that NZFSA needs to provide clearer guidance about some aspects of evaluation, including the level of evaluation which will be required for custom-made Food Control Plans under the new Food Bill, and how to, and who should, determine when a plan deviates sufficiently from a template or code of practice to require evaluation;

• that a consistent decision be taken about whether Overseas Market Access Requirements (OMARS) should always be covered by an evaluation. (The submitter noted that OMARs are included within the evaluation scope of dairy RMPs, but not RMPs for other types of animal products); and

• that consideration is given to introducing a mechanism whereby, if an industry operator does not pay an evaluator within a set timeframe, then NZFSA can be notified and the registration of the industry operator’s plan be suspended until payment is made.

9 Other matters raised

Some submitters took the opportunity to offer comments about issues which, whilst they relate to the overall risk management framework, are beyond the scope of the proposal outlined in the Discussion Paper. However, in order to provide a complete summary, these comments are itemised below:

• One submitter was concerned that the system of Risk Management Plans and Food Control Plans was now a “very large complicated programme” which industry operators, especially small to medium sized enterprises, find costly and difficult to understand.

• Suggestions were made by several submitters that greater efforts should be focussed on opening up the market for verification to third party agencies. It was argued that this would lower the costs of verification and make it more attractive for third party agencies to invest in obtaining accreditation to ISO 17020, because they could expect to win not only evaluation work, but also verification work⁵ ; and

• concern was voiced by one submitter about the quality of verification and a call made for NZFSA to monitor verification more closely:

“…verifiers have a tendency to set their own agendas…Verifier variation is a much more significant problem for operators than is evaluator verification….[verifiers] in many cases have never read the RMP, let alone the evaluation report...”

Appendix 1: List of Submitters

Submitter Type	Code
Evaluators/auditors	E
Sector/industry organisations	SO
Industry Operators	IO
Government, including TAs and PHUs (DHBs)	G
Members of the public	MP

Sub Number	Submitter	Code
1	Blaha, Francisco	E
2	Brunton, Garry	MP
3	Achieve Enterprises Limited	E
4	Fonterra Co-operative Group Limited	IO
5	Hutt Valley District Health Board	G
6	Avivet Ltd	E
7	Southern Food Safety Limited	E
8	AsureQuality Limited	E
9	United Fisheries Ltd	IO
10	Auckland Regional Public Health Service	G
11	Bee Industry Group of Federated Farmers	SO
12	Meat and Fibre Producers, Federated Farmers	SO
13	Solutions in Seafood Limited	E
14	Lochiel Consultants Limited	E
15	McDonald, Marie	MP
Sub Number	Submitter	Code
16	Moana Pacific Fisheries	IO
17	Poultry Industry Association of NZ	SO
18	Seafood Industry Council	SO
19	NZ Cold Storage Association Inc	SO
20	Meat Industry Association of NZ Inc	SO
21	Langdon, Su	E
22	Dairy Goat Co-operative	SO
23	E.V.A. Services NZ Ltd	E
24	Independent Fisheries Ltd	IO
25	PPCS	IO
26	Verification New Zealand Ltd	E
27	Waikato District Health Board	G

¹ The term risk based management plan is attributed a wide meaning both in this document and in the Discussion Paper, and covers all types of plans/programmes falling under the various food laws which NZFSA administers. These include Risk Management Plans (RMPs) under the Animal Products Act 1999, Wine Standards Management Plans (WSMPs) under the Wine Act 2003 and Food Safety Programmes (FSPs) and Food Control Plans (FCPs) under the existing and proposed Food Acts.

² The Discussion Paper stated that the VA, which already evaluates a limited number of plans, would if necessary make evaluators available, but that this was expected to occur in circumstances where no other evaluator is available, or if the VA has particularly relevant expertise that is not readily available from other sources.

³ Defined in the Discussion Paper as evaluators from private and State Owned Entities.

⁴ In answer to a query from one submitter, it is noted that, by a reasonable time frame, it is meant the hours or time taken to undertake the evaluation of a plan, rather than the number of days/weeks that elapse between the receipt of the plan to be evaluated and the submission of the evaluation report.

⁵ One of these submitters thought that the VA was the only recognised verification agency for RMPs under the Animal Products Act 1999 (APA). However, a number of sectors which fall under the APA are contestable, meaning that other agencies have been, or could be, recognised to undertake verification.