Guidance on handling outlier institutions

The NCIN's approach to the handling of outliers has been developed from Department of Health and Healthcare Quality Improvement Partnership guidance and experience within the network of dealing with these issues. The approach described below recognises that flexibility is required to account for different circumstances and therefore provides a set of principles to be followed rather than a detailed procedure to cover every eventuality. The NCIN co-ordinating team can advise on planning for the handling of outliers and may be able to assist with the management of the process.

Defining an outlier

DH and NAGCAE guidance defines statistical outliers as those organisations more than two standard deviations away from the expected performance. The analytical team should consider what is appropriate for the specific circumstances. Where large numbers of organisations are being compared, this may need to be increased to three or more standard deviations or a formal correction for multiple testing applied.

Principles for handling outliers

1. Ensure strong clinical involvement from the start

Clinical involvement will help to ensure that the analysis is clinically relevant and appropriate and will assist in communicating the results. Clinical input will also be important for determining the likely sensitivity of outputs. If there is not existing clinical engagement then it may be possible to arrange this via the relevant NCIN Site Specific Clinical Reference Group, which should in any case be made aware of the work.

2. Engage early with professional leadership

Engagement with professional leadership (for example Royal Colleges and professional organisations) will allow any concerns that may be raised to be addressed early. These organisations may also be able to assist with communicating the results and with follow up to ensure that findings are used to improve practice. As well as professional bodies, organisations like the Care Quality Commission may play a role in following up the findings.

3. Build sufficient time into plans for publication

The process for handling potential outliers is time consuming, both because it may be necessary to provide organisations with an opportunity to review the results and respond prior to publication and because those conducting the analysis may be communicating with multiple organisations at the same time. As an indication of the time required, the DH and NAGCAE guidance recommends a total of 85 working days from start to finish.

Publication in peer reviewed journals may raise particular difficulties, both around the communication of results prior to publication and the need to build in sufficient time to communicate with outlier institutions. Where work will be published in a journal the authors should discuss this with the editors to agree a mutually acceptable approach.

4. Inform all organisations included in the analysis that it is happening and the plans for publication. Give them the opportunity to receive their (embargoed) results in advance

Once plans for publication of the results are known, all organisations included in the analysis should be contacted with information about the work and when and how it will be published. If appropriate, organisations should be offered the chance to receive their embargoed results in advance and informed that they will be contacted individually if they are identified as a potential outlier. Communications should be addressed to appropriate roles within the organisation rather than specific named individuals (who may have changed roles). Depending on the nature of the work this could be the relevant clinical lead, medical director or another senior clinician.

The way in which organisations are informed of results will depend to some extent on the details of the work, in particular the number of organisations involved and the sensitivity of the results. For communication to large numbers of NHS trusts, post has to date proved the most effective means of reaching the appropriate individuals but is time consuming. Where comprehensive address lists exist, or for smaller numbers of organisations where these can be readily collated, email may offer a less labour intensive means of communication. The current NHS email systems do not provide the means to easily reach those occupying particular appointments, making it difficult to use these for large scale notifications. In some cases it may be more practical to cascade general communications via intermediary organisation (for example cancer networks or commissioning groups).

Preparing results for individual organisations and sending these by post can involve significant effort. This approach aims to minimise this by providing most organisations with generic information about the work and focussing attention on those most likely to request detailed information for review. In the medium term, the NCIN will investigate other mechanisms for distributing results.

Other organisations with an interest in the work should also be informed (for example strategic health authorities, cancer networks and cancer registries). Finally, work which identifies outliers may well generate media interest; the NCIN press office should be informed early (via the co-ordinating team) and can advise on media handling.

5. Inform potential outliers well in advance of publication and be prepared to address their concerns

Ideally a senior member of staff at potential outlier organisations (medical director or relevant clinical lead) should be contacted by a telephone call from an appropriate member of the analytical team, if possible a senior clinician, to explain the work and the results. This contact should be followed up with detailed information sent to appropriate senior appointments at the trust (for example the medical director, relevant lead clinician, and chief executive). The communication should clearly explain the analytical methodology and results, together with the criteria used to detect potential outliers, and offer the opportunity to engage in a detailed discussion.

At least 25 working days should be allowed for the organisation to respond, followed by sufficient time to adequately consider this (the DH and NAGCAE guidance recommends a further 30 days). During this period outlier organisations may wish to receive the data underlying the analysis. To facilitate this, the procedure for supplying these data to organisations should be thought through in advance. If data are identifiable then the basis on which the disclosure will be made should be considered carefully and appropriate confidentiality agreements prepared (where organisations are receiving their own data the UKACR confidentiality form may be sufficient for this).

It is possible that a detailed review of an organisation’s results will lead to changes or bring new information to light. Plans should be in place to handle this appropriately, for example by revising the planned publication before it goes to press or, afterwards, by updating or annotating the published results with the new information if this is possible.

6. Be prepared to conduct additional analyses

There may be a requirement for the team performing the initial work to undertake additional analyses. This situation may arise for two reasons. Firstly the availability of more up to date data. This is always a strong possibility in this situation, and it is imperative that if more up to date data become available that these are reviewed by the analytical team, otherwise the whole publication may be undermined. The second reason is that individual trusts or clinicians may request comparisons of their own data with the data being used in the study. Capacity needs to be identified for these possibilities at an early stage.

7. Respond sensitively to requests and challenges

This whole process is likely to prove challenging to outlier organisations and inevitability there is likely to be strong challenge when the results of studies are presented. It is important that a named individual from the study team is available to speak to trusts and clinicians with concerns. Finally, the identification of outliers may well lead to media interest; those involved in dealing with these situations should be sufficiently trained to handle media enquiries.

8. Emphasise positive variation and opportunities to share good practice

There is a tendency for attention to focus on outliers whose results are worse than expected. Analyses of variation in outcomes should not be seen as pejorative exercises but as opportunities to improve outcomes by learning from those whose results are particularly good. Communications should, wherever possible, emphasise the achievements of any teams and organisations with better than expected results. There may also be positive messages to drawn from the overall results, for example, general improvements in outcomes over time.