Noise Audit

In some organizations, there are jobs or roles where the routine is to make judgments, as they handle different cases. Courtroom judges, insurance adjusters, and radiologists are examples of roles where decisions require judgment on the specifics of the case.

We can study error in our organizational judgment capability by studying the variation in the judgments. The two types of error in judgment are bias and noise.

When there is variation across different judges on similar (or identical) cases, we call that form of error "noise".

This archery analogy is used by the authors of "Noise: A Flaw in Human Judgment" to illustrate the differences. Imagine four teams shooting at a target. One team is accurate. Another team is biased - they are consistently off the mark. A third team is noisy - they are inconsistently inaccurate. The fourth team is both biased and noisy, which is a good proxy for many of our decision making environments.

‍

‍

Noise is difficult to study because it is largely invisible. Scattered shots present a less recognizable pattern than a clear bias.

A "Noise Audit" is a way to make it visible, and through that, to examine the quality of the professional judgments that are produced by the employees of an organization.

In a noise audit, a set of hypothetical cases - similar in shape and variety to real cases - are prepared for the study. A group of "judges" are pulled into the study to examine the full set of cases, and issue a judgement for each. The variation of the judgments, across cases, across judges, across judges for a single case, and across cases for a single judge are analyzed.

In this summary example, the judgment was expressed as a number (e.g. a sentencing in a court case, in years). Even though the cases are made-up, the study can still reveal patterns in the consistency of how judges arrive at decisions. Note that the study should be conducted anonymously; the purpose is to draw conclusions about the organizational capability, not the capabilities of individual judges (i.e. the participants in the study).

‍

‍

‍

Statistical analysis can isolate the different kinds of noise exposed in the study. If the cases are trusted to be representative of the "real work" of the organization, then this can become a trigger for improving the decision architecture, to reduce noise in judgments.

‍

‍

The total measurement error exposed by the study can be broken down into two components, bias and (system) noise. The noise component can be further broken down into these subcomponents:

• Level Noise is variability in the average level of judgments by different judges
• Pattern Noise is variability in judges’ responses to particular cases
• Stable Pattern Noise is a consistent source of variability in specific judgments due to characteristics of the case
• Occasion Noise is variability in specific judgments due to the characteristics of the moment in which the judge made the judgment

‍

‍

To produce a noise audit that can yield this analysis, try the following approach: (from Appendix A of the book, "Noise")

‍

Procedure for conducting a noise audit:

1. Define Audit Team - Build a project team for the audit that includes subject matter experts (i.e. experts on the judgments under study), a strong sponsor (who is accountable for the overall quality of the judgments), a set of judges (who will be “audited”), and a project manager (who can run the audit as a project).
2. Create Case Materials - This is the most crucial step. The subject matter experts must create a set of “cases” that mimic the real work, and will drive a fake judgment, for the audit purposes. The cases must present credible simulation of the real work, otherwise the results of the audit will not be trusted.
3. Create a Questionnaire - This should cover the full set of cases, and be given after the participants complete all the cases. The questionnaire asks for details on the participants thinking:some text
   1. What were the key factors that led to their response?
   2. Which of the facts (for each case) were deemed most important?
   3. How do the judgments of the case compare to an overall average (i.e. get the “outside view”)  ?
4. Pre Launch meeting - Share the design of the case materials and the possible outcomes of the audit with the leadership sponsors. Ask for objections, and get a commitment to accept the results - in advance. For extra credit, ask the sponsors for their expectations of the study - “What level of disagreement would you expect between two judges?” “What would be considered an acceptable level of disagreement across judges?”
5. Run Study - Be careful not to refer to the study as a “noise audit” when introducing it to the participants (i.e. the judges). Refer to it as a “decision making study” instead. Assure participants that individual answers will not be shared, and/or that the responses will be kept anonymous. Have all participants complete the exercise at the same time (e.g. over a half day) and isolate them from each other to keep responses independent.
6. Analysis - The project team runs the statistical analysis to measure the amount of noise and break it down into constituent components, like pattern noise and level noise.Seek to understand the sources of the variation in responses via the questionnaire answers. Look for patterns and point to possible issues with training, policies, or availability of supporting information.
7. Conclusions & Next Steps - Present the results back to the sponsors, with recommendations for improvements in tools and procedures that can improve the decision hygiene and decision architecture.

‍