What makes a good decision, good?

Decision quality is measured by its process, not its outcome. Measuring decisions by observing the outcome is subject to outcome bias, or the resulting fallacy. Even if the best decision was made given the information available at the time, luck is a meaningful variable.

“Outcomes don’t tell us what’s our fault and what isn’t, what we should take credit for and what we shouldn’t. Unlike in chess, we can’t simply work backward from the quality of the outcome to determine the quality of our beliefs or decisions. This makes learning from outcomes a pretty haphazard process.” ― Annie Duke

So if we say decision quality can only be observed from its process, what exactly are we looking for? The answer seems to be the ratio of process rigor (that comes at a cost - people and time) to risk (how consequential the decision is).

Of course, the last thing we'd want is relatively inconsequential decisions going through an overly-thorough decision making process - so there's a bit of an art to deploying the right tactics for the right decisions. This is something we explore in the Uncertainty Project.

Is a decision 'good' just because it's 'data-driven'?

There's plenty of chatter about data-driven decisions (for good reason!), but what does it mean when we say process is an indicator of a quality decision? Are there other factors besides information and analysis?

McKinsey ran a study on 1,048 R&D decisions made over five years including investments in new products, M&A decisions, and large capital expenditures - they were trying to answer this question of what the meaningful factors were in a quality decision.

Specifically, what factors indicate high quality decision making?

For each decision, they asked questions that surfaced techniques used for data and analysis as well as process and dialog - which factors were better indicators?

"process mattered more than analysis—by a factor of six (Exhibit 2). This finding does not mean that analysis is unimportant, as a closer look at the data reveals: almost no decisions in our sample made through a very strong process were backed by very poor analysis. Why? Because one of the things an unbiased decision-making process will do is ferret out poor analysis. The reverse is not true; superb analysis is useless unless the decision process gives it a fair hearing."

‍

‍

In this context, 'dialog and process' means deploying techniques to curb the impact of cognitive biases - or as Daniel Kahneman and Olivier Sibony describe these systemic variations in judgement, 'Noise'.

How can we systematically improve decision quality?

This question is, of course, a core tenet of the Uncertainty Project. Some examples of questions that tease out these techniques (whether used implicitly or explicitly) are:

• Were dissenting opinions and alternative options entertained?
• Were there principles and criteria that drove the evaluation of those options?
• Was the information supporting the decision effectively interrogated (avoiding the issues of base rate neglect and the feature-positive effect)?
• Was there counterfactual thinking that explored the probabilities of outcomes?
• Were 'tripwires' or kill-criteria defined in case new information changes how we might think about this decision?
• Were steps taken to reduce the impact of groupthink and other biases that impact collaboration?

Many individuals and teams implement these tactics implicitly. Some people or organizational cultures are better at skillfully disagreeing or playing devils advocate. This is the exception, not the rule.

But there's value in making these processes explicit. Much like a scientific paper breaks down the methods of the experiment, the decision process can show and justify the rigor. I believe the technical term for this is 'CYA' 👀

It insulates from the worst scenario for outcome bias - when a good decision results in a bad outcome.

Decision making processes in companies, teams, and individuals already exist - they're just implicit. Making these processes explicit opens them up to learning and iterating.

“A wise leader, therefore, does not see herself as someone who simply makes sound decisions; because she realizes she can never, on her own, be an optimal decision maker, she views herself as a decision architect in charge of designing her organization’s decision-making processes.” ― Olivier Sibony

‍