The Metric Can Be Right
Make a model choice, then look at what that comparison left out. Each step revisits the same decision with a wider view.
- Choose
- Reveal
- Consequences
- Review
- Decide
Part 1 of 5 · A reasonable first choice
Your bank needs a fraud-detection model.
Two models were tested on the same 10,000 transactions. Which would you deploy?
$200 to process these 10,000 transactions
$500 to process these 10,000 transactions
These are invented figures for one test, not measured bank results. “To process” is only the cost of running each model on those transactions; it does not include investigating alerts or the consequences of mistakes.
Part 2 of 5 · Reveal the compression
What did accuracy hide?
The aggregate scores are close. First predict: do you think the two models make similar kinds of mistakes?
Same transactions, different mistakes
A fraud model scores each transaction, and a decision rule determines whether to let it through or block it. A missed fraud is a fraudulent transfer that goes through; a false decline is a legitimate payment that gets blocked.
The full test contains 10,000 transactions. To make individual outcomes visible, this diagram shows 1,000 of them: exactly one-tenth of the correct decisions, missed fraud cases, and false declines. Each dot is one transaction, and the same position represents that transaction for both models. The counts below the diagram return to the full 10,000.
Model A
Model B
Same transaction positions in both models; the colors show each model’s outcome.
What changed? Accuracy counts both errors the same. It does not tell you which error the model made.
Part 3 of 5 · Different consequences
Should those errors count equally?
You have seen both kinds of mistake. Their consequences can be different:
A fraudulent transfer goes through. The bank may lose money.
A legitimate payment is blocked. The customer is inconvenienced, and the bank may lose business.
The error counts have not changed. Now estimate what each kind of mistake could cost.
A missed fraud case remains fixed at an illustrative $600. Move the slider to see when the extra false declines change which model has the lower estimated cost.
What changed? The models and their accuracy did not change. The value assigned to an error changed—and that can change the preferred model.
Part 4 of 5 · Widen the system boundary
What happens when a person reviews an alert?
Both models flag this $2,400 transfer as high risk. A human analyst then reviews the alert and decides what to do. Here we compare two hypothetical alert screens—not a built-in difference between the predictive models. One screen shows the alert alone; the other also shows the evidence the analyst needs.
The analyst sees the alert, but the supporting evidence is not shown. They must find it in other systems before deciding what to do.
The analyst looks up the recipient, device, and account changes separately. Illustrative human review: 10 minutes.
The alert includes the new recipient, unusual device, and recent account changes, so the analyst can review that context in one place.
The analyst checks the evidence supplied with the alert. Illustrative human review: 2 minutes.
Across the same 10,000 transactions
Model A raises 160 alerts; Model B raises 260. If the case times above were typical for each alert, what would that mean for the human review workload?
The 10- and 2-minute examples are not measured averages. If real alerts take different amounts of time, these totals change. Review speed also depends on what the product shows the analyst, so a deployment comparison needs to include the model and its review workflow.
What changed? The evaluation boundary grew from model output to human review. A single case suggests a possible difference; it does not establish an average workload.
Part 5 of 5 · Decide what to measure next
When is more information worth collecting?
Revisit your first choice with the wider view. What could still change it?
Optional counterfactual
Now suppose the same percentages came from only 1,000 transactions.
The earlier 10,000-transaction result has not changed. With fewer cases, the error-rate estimates would be less stable. Would your next step change?
Beyond this model
What did the metric throw away?
The right question is not “What else could we measure?” There is always more. Ask whether omitted information could plausibly change this decision. If it cannot, move on. If it can, investigate that boundary.
The full essay develops why a perfectly correct metric can still answer a smaller question than the decision requires.
Read The Metric Can Be Right and the Decision Can Still Be Wrong →How this illustrative model works
The test contains 10,000 transactions, including 200 fraudulent ones. Model A misses 60 fraud cases and falsely declines 20 legitimate ones; Model B misses 20 and falsely declines 80. Thus Model A has 9,920 correct predictions (99.2%) and Model B has 9,900 (99.0%). The marks show a representative one-tenth slice of that evaluation, not a separate measurement.
The cost comparison assumes $600 per missed fraud case, a reader-adjustable amount per false decline, and the stated model running costs. The human review projection uses 160 Model A alerts or 260 Model B alerts per 10,000 transactions, illustrative case times of 10 and 2 minutes, and $40 per analyst-hour. It does not establish actual review times, true loss distributions, customer effects, or a deployment recommendation. All figures and interfaces are invented to expose the decision boundary.