What Did the Metric Throw Away?
Could information removed by this metric change the decision?
Situation
A metric appears to give a clear answer, but observations or experience don't quite agree.
The idea
Metrics are compression. They can be perfectly correct while excluding distinctions that matter to the actual decision.
Useful when
Model evaluationKPIsaveragesrankingsperformance comparisons
Watch for
Don't respond by measuring everything. Missing information matters only if it could alter the decision.
Where this came from
ML evaluation, Product work
Lineage
Pommerman's distinction between training reward and winning developed into asking which decision-relevant information a metric removes.