The idea
Metrics are compression. They can be perfectly correct while excluding distinctions that matter to the actual decision.
Situation
A metric appears to give a clear answer, but observations or experience don't quite agree.
Useful when
Model evaluationKPIsaveragesrankingsperformance comparisons
Watch for
Don't respond by measuring everything. Missing information matters only if it could alter the decision.
Where this came from
ML evaluation, Product work
Lineage
Pommerman's distinction between training reward and winning developed into asking which decision-relevant information a metric removes.