The Shape of the Evidence
What would I expect to see around this result if my explanation were actually right?
Situation
One result looks substantially better than the alternatives.
The idea
A result is more convincing when the surrounding evidence has a structure consistent with the explanation. An isolated maximum may be real, but it demands explanation.
Useful when
Experimentsbenchmarksmodel selectionvendor comparisonsapparently exceptional results
Watch for
Real phenomena can have thresholds and discontinuities; smoothness is evidence, not proof.
Where this came from
Quantitative trading, Machine learning
Example
A model performs best at one exact parameter value while nearby configurations perform poorly. Before selecting it, ask whether the spike represents a real phenomenon or experimental noise.
Lineage
Skepticism about unusually good quantitative research results developed into examining the structure of surrounding evidence.