What we said · What happened

Calibration: what we said, and what happened

A hit rate tells you how often we were right. It cannot tell you whether a 70% was really a 70%. This page can: every probability we have ever published, grouped by what we said, against how often it came true — with the sample size on every row, because a bucket of two calls proves nothing in either direction.

Every call, in one chart

0%0%20%20%40%40%60%60%80%80%100%100%perfect calibration We said 38% · it happened 38% · 684 calls We said 44% · it happened 47% · 1418 calls We said 55% · it happened 55% · 6196 calls We said 64% · it happened 63% · 3240 calls We said 74% · it happened 76% · 2672 calls We said 84% · it happened 85% · 1254 calls We said 93% · it happened 93% · 373 calls What we saidWhat happened
Stated probability against observed frequency per bucket, with sample sizes
We saidAverageHappenedGapCalls
30–40%38%38%0684
40–50%44%47%+21418
50–60%55%55%06196
60–70%64%63%−13240
70–80%74%76%+22672
80–90%84%85%+11254
90–100%93%93%0373

Across 15837 settled calls, the average gap between what we said and what happened is 0.9 points. The widest readable bucket is 40–50%: we said 44% and it happened 47% on 1418 calls.

By market

Stated probability, observed frequency and sample size per bucket and market
We said1X2Double chanceOver/Under 2.5BTTS
30–40%38% → 38% · 684———
40–50%44% → 47% · 1418———
50–60%55% → 60% · 913—55% → 54% · 235955% → 54% · 2924
60–70%65% → 67% · 54369% → 69% · 20164% → 63% · 145363% → 59% · 1043
70–80%74% → 79% · 28475% → 76% · 221472% → 75% · 16172% → 92% · 13
80–90%84% → 80% · 12384% → 85% · 112482% → 100% · 7—
90–100%92% → 87% · 1593% → 93% · 358——

What we said → what happened · calls. Buckets under 20 calls are readings of nothing; they are printed so you can see they are thin.

How to read it

The diagonal is a model that means exactly what it says. A point above it is a bucket where things happened more often than we said — we were too modest. A point below is one where they happened less often — we were too confident. The distance from the line is the error in percentage points, and the table beside the chart prints it with its sign. A longer walk through reliability diagrams is in calibration in football predictions, and what a good Brier score looks like puts the number beside its benchmarks.

Why a 90% call must lose one time in ten

A model that said 90% and was right every time would not be a better model. It would be a worse one, hiding a 97% behind a 90%. Honest probabilities lose exactly as often as they promise to, which is why a screenshot of one 90% pick that failed proves nothing about the model that made it, ours or anyone else's. The only fair test is the one on this page: many calls at the same number, and how many came in.

Why the sample size is on every row

Because a bucket of two calls can read as 100% or 50% and mean nothing either way. The chart refuses to draw a point below 20 calls; the table keeps the small buckets and says how small they are. As the ledger grows the thin buckets fill in and the points appear on their own — the page is rebuilt from the settled rows several times a day.

What is here, and what is not

Four markets: match result, double chance, over/under 2.5 goals and both teams to score. Those are the markets whose stated probability the public ledger carries per call; over/under 1.5 and 3.5 and correct score are settled on the track record but not plotted here. Every model that has published on this site is included, retired ones too. The rows are the same ones you can download from the open dataset, so anyone can rebuild this chart and check it.

Frequently asked questions

What does it mean for a prediction to be calibrated?

That the number means what it says. If every call we made at 70% is collected together, about seven in ten of them should have come true. More is not better: a set of 70% calls that lands nine times in ten was underselling itself, and a set that lands half the time was overselling. Calibration is the property a hit rate cannot measure, because a wide market wins often whatever the model knows.

How is this chart built?

Every settled call on the public ledger contributes one point: the probability we published for the side we called, and whether that side came in. Calls are grouped into ten-point buckets by what we said; each bucket plots its average stated probability against the share that happened. The chart is recomputed from the append-only ledger at every build and never edited by hand.

Why does every bucket print its sample size?

Because a bucket of two calls can read as 50% or 100% and mean nothing either way. A point is only drawn once a bucket holds at least 20 calls; smaller buckets stay in the table with their count so you can see exactly how thin they are.

Which markets are included?

Match result (1X2), double chance, over/under 2.5 goals and both teams to score — the markets whose stated probability the public ledger carries. Over/under 1.5 and 3.5 and correct score are settled on the track record but not plotted here. Every model that has published on this site is included.