Transparency

Calibration in football predictions: does 70% really mean 70%?

Published 2026-08-21

A model tells you the home team is 70% likely to win. The match is played exactly once; the team either wins or it doesn’t. So what could it even mean for that 70% to be right? The answer is calibration in football predictions: across all the matches a model prices at 70%, roughly seven in ten should actually end that way. Calibration is the most checkable claim a probability can make — a single prediction can never be wrong on its own, but a thousand of them together absolutely can. It’s the property our public track record exists to put on display, and the one most prediction sites never let you test.

What calibration in football predictions actually means

Take every prediction a model has ever issued and sort them into buckets by stated probability: everything called 40–50% in one pile, 50–60% in the next, and so on. Then, for each pile, count how often the predicted outcome actually happened.

A calibrated model’s piles match their labels. Outcomes it called 45% happen about 45% of the time; outcomes it called 75% happen about 75% of the time. Not on any single match — over the pile.

This is a different property from being accurate. Accuracy asks whether the top pick won. Calibration asks whether the number attached to the pick was honest. A model can post a decent accuracy figure while its probabilities are systematically inflated — and for anyone using those probabilities to make decisions, the inflation is what does the damage. Stake-sizing frameworks, value comparisons against bookmaker odds, even the simple judgement “this is confident enough to act on” all consume the probability, not the pick. Feed them miscalibrated numbers and every downstream calculation inherits the error.

The reliability diagram: calibration made visible

The standard picture is a reliability diagram: stated probability on the horizontal axis, observed frequency on the vertical. Each bucket becomes one point. A perfectly calibrated model traces the diagonal — 60% predictions land 60% of the time, and the point sits exactly on the line.

Here’s what the underlying table might look like for a hypothetical model’s home-win predictions:

The lower buckets sit close to their labels. The top two drift below the line: matches this model calls 70–80% are landing at 66%. That’s overconfidence — the most common failure mode in football modelling, and the shape to look for first in any reliability diagram. The opposite pattern, observed frequencies persistently above the stated probabilities, is underconfidence: the model knows more than it’s willing to say, and its cautious numbers waste real information.

Why is overconfidence the default? Partly because models are fitted to finite history and mistake some noise for signal, so their extremes are sharper than the world justifies. And partly because of selection pressure: confident numbers are more compelling, so sites that publish “82% banker” outperform honest ones in marketing even while underperforming them in reality. A site that never shows you the bucket table is asking you to take its confidence on faith.

Calibration is necessary — but not sufficient

Here’s the catch that keeps calibration from being the whole story. In a league where home teams historically win about 44% of matches, a “model” that predicts 44% for every single home team is perfectly calibrated. Its one giant bucket lands exactly on its label. It is also completely useless — it contains no information about any particular match.

What separates a real model from that flatline is sharpness: the willingness to move away from the base rate, to call one match 25% and another 70%, while staying calibrated at both ends. Sharp but miscalibrated is dangerous; calibrated but flat is empty. Good forecasting is the tension between the two — maximise sharpness subject to calibration, as the forecasting literature puts it.

This is exactly what proper scoring rules measure. The Brier score rewards both properties at once, which is why it can’t be gamed by either the coward’s flatline or the marketer’s false confidence — and why it’s the headline metric on our track record rather than a bare win rate.

How to check calibration yourself

Any prediction source that publishes its full history with stated probabilities can be audited with nothing more than a spreadsheet:

1. Collect every prediction — not a curated sample. Losses included, or the exercise is void before it starts.
2. Bucket by stated probability, using bands wide enough that each bucket holds a real sample. Ten-point bands are conventional.
3. Compute each bucket’s observed frequency and compare it with the bucket’s label.
4. Respect the noise. Small buckets lie. With 50 predictions at a true 70%, ordinary binomial variance puts the observed frequency anywhere between about 63% and 76% one standard deviation out — a gap that looks like miscalibration and is pure sample size. Judge buckets with hundreds of predictions, treat buckets with dozens as suggestive at most, and expect verdicts to firm up only as the record grows.
5. Look at the pattern, not one bucket. A single stray point is noise; every high bucket sitting below the diagonal is a diagnosis.

One football-specific wrinkle: the three 1X2 outcomes live in different probability neighbourhoods. Draw predictions cluster in a narrow band around the high 20s — no honest model ever calls a draw 70% — so a draw calibration check happens almost entirely within a few buckets. Home wins span the widest range and give the richest diagram. Checking each outcome separately tells you more than pooling them.

And that’s the real test of a prediction site: not whether the audit comes out flattering, but whether the site gives you the material to run it at all. That requires probabilities frozen before kick-off, an append-only history, and results settled in public — the standard our methodology commits to. A site that shows you only its wins hasn’t failed a calibration audit; it has refused to sit the exam.

What to take away

A probability is a promise about frequency. Calibration is whether the promise is kept, a reliability diagram is how you check, and sharpness is what makes the promise worth anything in the first place. When you evaluate any football prediction model — this one included — ask for the bucket table before you ask for the win rate. The win rate flatters; the diagonal doesn’t.