Brier score decomposition: reliability, resolution and uncertainty

Two models publish a year of forecasts on the same matches. One scores 0.240, the other 0.250. The first is worse calibrated by a factor of thirty-two — and it is still the more useful model. The raw scores do not tell you that. The Brier score decomposition does: it splits a single Brier score into three terms — reliability, resolution and uncertainty — that answer three separate questions about a forecaster. A score on its own is a verdict with no reasoning attached. The decomposition is the reasoning.

If you have not met the metric itself, start with how the Brier score is computed. Everything below assumes the score exists and asks what it is made of.

What the Brier score decomposition splits

Take a binary event — over 2.5 goals, both teams to score, home win yes or no — and every forecast you have published on it. Group the forecasts by the probability you stated, so that each group carries one stated value. Write N for the total number of forecasts, nₖ for the count in group k, pₖ for the probability stated in that group, oₖ for the fraction of those matches where the event actually happened, and o for the fraction across all N.

The mean Brier score then obeys an exact identity:

BS = REL − RES + UNC

• Reliability (REL) = (1/N) Σ nₖ(pₖ − oₖ)². How far your stated probabilities sit from what actually happened at that probability. Zero is perfect; it is added, so lower is better.
• Resolution (RES) = (1/N) Σ nₖ(oₖ − o)². How far your groups pull away from the base rate. It is subtracted, so higher is better. A forecaster who says the same thing about every match has zero resolution.
• Uncertainty (UNC) = o(1 − o). The variance of the event itself. Nothing you do changes it. It is a property of the market, not of you.

Two of those terms describe the forecaster. The third describes the question. Almost every misreading of a published Brier score comes from failing to separate them.

A worked example: 100 forecasts, three numbers

Hypothetical figures, chosen to keep the arithmetic visible. A model publishes over/under 2.5 forecasts on 100 matches using four probability levels. The event occurred 50 times in total, so o = 0.50.

Uncertainty: 0.50 × 0.50 = 0.250

Reliability: [25(0.30 − 0.32)² + 25(0.45 − 0.44)² + 25(0.55 − 0.56)² + 25(0.70 − 0.68)²] / 100 = 0.025 / 100 = 0.00025

Resolution: [25(0.32 − 0.50)² + 25(0.44 − 0.50)² + 25(0.56 − 0.50)² + 25(0.68 − 0.50)²] / 100 = 1.80 / 100 = 0.018

Brier score: 0.00025 − 0.018 + 0.250 = 0.2323

Score the 100 forecasts directly, one squared error at a time, and you get 23.225 / 100 — the same number. The decomposition is not an approximation; it is the same total, sorted into piles.

And now the score reads as a sentence rather than a digit. The market handed this forecaster a bill of 0.250 before they said anything. Sorting matches into groups that genuinely differ earned 0.018 of it back. Stating probabilities slightly off the observed frequencies cost 0.00025. Everything the model contributed lives in that third decimal place, which is exactly why Brier scores quoted to two decimals are unreadable.

Reliability is the cheap part; resolution is the expensive part

Compare three forecasters on the same 100 matches.

The timid model publishes 0.50 every time. It is perfectly calibrated — half of the matches did go over. REL = 0, RES = 0, and BS = 0 − 0 + 0.250 = 0.250. Flawless reliability, zero information, and a score no better than announcing the base rate. A model can be perfectly calibrated and completely useless, and only the decomposition shows it.

The overconfident model separates matches exactly as well as the first table did, but pushes its probabilities to the extremes: 0.20, 0.40, 0.60 and 0.80 against the same observed frequencies. Resolution is untouched at 0.018, because resolution depends only on oₖ. Reliability climbs to 0.80 / 100 = 0.008 — thirty-two times worse. Its score: 0.008 − 0.018 + 0.250 = 0.240.

Which is still better than the perfectly calibrated model. This is the pair from the opening paragraph, and it is the most useful thing the decomposition teaches: badly stated probabilities that carry real information beat well-stated probabilities that carry none.

It gets sharper. Replace each of the overconfident model’s stated probabilities with its group’s observed frequency and reliability falls to zero, leaving 0 − 0.018 + 0.250 = 0.232. That is its potential Brier score — what it would score after recalibration, using no new information about football at all. Reliability is a post-processing problem, fixable by a transformation applied to the outputs. Resolution is not: no relabelling of the timid model’s forecasts can create discrimination it never had. That is why calibration is worth auditing but is never the whole test of a model.

Uncertainty is why Brier scores don’t transfer between markets

The uncertainty term peaks at 0.25 when the event is a coin flip and collapses as the base rate moves toward either edge. Take a market where the event happens 20% of the time: UNC = 0.20 × 0.80 = 0.16.

A forecaster with no skill whatsoever in that market — one who publishes 0.20 for every fixture, forever — scores exactly 0.16. That beats the genuinely informative model above, which scored 0.2323, while knowing nothing at all.

So a lower Brier score is not evidence of a better model when the two scores come from different markets, different leagues with different home-win and draw rates, or different periods with different scoring levels. Often it is only evidence of a more lopsided base rate. Any comparison worth making holds the uncertainty term fixed, either by scoring both forecasters on the identical set of matches or by dividing it out.

Dividing it out gives the Brier skill score: 1 − BS/UNC, which the identity rewrites as (RES − REL) / UNC. For our worked example that is (0.018 − 0.00025) / 0.250 = 0.071, a 7.1% improvement on knowing only the base rate. The numerator is the entire contribution of the forecaster, with the difficulty of the question divided away.

How to read a published track record with this in mind

Four questions worth asking of any prediction record, ours included:

1. What is the base rate of the scored event? Without it, the headline score has no scale. A record covering markets with different base rates cannot be summarised by one Brier number.
2. Is the forecaster’s spread of probabilities wide or narrow? Narrow spreads buy safe-looking scores by declining to say anything. Resolution is the term that catches it.
3. Do the observed frequencies track the stated ones? That is reliability, and it is the part a good record makes checkable by publishing every forecast before kick-off rather than a summary afterwards.
4. How many forecasts are in each group? Every term here is an average over nₖ. Groups of a dozen matches produce reliability figures that are mostly sampling noise.

Our public track record exists so those questions can be answered from the outside rather than taken on trust — which is the only reason a score means anything at all.

A note on 1X2

Everything above is written for a binary event, where the decomposition is cleanest. Home/draw/away is three outcomes, and the standard approach is to apply the decomposition to each outcome separately and add the three sets of terms, giving the multi-category Brier score its own reliability, resolution and uncertainty. The interpretation carries over unchanged: uncertainty belongs to the league, resolution belongs to the model, and reliability is the part that recalibration can repair.