Two models can correctly classify six out of ten events while making very different claims about their uncertainty. Accuracy records whether the selected side was right. It does not fully describe the quality of the probability attached to that selection.
Start with identical decisions
In this constructed example, the selected outcome occurs in six of ten separate binary events. Model A assigns a 60% probability to every event. Model B assigns 90% to every event. With a 50% classification threshold, both choose the same side on all ten occasions and both achieve 60% accuracy.
The decisions match, but the probability claims do not. Model B expresses much greater confidence. A probability-sensitive measure can make that distinction visible without changing the underlying outcomes.
Calculate the binary Brier score
For a binary event, the Brier score is the average squared difference between predicted probability p and observed outcome y, where y is 1 when the event occurs and 0 otherwise. Under this definition, a lower score is better.
For A, each of the six occurrences contributes (0.60 − 1)² = 0.16. Each of the four non-occurrences contributes (0.60 − 0)² = 0.36. The average is (6 × 0.16 + 4 × 0.36) / 10 = 0.24. For B, the calculation is (6 × 0.01 + 4 × 0.81) / 10 = 0.33.
A therefore has a lower Brier score in this example despite identical classification accuracy. The calculation concerns these ten constructed outcomes; it does not establish the performance of a real forecasting system.
Calibration asks a related but different question
If events assigned a 60% probability occur about 60% of the time, that is consistent with calibration in that probability group. Ten observations at a single probability level do not establish that a model is well calibrated overall. Inspect other probability ranges and retain their observation counts.
The Brier score is not a pure measure of calibration alone. Other aspects of probability performance affect it. Reliability diagrams, discussed in scikit-learn's calibration documentation, help compare forecast probability with observed frequency. Sparse bins can be unstable, so a jagged plot without counts is difficult to interpret.
Preserve an honest evaluation window
Record forecast probabilities before outcomes become known. Separate the period used to develop or adjust a model from the period used to evaluate it. For time-ordered research, use earlier information to prepare the method and examine later observations without retroactively tuning the earlier decision.
Keep all records relating to the same fixture on the appropriate side of the evaluation boundary. Different bookmaker rows are not independent match outcomes. Report the event definition, period, unique fixture count, classification threshold, probability scores and calibration-bin counts.
Finally, probability quality and financial returns are different assessments. Prices and costs would need separate consideration. The lesson here is narrower: a headline accuracy percentage can hide meaningful differences in how models express uncertainty.
Sources and method
Calculations and scenarios are original teaching examples, not observed match data, product performance or predictions. Sources support the stated concepts and methods. This article was prepared with AI assistance; sources and calculations were checked.
- scikit-learn: probability calibration — accessed 14 September 2026
- scikit-learn: time series cross-validation — accessed 14 September 2026