Calibrate confidence so 70% starts to mean something

The short answer

Calibration means that events you predict at 70% happen roughly seven times out of ten across comparable decisions. Record explicit probabilities before outcomes, define resolution criteria and periodically compare predicted confidence with actual frequency.

Run a simple calibration loop

1. 1. Make a resolvable forecast State what will happen, by when and exactly how the result will be judged.
2. 2. Assign probability Use a number before the outcome is known. Avoid changing the original estimate.
3. 3. Record the basis Name the strongest evidence, base rate and assumption behind the estimate.
4. 4. Resolve consistently Score the outcome using the predefined rule, including failures and forgotten forecasts.
5. 5. Compare buckets Group estimates such as 50-59%, 60-69% and 70-79%, then compare confidence with observed success.

Look for directional bias

If 80% forecasts happen only half the time, you may be overconfident, using weak base rates or defining success too generously. If 60% forecasts almost always happen, you may be underconfident or avoiding uncertain predictions.

Calibration improves slowly. Do not infer a stable bias from a handful of unrelated forecasts. Compare similar decision types and keep uncertainty visible.

Questions

How many forecasts are needed?

More is better, but begin with recurring decisions and avoid conclusions from very small mixed samples.

What probability should I use when unsure?

Use 50% when the outcomes are genuinely balanced, then state which missing evidence would move you.

Does a wrong forecast mean a bad decision?

No. Evaluate the reasoning, calibration and outcome separately.