Confidence you can trust
When it says 90% sure, is it right 90% of the time?
Only for models that return a confidence. Others show not supported.
For values the model has to estimate, we check whether its 80% range contains the truth about 80% of the time.
Calibration error lower is better
Clairic
Cloudflare
NuExtract3
NuMind
GLiNER2
Fastino
Expected calibration error of the model's confidence: how far its stated confidence is from how often it's right.
80% range coverage should be close to 80%
Clairic
Cloudflare
NuExtract3
NuMind
GLiNER2
Fastino
Share of true values that land inside the model's 80% range. Close to 80% is good; far above or below is not.
Test items and precision
- Items
- Every scored field for calibration; 200 estimate items per test set for the 80% ranges
- Precision
- Calibration is precise. The 80% range check is about ±5.5 points per test set and ±4 pooled, enough to tell 70% from 80% from 90% only when both sets are pooled.