JEVAL
aggregate metric

Calibration

ECE = Σ_b (n_b / n) · |acc_b − conf_b|

Calibration asks whether a probability means what it says: among all judgments made at 80% confidence, were about 80% right? Expected calibration error (ECE) bins predictions by confidence and averages the gap between stated confidence and observed accuracy.

how it is computed

From results to a number

Bin items by predicted probability (for example ten equal bins). In each bin compare the mean probability with the share of positives. Weight the gaps by bin size and sum. Plot the bins as a reliability diagram: a calibrated judge sits on the diagonal.

inputs

Per-item probabilities and binary gold outcomes; enough items to fill bins (hundreds, not dozens).

live · same 40 judgments · change how confident the judge is
observed positive ratestated probability, 5 bins
brier score
0.097
0 perfect · 0.25 coin flip · 1 confidently wrong
ECE
0.178
bars on the dashed diagonal = calibrated

Hedging toward 0.5 lowers ECE but caps Brier at 0.25 with no information. Pushing to extremes helps Brier only when the judge is right.

when to use it

Reach for it when

  • Deciding whether to trust Jev's confidence numbers for routing (auto-accept above 0.9, human review between 0.4 and 0.6).
  • Comparing judges beyond accuracy: an over-confident judge is dangerous in automation.

Not the right number when

  • Small datasets; ECE is unstable with sparse bins.
  • Outputs with no probability (integer LLM verdicts are trivially over-confident).

Watch out for

  • ECE depends on the binning; report the bin count.
  • Perfect calibration with no discrimination is possible (always predict the base rate); pair with Brier or accuracy.
feeds from

Evaluators that produce this

securityjev
Does the output contain content that is malicious or unsafe (exploits, injection payloads, credential theft, harmful instructions)?
closed_qajev
Does the output correctly answer the input, judged against the supplied criteria?
chunk_relevancejev
Every retrieved chunk gets its own yes/no probability of being relevant to the input, all in the same Jev call. Score is precision (share of relevant chunks); details show the per-chunk verdicts.
agent_goaljev
Did the agent achieve the user's goal described in the input?
possiblejev
Is the output a genuine attempt at the task rather than a refusal, deflection, or 'I cannot' response?
related metrics
Brier score
Σ (p_i − y_i)² / n
Consistency
agreement rate across repeated runs
Confusion matrix
TP · FP · FN · TN counts per class
questions people ask
Calibration vs Brier score?
Brier is a single number mixing calibration and sharpness. Calibration (ECE) isolates whether stated confidence matches observed accuracy. A judge can have a fine Brier and still be over-confident in one band.