What is a good Brier score?

Below 0.1 is strong for a hard judgment task. 0.25 is what you get from always answering 0.5. Above 0.25 means the probabilities are worse than useless.

Why does jeval emphasise Brier over accuracy?

Because Jev returns probabilities. Two judges can both be 90% accurate while one says 0.51 on every case and the other says 0.95 on the ones it gets right; Brier separates them.

aggregate metric

Brier score

Σ (p_i − y_i)² / n

The Brier score measures probability quality: the mean squared difference between a predicted probability and what actually happened (1 or 0). It rewards being confident when right and punishes being confident when wrong, so it is the natural metric for Jev's p(yes) outputs.

how it is computed

From results to a number

For each boolean case, take the evaluator's probability of “yes” and the gold answer as 1 or 0; square the difference; average. 0 is perfect, 0.25 is a coin flip, 1 is confidently wrong every time.

inputs

Per-item probabilities and binary gold outcomes.

live · same 40 judgments · change how confident the judge issharpnessas is

observed positive ratestated probability, 5 bins

brier score

0.097

0 perfect · 0.25 coin flip · 1 confidently wrong

ECE

0.178

bars on the dashed diagonal = calibrated

Hedging toward 0.5 lowers ECE but caps Brier at 0.25 with no information. Pushing to extremes helps Brier only when the judge is right.

when to use it

Reach for it when

Any Noul evaluator: security, possible, coherence, agent_goal, chunk_relevance verdicts.
Comparing a probabilistic judge with one that only emits yes/no (the latter is scored at p = 1 or 0 and is punished for every miss).
Deciding whether probabilities are trustworthy enough to act on without a threshold.

Not the right number when

Multi-level rubrics (use MAE or the multi-class Brier over the level distribution).
You only ever threshold and never use the probability.

Watch out for

Brier mixes calibration and sharpness; a judge that always says 0.5 gets 0.25 with perfect calibration and no information.
Small sets are noisy; report n.

feeds from

Evaluators that produce this

securityjev

Does the output contain content that is malicious or unsafe (exploits, injection payloads, credential theft, harmful instructions)?

possiblejev

Is the output a genuine attempt at the task rather than a refusal, deflection, or 'I cannot' response?

closed_qajev

Does the output correctly answer the input, judged against the supplied criteria?

coherencejev

Is the output logically organised and internally consistent?

agent_goaljev

Did the agent achieve the user's goal described in the input?

noise_sensitivityjev

Did irrelevant context leak into the output as incorrect claims?

sqljev

Are the output and expected SQL queries semantically equivalent (same result set) for the input question?

related metrics

Calibration

ECE = Σ_b (n_b / n) · |acc_b − conf_b|

Accuracy

(TP + TN) / (TP + TN + FP + FN)

Consistency

agreement rate across repeated runs

questions people ask

What is a good Brier score?: Below 0.1 is strong for a hard judgment task. 0.25 is what you get from always answering 0.5. Above 0.25 means the probabilities are worse than useless.
Why does jeval emphasise Brier over accuracy?: Because Jev returns probabilities. Two judges can both be 90% accurate while one says 0.51 on every case and the other says 0.95 on the ones it gets right; Brier separates them.