Mean absolute error
Σ |predicted_i − gold_i| / n
Mean absolute error measures, for graded rubrics, how far the evaluator's level is from the gold level on average. A judge that is often off by one level has MAE near 1; a judge that is right or badly wrong can have the same accuracy but a worse MAE.
From results to a number
For each Score case, take |expected level − gold level|; average. The jeval benchmark reports it in rubric-level units per judge.
inputs
Per-item predicted levels (or expected levels) and gold levels on the same scale.
live · 12 rubric cases on a 0–4 scale
gold level judge's expected levelMAE 0.28 levelsaccuracy ±1 100%
Reach for it when
- Comparing judges on 0–4 rubrics: it rewards being close, unlike accuracy.
- Detecting systematic bias (always one level generous) when combined with the signed mean error.
Not the right number when
- Categorical outputs with no order (use accuracy or kappa).
- Probabilities (use Brier).
Watch out for
- Only meaningful when both sides use the same scale; do not compare MAE across 0–3 and 0–4 rubrics.
- Outliers matter less than in RMSE; if catastrophic misses matter, report both.
Evaluators that produce this
faithfulnessjev
Are the claims in the output grounded in the retrieved context?answer_correctnessjev
Factual agreement between the output and the expected answer.llm_judgejev
Scores the output against a free-text rubric in `criteria`, the way a generic LLM judge would, but as a calibrated score.translationjev
Quality of the output as a translation of the input, optionally against an expected translation.tool_call_accuracyjev
Do the agent's tool calls match the expected calls in tool choice and arguments, allowing semantically equivalent values?trajectoryjev
How closely the agent's step sequence follows the expected trajectory.- Is a lower MAE always better?
- Yes. Zero means every judgment landed on the gold level. The benchmark shows MAE next to accuracy because two judges with the same accuracy can differ a lot in how far off they are when wrong.