JEVAL
aggregate metric

Mean score

Σ score_i / n

Mean score averages the normalised 0–1 score across items. Because Jev returns an expected level (probability-weighted), the mean of expected levels is itself a calibrated average and moves smoothly as quality changes.

how it is computed

From results to a number

Average `score` over items. For rubric evaluators, `score` is the expected level divided by the top level; for probability evaluators it is p(good); deterministic evaluators return their own 0–1 value.

inputs

Per-item normalised scores.

live · one judgment's distribution over five levels
02%
15%
213%
330%
450%
integer verdict (LLM judge)
4
steps only when the argmax changes
expected level (Jev) = Σ pᵢ · i
3.21
moves smoothly; the mean of these is a calibrated average
when to use it

Reach for it when

  • Dashboards: one number per evaluator per day.
  • A/B comparisons where a 2-point shift matters and pass rate is too coarse.
  • Ranking many candidates.

Not the right number when

  • Bimodal results (half perfect, half terrible) where the mean hides the split; show the histogram.

Watch out for

  • Means of ordinal levels assume equal spacing between levels; the rubrics are written to make that reasonable but it is an assumption.
  • Report n and a confidence interval on small sets.
feeds from

Evaluators that produce this

faithfulnessjev
Are the claims in the output grounded in the retrieved context?
answer_relevancyjev
How directly and completely the output addresses the input question.
answer_correctnessjev
Factual agreement between the output and the expected answer.
llm_judgejev
Scores the output against a free-text rubric in `criteria`, the way a generic LLM judge would, but as a calibrated score.
translationjev
Quality of the output as a translation of the input, optionally against an expected translation.
json_similaritycode · free
Recursive structural similarity between output JSON and expected JSON (fields, arrays, numbers, strings).
levenshteincode · free
1 − normalised edit distance between output and expected.
related metrics
Pass rate
passed / total
Mean absolute error
Σ |predicted_i − gold_i| / n
Win rate
wins + 0.5 · ties / comparisons
questions people ask
Why is Jev's expected level better to average than an LLM's integer score?
An integer verdict throws away the model's uncertainty. The expected level (Σ p_i × i) keeps it, so a dataset that moves from “mostly 3, some 4” to “mostly 4” shows up as a smooth increase rather than a step.