One result is a fact. A thousand is a metric.
Evaluators judge one item at a time. These are the numbers you roll them into, what each one hides, and how Jev's probabilities change which ones you can use.
A confusion matrix counts, for a binary or multiclass check, how often each true label was predicted as each label: true positives, false positives, false negatives and true negatives. Every classification metric on this site is a ratio of its cells.
Accuracy is the share of items the evaluator got right. For labels it is exact agreement; for graded rubrics jeval's benchmark counts a judgment correct when it lands within one level of the gold level.
Precision is the share of predicted positives that were really positive; recall is the share of real positives the evaluator caught; F1 is their harmonic mean, which is only high when both are. Together they describe a detector better than accuracy does.
Pass rate is the share of items whose evaluator result met its threshold. It is the number most teams gate releases on, and the one most sensitive to where you set the threshold.
Mean score averages the normalised 0–1 score across items. Because Jev returns an expected level (probability-weighted), the mean of expected levels is itself a calibrated average and moves smoothly as quality changes.
Mean absolute error measures, for graded rubrics, how far the evaluator's level is from the gold level on average. A judge that is often off by one level has MAE near 1; a judge that is right or badly wrong can have the same accuracy but a worse MAE.
The Brier score measures probability quality: the mean squared difference between a predicted probability and what actually happened (1 or 0). It rewards being confident when right and punishes being confident when wrong, so it is the natural metric for Jev's p(yes) outputs.
Calibration asks whether a probability means what it says: among all judgments made at 80% confidence, were about 80% right? Expected calibration error (ECE) bins predictions by confidence and averages the gap between stated confidence and observed accuracy.
Consistency is how often an evaluator gives the same verdict when asked the same question again. Sampling-based LLM judges drift between runs; a decision model like Jev returns the same distribution for the same input, which the jeval benchmark measures with repeated passes.
Cohen's kappa measures agreement between two raters (an evaluator and a human, or two evaluators) after subtracting the agreement you would expect by chance. It is the standard way to report how well an automated judge agrees with human labels.
Win rate is the share of pairwise comparisons a candidate wins against a baseline, counting ties as half. It is how Battle and Summary results aggregate, and how A/B tests between prompts or models are usually reported.