How does the jeval benchmark define accuracy?

Choice and boolean cases: exact agreement with the gold label. Score cases: within one rubric level of the gold level.

aggregate metric

Accuracy

(TP + TN) / (TP + TN + FP + FN)

Accuracy is the share of items the evaluator got right. For labels it is exact agreement; for graded rubrics jeval's benchmark counts a judgment correct when it lands within one level of the gold level.

how it is computed

From results to a number

Count correct items, divide by total. For Score evaluators, define “correct” with a tolerance (the benchmark uses ±1 level on 0–4 scales and exact on 0–1).

inputs

Per-item decisions or levels plus gold labels.

live · 40 judgments · move the thresholdthreshold0.50

p = 0p = 1

gold positive, flagged gold positive, missed gold negative, flagged gold negative, correctly ignored

predicted positivepredicted negativegold positive

TP16hit

FN5miss

gold negative

FP1false alarm

TN18correct rejection

precision0.94

TP / (TP + FP)

recall0.76

TP / (TP + FN)

F10.84

2PR / (P + R)

accuracy0.85

(TP + TN) / n

when to use it

Reach for it when

Balanced datasets where every error costs the same.
Headline number in a benchmark, always alongside the matrix.
Comparing judges on the same gold set (the jeval benchmark's primary metric).

Not the right number when

Imbalanced data: 97% accuracy on a 3% positive class is the trivial classifier.
Ranking or probability quality (use Brier or calibration).

Watch out for

Accuracy hides direction; a judge that always says “faithful” scores well on mostly-faithful data.
With tolerance-based accuracy, report the tolerance.

feeds from

Evaluators that produce this

classificationcode · free

Predicted label in output equals the expected label.

exact_matchcode · free

Output equals expected after whitespace and case normalisation.

factualityjev

Compares the output to the expected answer and classifies the relationship (subset, superset, equivalent, disagreement, stylistic).

answer_correctnessjev

Factual agreement between the output and the expected answer.

sqljev

Are the output and expected SQL queries semantically equivalent (same result set) for the input question?

related metrics

Confusion matrix

TP · FP · FN · TN counts per class

Precision, recall and F1

P = TP/(TP+FP) · R = TP/(TP+FN) · F1 = 2PR/(P+R)

Mean absolute error

Σ |predicted_i − gold_i| / n

Cohen's kappa

κ = (p_o − p_e) / (1 − p_e)

questions people ask

How does the jeval benchmark define accuracy?: Choice and boolean cases: exact agreement with the gold label. Score cases: within one rubric level of the gold level.