JEVAL
aggregate metric

Cohen's kappa

κ = (p_o − p_e) / (1 − p_e)

Cohen's kappa measures agreement between two raters (an evaluator and a human, or two evaluators) after subtracting the agreement you would expect by chance. It is the standard way to report how well an automated judge agrees with human labels.

how it is computed

From results to a number

p_o is observed agreement (the accuracy of one rater against the other). p_e is chance agreement computed from each rater's label frequencies. κ of 1 is perfect, 0 is chance, negative is worse than chance. For ordered levels use weighted kappa so near-misses count partially.

inputs

Two label sets over the same items.

live · two raters, same items
chance agreement p_e
0.50
p² + (1−p)²
raw agreement
0.85
what “accuracy vs human” would report
kappa
0.70
(p_o − p_e) / (1 − p_e)

Push prevalence toward 0.95: chance agreement climbs, and the same raw agreement yields a much smaller kappa. That is why agreement alone flatters imbalanced datasets.

when to use it

Reach for it when

  • Validating an evaluator against human annotation before trusting it.
  • Reporting inter-annotator agreement on the gold set itself.
  • Imbalanced label sets where raw agreement is inflated.

Not the right number when

  • Continuous scores (use correlation or MAE).
  • More than two raters (use Fleiss' kappa or Krippendorff's alpha).

Watch out for

  • Kappa is sensitive to prevalence; two datasets with the same accuracy can have very different kappa.
  • Conventions (0.6 “substantial”, 0.8 “almost perfect”) are rough; report the value, not the adjective.
feeds from

Evaluators that produce this

classificationcode · free
Predicted label in output equals the expected label.
moderationjev
Classifies the output into a moderation category (none, harassment, hate, sexual, violence, self-harm, dangerous).
factualityjev
Compares the output to the expected answer and classifies the relationship (subset, superset, equivalent, disagreement, stylistic).
closed_qajev
Does the output correctly answer the input, judged against the supplied criteria?
related metrics
Accuracy
(TP + TN) / (TP + TN + FP + FN)
Confusion matrix
TP · FP · FN · TN counts per class
Consistency
agreement rate across repeated runs
questions people ask
Why not just report agreement?
Because two raters who both label 95% of items “safe” agree 90% of the time by chance alone. Kappa tells you how much agreement is beyond that.