Answer correctness
Answer correctness rates the factual agreement between an output and a reference answer on a 0–4 scale. It mirrors RAGAS answer_correctness and autoevals AnswerCorrectness and is the graded counterpart to Factuality's five-way classification.
What the evaluator does
Jev compares `output` with `expected` for the question in `input` and returns a level from “contradicts or entirely wrong” to “fully correct and equivalent”.
Rate how factually correct `output` is compared with the reference `expected` answer to `input`.
The question, verbatim
This is the question the API sends for the example below, generated from the same code path the playground and API use. Jev sees the request fields as state and returns a probability for each outcome. Nothing is generated, so there is nothing to parse.
Rate how factually correct `output` is compared with the reference `expected` answer to `input`.
- 0The output contradicts the expected answer or is entirely wrong
- 1The output is mostly wrong with a few correct elements
- 2The output is partially correct but omits or garbles important parts
- 3The output matches the expected answer with minor omissions or extra detail
- 4The output is fully correct and equivalent to the expected answer
Reach for it when
- Golden-set regression where partial credit matters.
- Numeric and date answers where being close is not being right (the rubric treats them as wrong).
- Reporting a mean correctness score over a dataset.
Not the right tool when
- You need the type of disagreement (use factuality).
- There is no reference (use closed_qa with criteria or llm_judge).
Watch out for
- Checking arithmetic or date answers requires computing them; this is a category where reasoning models can beat a decision model, and the benchmark measures it.
- Make the reference the answer, not a paragraph about the answer.
What to send
score 0–1 (expected level ÷ top level), label “x.x / n”, confidence, a probability per level, passed at 75% of the scale.
{
"evaluators": [
"answer_correctness"
],
"input": "A key was created on 2026-09-22 with a 90-day expiry. When does it expire?",
"output": "November 21, 2026.",
"expected": "2026-12-21"
}Level 0: off by a month.
Aggregate with
One result per item is a fact; a dataset of them is a metric. These are the aggregations that fit this evaluator's output shape.
Mean score averages the normalised 0–1 score across items.
Mean absolute error measures, for graded rubrics, how far the evaluator's level is from the gold level on average.
Pass rate is the share of items whose evaluator result met its threshold.
- Correctness vs factuality: which should I use?
- Use answer_correctness for a graded score you can average; use factuality when you need to know whether the answer was a subset, superset or contradiction.