JEVAL
jevautoevalschoicemirrors autoevals.Factuality

Factuality

Factuality compares a submitted answer with an expert (expected) answer and classifies their relationship as subset, superset, equivalent, disagreement, or a difference that does not affect the facts. It is the autoevals Factuality evaluator, answered by Jev as a five-way choice with a probability per option.

Run this example in the console 1 jev question · answered inside one request
how it works // frame by frame

What the evaluator does

Jev sees the input question, the output and the expected answer, and picks one of five options A–E. The option maps to a score the same way autoevals does (A 0.4, B 0.6, C 1.0, D 0, E 1.0). Because it is a Choice, you also get the full probability distribution, so a 55/45 split between “subset” and “equivalent” is visible instead of hidden behind a single letter.

frame 1/4 · the state
state
inputWhat are Jev's rate limits?
output1,200 requests per minute.
expected250,000 tokens/s and 1,200 requests/min.
question → jev

Compare the factual content of `output` with `expected`. Ignore style. Select the option that best describes the relationship.

options
A
output is a subset of expected, consistent
86%
B
output is a superset of expected, consistent
2%
C
same details
5%
D
disagreement
4%
E
differ, but not in facts
3%
A→ score 0.4conf 86%
exactly what jev is asked

The question, verbatim

This is the question the API sends for the example below, generated from the same code path the playground and API use. Jev sees the request fields as state and returns a probability for each outcome. Nothing is generated, so there is nothing to parse.

relchoice · one of n

Compare the factual content of `output` (the submitted answer) with `expected` (the expert answer) for the question in `input`. Ignore differences in style, grammar, or punctuation. Select the option that best describes the relationship.

  1. AThe submitted answer is a subset of the expert answer and is fully consistent with it.
  2. BThe submitted answer is a superset of the expert answer and is fully consistent with it.
  3. CThe submitted answer contains all the same details as the expert answer.
  4. DThere is a disagreement between the submitted answer and the expert answer.
  5. EThe answers differ, but these differences don't matter from the perspective of factuality.
when to use it

Reach for it when

  • You have a reference answer and want to know whether the model's answer agrees with it, not whether it is worded the same.
  • Regression testing an assistant against a golden set where partial answers should score differently from wrong ones.
  • You need to separate “said less than the reference” from “said something different”.

Not the right tool when

  • No reference answer exists (use answer_relevancy or llm_judge with criteria).
  • The reference is a long document rather than an answer (use faithfulness with the document as context).
  • You need exact-string agreement (use exact_match or levenshtein, which are free).

Watch out for

  • Option E (“differ but doesn't matter”) can absorb subtle numeric changes if the question does not make precision matter; state units and precision in the input.
  • Very long expected answers make “subset” the default; keep references tight.
  • A superset can still contain an error; pair with faithfulness when the output adds detail.
inputs and example

What to send

required fields
outputexpected
optional fields
input
in suites
CorrectnessEverything
result shape

score mapped from the chosen option, label = option key, confidence, a probability per option, passed by option.

POST /api/v1/evaluateopen in console 
{
  "evaluators": [
    "factuality"
  ],
  "input": "What are Jev's rate limits?",
  "output": "1,200 requests per minute.",
  "expected": "250,000 tokens per second and 1,200 requests per minute."
}
expected

Option A (subset): the output is consistent with the reference but omits the token-rate limit. Score 0.4.

rolling it up

Aggregate with

One result per item is a fact; a dataset of them is a metric. These are the aggregations that fit this evaluator's output shape.

Accuracy
(TP + TN) / (TP + TN + FP + FN)

Accuracy is the share of items the evaluator got right.

Confusion matrix
TP · FP · FN · TN counts per class

A confusion matrix counts, for a binary or multiclass check, how often each true label was predicted as each label: true positives, false positives, false negatives and true negatives.

Mean score
Σ score_i / n

Mean score averages the normalised 0–1 score across items.

related evaluators
answer_correctnessjev
Factual agreement between the output and the expected answer.
answer_similarityjev
Semantic similarity between the output and the expected answer.
closed_qajev
Does the output correctly answer the input, judged against the supplied criteria?
exact_matchcode · free
Output equals expected after whitespace and case normalisation.
questions people ask
What does Factuality return?
One of five options (A–E) with a probability for each, a confidence, and a mapped score: A 0.4, B 0.6, C 1, D 0, E 1.
Is jeval's Factuality the same as autoevals Factuality?
Same options, same score weights and same intent. The difference is the answer comes from Jev as a calibrated distribution in one request instead of an LLM completion.