Consistency
agreement rate across repeated runs
Consistency is how often an evaluator gives the same verdict when asked the same question again. Sampling-based LLM judges drift between runs; a decision model like Jev returns the same distribution for the same input, which the jeval benchmark measures with repeated passes.
From results to a number
Run each case k times. For each case, the modal verdict's share of the k runs is its agreement; average over cases. The benchmark also reports the standard deviation of the expected level for rubric cases.
inputs
Repeated evaluator results for identical inputs.
Reach for it when
- Before using an evaluator as a CI gate: flaky judges make flaky builds.
- Comparing LLM judges at temperature 0 (still not deterministic) with Jev.
- Estimating how many repeats you need for a stable mean.
Not the right number when
- Inputs change between runs; that is drift, not inconsistency.
Watch out for
- High consistency is not correctness; a judge can be reliably wrong.
- Provider-side model updates change behaviour across days; pin model versions (Jev aliases resolve to a fixed version).
Evaluators that produce this
faithfulnessjev
Are the claims in the output grounded in the retrieved context?llm_judgejev
Scores the output against a free-text rubric in `criteria`, the way a generic LLM judge would, but as a calibrated score.securityjev
Does the output contain content that is malicious or unsafe (exploits, injection payloads, credential theft, harmful instructions)?factualityjev
Compares the output to the expected answer and classifies the relationship (subset, superset, equivalent, disagreement, stylistic).tool_call_accuracyjev
Do the agent's tool calls match the expected calls in tool choice and arguments, allowing semantically equivalent values?- Why is consistency a benchmark metric?
- Because an evaluator that changes its mind between runs cannot gate a release. The jeval benchmark repeats each case and reports agreement per judge; Jev's repeat consistency has measured at 98%.