Pass rate
passed / total
Pass rate is the share of items whose evaluator result met its threshold. It is the number most teams gate releases on, and the one most sensitive to where you set the threshold.
From results to a number
Every jeval result carries a `passed` boolean (0.75 of the scale for rubrics, 0.5 for probabilities, evaluator-specific for deterministic scores). Count them. Batch responses return this per evaluator in `aggregate`.
inputs
Per-item `passed` booleans.
Reach for it when
- Release gates and CI checks.
- Trend lines over time for a fixed dataset.
- Comparing prompt or model versions at a fixed threshold.
Not the right number when
- Understanding why something failed (look at the level distribution or the details).
- Small datasets, where one item moves the rate by several points.
Watch out for
- A pass rate without the threshold is meaningless; jeval's default thresholds are listed on each evaluator page.
- Pass rate throws away the probability; two datasets at 80% can have very different Brier scores.
Evaluators that produce this
faithfulnessjev
Are the claims in the output grounded in the retrieved context?llm_judgejev
Scores the output against a free-text rubric in `criteria`, the way a generic LLM judge would, but as a calibrated score.closed_qajev
Does the output correctly answer the input, judged against the supplied criteria?valid_jsoncode · free
Output parses as JSON.containscode · free
Output contains the expected string (case-insensitive).agent_goaljev
Did the agent achieve the user's goal described in the input?- What thresholds does jeval use for passed?
- Rubric (Score) evaluators pass at 75% of the scale, probability (Noul) evaluators at 0.5, and deterministic evaluators at a value chosen per evaluator, such as 0.9 for JSON similarity and 0.8 for Levenshtein.