How is Closed QA different from LLM-as-judge?

Closed QA is binary: meets the criteria or not, as a probability. LLM-as-judge returns a graded 0–4 level over the same kind of rubric.

jevautoevalsbooleanmirrors autoevals.ClosedQA

Closed QA

Closed QA asks a single yes/no question: does the output correctly answer the input when judged strictly against the supplied criteria? It mirrors autoevals ClosedQA and returns Jev's probability that the answer meets the criteria.

Run this example in the console 1 jev question · answered inside one request

how it works // frame by frame

What the evaluator does

Jev reads input, output and criteria and returns p(yes). jeval applies a 0.5 threshold for pass/fail and reports |p − 0.5| × 2 as confidence. The criteria text is the rubric, so it should say what “correct” means for this question.

frame 1/4 · the state

state

inputI was charged twice, can you refund?

outputSorry — I see two charges and refunded one.

criteriaapologise · confirm duplicate · refund + timeline · no blame

question → jev

`output` answers `input` correctly, judged strictly against the rubric in `criteria`.

p(yes)

22%

noyes

fails criteriaconfidence |p − 0.5| × 2 = 56%the 0.5 line is yours to move

exactly what jev is asked

The question, verbatim

This is the question the API sends for the example below, generated from the same code path the playground and API use. Jev sees the request fields as state and returns a probability for each outcome. Nothing is generated, so there is nothing to parse.

oknoul · p(yes)

`output` answers the question in `input` correctly, judged strictly against the rubric in `criteria`.

when to use it

Reach for it when

Grading answers where correctness has a crisp definition you can write down.
Support, QA and tutoring bots with a checklist per question.
Gating a release on a pass rate over a golden set.

Not the right tool when

Correctness is a matter of degree (use llm_judge, which returns a 0–4 level).
You have a reference answer and want the relationship type (use factuality).
No criteria can be written; the judgment would be arbitrary.

Watch out for

Vague criteria produce probabilities near 0.5; write them as testable clauses.
Criteria that contradict the input confuse the judgment; keep them about the answer, not the question.

inputs and example

What to send

required fields

inputoutputcriteria

optional fields

none

in suites

CorrectnessEverything

result shape

score 0–1 (p of the good outcome), label “nn% yes”, confidence |p − 0.5| × 2, probabilities yes/no, passed at 0.5.

POST /api/v1/evaluateopen in console

{
  "evaluators": [
    "closed_qa"
  ],
  "input": "Customer: I was charged twice. Can you refund the duplicate?",
  "output": "Sorry about that. I can see two charges and I've refunded one of them.",
  "criteria": "Must apologise, confirm the duplicate was found, state a refund was issued with a timeline, and not blame the customer."
}

expected

Fails: the timeline clause is missing, so p(yes) is low.

rolling it up

Aggregate with

One result per item is a fact; a dataset of them is a metric. These are the aggregations that fit this evaluator's output shape.

Pass rate

passed / total

Pass rate is the share of items whose evaluator result met its threshold.

Precision, recall and F1

P = TP/(TP+FP) · R = TP/(TP+FN) · F1 = 2PR/(P+R)

Precision is the share of predicted positives that were really positive; recall is the share of real positives the evaluator caught; F1 is their harmonic mean, which is only high when both are.

Brier score

Σ (p_i − y_i)² / n

The Brier score measures probability quality: the mean squared difference between a predicted probability and what actually happened (1 or 0).

related evaluators

llm_judgejev

Scores the output against a free-text rubric in `criteria`, the way a generic LLM judge would, but as a calibrated score.

factualityjev

Compares the output to the expected answer and classifies the relationship (subset, superset, equivalent, disagreement, stylistic).

answer_correctnessjev

Factual agreement between the output and the expected answer.

questions people ask

How is Closed QA different from LLM-as-judge?: Closed QA is binary: meets the criteria or not, as a probability. LLM-as-judge returns a graded 0–4 level over the same kind of rubric.