JEVAL
jevautoevalschoicemirrors autoevals.Summary

Summary

Summary judges which of two summaries of a source text is better: the candidate output or the reference. It mirrors autoevals Summary and is a Battle specialised for summarisation.

Run this example in the console 1 jev question · answered inside one request
how it works // frame by frame

What the evaluator does

Jev reads the source (`input`), the two summaries, and picks the more accurate, complete and concise one, or a tie.

frame 1/4 · the state
state
input(source text about Jev batching)
outputReads state once; answers all questions in parallel; batching is cheaper and faster.
expectedProcesses questions in parallel over a single state read.
question → jev

`input` is a source. `output` and `expected` are two summaries. Which is more accurate, complete and concise?

options
output
candidate summary
58%
expected
reference summary
22%
tie
equally good
20%
output→ score 1conf 58%
exactly what jev is asked

The question, verbatim

This is the question the API sends for the example below, generated from the same code path the playground and API use. Jev sees the request fields as state and returns a probability for each outcome. Nothing is generated, so there is nothing to parse.

winchoice · one of n

`input` is a source text. `output` and `expected` are two summaries of it. Which summary is more accurate, complete, and concise?

  1. output`output` is the better summary
  2. expected`expected` is the better summary
  3. tieEqually good
when to use it

Reach for it when

  • Comparing summarisation prompts or models against a reference summary.
  • Checking that a shorter summary did not lose the important point.

Not the right tool when

  • No reference summary exists (use faithfulness against the source as context plus conciseness).
  • Summaries of different sources.

Watch out for

  • Longer summaries are not better; the rubric weighs concision, but keep candidates comparable in length when you can.
inputs and example

What to send

required fields
inputoutputexpected
optional fields
none
in suites
CorrectnessEverything
result shape

score mapped from the chosen option, label = option key, confidence, a probability per option, passed by option.

POST /api/v1/evaluateopen in console 
{
  "evaluators": [
    "summary"
  ],
  "input": "Jev ingests the state once and evaluates every question against it in parallel...",
  "output": "Jev reads the state once and answers all questions in parallel, so batching is cheaper and faster.",
  "expected": "Jev processes questions in parallel over a single state read."
}
expected

Close call; the candidate adds the practical consequence, so it edges ahead.

rolling it up

Aggregate with

One result per item is a fact; a dataset of them is a metric. These are the aggregations that fit this evaluator's output shape.

Win rate
wins + 0.5 · ties / comparisons

Win rate is the share of pairwise comparisons a candidate wins against a baseline, counting ties as half.

Mean score
Σ score_i / n

Mean score averages the normalised 0–1 score across items.

related evaluators
battlejev
Head-to-head: is the output a better response to the input than the expected (baseline) response?
faithfulnessjev
Are the claims in the output grounded in the retrieved context?
concisenessjev
Does the output avoid unnecessary length and repetition?
questions people ask
Can Summary run without a reference?
No, it is a comparison. For reference-free summary quality use faithfulness (grounding in the source) with conciseness.