How do I use Battle for A/B tests?

Put version A in output and version B in expected for half the dataset, swap for the other half, then average the score. A mean above 0.5 means A wins.

jevautoevalschoicemirrors autoevals.Battle

Battle

Battle is a head-to-head comparison: given the input, is the candidate output a better response than the baseline (expected) response? It mirrors autoevals Battle and returns a three-way choice with probabilities.

Run this example in the console 1 jev question · answered inside one request

how it works // frame by frame

What the evaluator does

Jev compares `output` and `expected` as two responses to `input` and picks output / expected / tie. Score is 1, 0 or 0.5.

frame 1/4 · the state

state

inputExplain a Brier score in one sentence.

outputMean squared difference between predicted probabilities and outcomes; lower is better.

expectedIt measures how good probabilities are.

question → jev

Two responses to `input`: `output` (candidate) and `expected` (baseline). Which is better overall?

options

output

candidate is better

84%

expected

baseline is better

tie

equally good

11%

output→ score 1conf 84%

exactly what jev is asked

The question, verbatim

This is the question the API sends for the example below, generated from the same code path the playground and API use. Jev sees the request fields as state and returns a probability for each outcome. Nothing is generated, so there is nothing to parse.

winchoice · one of n

Two responses to `input`: `output` (candidate) and `expected` (baseline). Which is the better response overall in accuracy, completeness, and helpfulness?

outputThe candidate `output` is better
expectedThe baseline `expected` is better
tieThey are equally good

when to use it

Reach for it when

A/B testing two prompts or two models on the same inputs.
Preference data collection without a human in the loop.
Deciding whether a rewrite is an improvement.

Not the right tool when

You need an absolute quality score rather than a relative one (use llm_judge).
The two responses answer different questions.

Watch out for

Position bias is a classic LLM-judge failure; Jev is not generating text, but you should still randomise which side is `output` across a dataset.
Ties are informative: a high tie probability means the change did not matter.

inputs and example

What to send

required fields

inputoutputexpected

optional fields

none

in suites

CorrectnessEverything

result shape

score mapped from the chosen option, label = option key, confidence, a probability per option, passed by option.

POST /api/v1/evaluateopen in console

{
  "evaluators": [
    "battle"
  ],
  "input": "Explain what a Brier score is in one sentence.",
  "output": "The Brier score is the mean squared difference between predicted probabilities and what actually happened; lower is better.",
  "expected": "It measures how good probabilities are."
}

expected

output wins: precise, complete, still one sentence.

rolling it up

Aggregate with

One result per item is a fact; a dataset of them is a metric. These are the aggregations that fit this evaluator's output shape.

Win rate

wins + 0.5 · ties / comparisons

Win rate is the share of pairwise comparisons a candidate wins against a baseline, counting ties as half.

Mean score

Σ score_i / n

Mean score averages the normalised 0–1 score across items.

related evaluators

summaryjev

Which summary of the input text is better: the output or the expected summary?

llm_judgejev

Scores the output against a free-text rubric in `criteria`, the way a generic LLM judge would, but as a calibrated score.

answer_relevancyjev

How directly and completely the output addresses the input question.

questions people ask

How do I use Battle for A/B tests?: Put version A in output and version B in expected for half the dataset, swap for the other half, then average the score. A mean above 0.5 means A wins.