How do I read a win rate of 0.5?

The candidate and baseline are indistinguishable on this dataset. Above 0.5 favours the candidate, below favours the baseline.

aggregate metric

Win rate

wins + 0.5 · ties / comparisons

Win rate is the share of pairwise comparisons a candidate wins against a baseline, counting ties as half. It is how Battle and Summary results aggregate, and how A/B tests between prompts or models are usually reported.

how it is computed

From results to a number

Each Battle result is 1 (candidate wins), 0 (baseline wins) or 0.5 (tie). Average them. Swap positions across the dataset to cancel position bias, then average both directions.

inputs

Per-item pairwise outcomes.

when to use it

Reach for it when

Comparing two prompts, two models or two RAG configurations directly.
Preference-style evaluation where absolute scores are hard to define.

Not the right number when

You need an absolute quality level for a single system.
The two systems answer different inputs.

Watch out for

Position bias: always evaluate both orderings.
A 52% win rate on 50 items is noise; compute a confidence interval.

feeds from

Evaluators that produce this

battlejev

Head-to-head: is the output a better response to the input than the expected (baseline) response?

summaryjev

Which summary of the input text is better: the output or the expected summary?

related metrics

questions people ask

How do I read a win rate of 0.5?: The candidate and baseline are indistinguishable on this dataset. Above 0.5 favours the candidate, below favours the baseline.