JEVAL
jevpromptfoochoicemirrors promptfoo select-best

Select best

Select best compares several candidate responses and picks the one that best satisfies your criteria, returning a probability per candidate. `output` is candidate 0 and `outputs` holds the alternatives. It mirrors promptfoo's `select-best`.

Run this example in the console 1 jev question · answered inside one request
how it works // frame by frame

What the evaluator does

One Jev Choice question over the candidates. The label is the winning index; the details list every candidate with its probability; pass means `output` (candidate 0) won.

frame 1/4 · the state
state
outputRefunds are issued within 5-7 business days after we receive the returned item.
outputs1: Refunds take about a week. · 2: We refund you once the return arrives, typically 5-7 business days.
criteriaprecise timing · item must be received
question → jev

Which candidate best satisfies `criteria`?

options
0
output · 5-7 business days after we receive the item
72%
1
Refunds take about a week.
3%
2
once the return arrives, typically 5-7 business days
25%
0→ score 1conf 72%
exactly what jev is asked

The question, verbatim

This is the question the API sends for the example below, generated from the same code path the playground and API use. Jev sees the request fields as state and returns a probability for each outcome. Nothing is generated, so there is nothing to parse.

bestchoice · one of n

`output` is candidate 0 and `outputs` are candidates 1…n, all responses to `input`. Which candidate best satisfies `criteria`?

  1. 0Candidate 0: Refunds are issued within 5-7 business days after we receive the returned item.
  2. 1Candidate 1: Refunds take about a week.
  3. 2Candidate 2: We refund you once the return arrives, typically 5-7 business days.
when to use it

Reach for it when

  • Ranking prompt or model variants on the same input.
  • Choosing among n sampled responses before showing one to a user.
  • Preference data collection without a human in the loop.

Not the right tool when

  • Absolute quality of one response (use g_eval or llm_judge).
  • Exactly two candidates with a baseline (battle is simpler).

Watch out for

  • Shuffle candidate order across a dataset to average out any position preference.
  • Very long candidates are truncated in the option descriptions; the full texts remain in the state.
inputs and example

What to send

required fields
outputoutputscriteria
optional fields
input
in suites
PromptfooEverything
result shape

score mapped from the chosen option, label = option key, confidence, a probability per option, passed by option.

POST /api/v1/evaluateopen in console 
{
  "evaluators": [
    "select_best"
  ],
  "input": "How long do refunds take?",
  "output": "Refunds are issued within 5-7 business days after we receive the returned item.",
  "outputs": [
    "Refunds take about a week.",
    "We refund you once the return arrives, typically 5-7 business days."
  ],
  "criteria": "Precise timing and the condition that the item must be received."
}
expected

Candidate 0 wins with a clear margin.

rolling it up

Aggregate with

One result per item is a fact; a dataset of them is a metric. These are the aggregations that fit this evaluator's output shape.

Win rate
wins + 0.5 · ties / comparisons

Win rate is the share of pairwise comparisons a candidate wins against a baseline, counting ties as half.

Mean score
Σ score_i / n

Mean score averages the normalised 0–1 score across items.

related evaluators
battlejev
Head-to-head: is the output a better response to the input than the expected (baseline) response?
summaryjev
Which summary of the input text is better: the output or the expected summary?
g_evaljev
G-Eval's 1–5 criteria score without the chain-of-thought: Jev returns the probability of each level directly, and the reported score is the probability-weighted mean G-Eval approximates by sampling.
questions people ask
What does select_best return?
The index of the best candidate as the label, the probability of that choice as the score, a probability for every candidate, and a details row per candidate.