JEVAL
code · freepromptfoodeterministicmirrors promptfoo bleu

BLEU

BLEU scores the output against a reference with sentence-level BLEU-4: geometric mean of 1- to 4-gram precision with add-one smoothing and a brevity penalty. It mirrors promptfoo's `bleu` with its 0.5 default threshold.

Run this example in the console runs in code · $0
how it works // frame by frame

What the evaluator does

Clipped n-gram precisions for n = 1..4 are smoothed, averaged in log space and multiplied by a brevity penalty when the output is shorter than the reference.

frame 1/4 · the two inputs
output
the model returns calibrated probabilities not text
expected
the model returns calibrated probabilities rather than text
1.1–4-gram clipped precision
2.add-one smoothing
3.geometric mean × brevity penalty
0.61score 0.61
runs in code · $0 · ~0 ms · deterministic
when to use it

Reach for it when

  • Translation and templated generation where wording should track a reference closely.
  • Regression tests with a single reference per item.

Not the right tool when

  • Open-ended answers with many valid phrasings (BLEU is low and noisy there).

Watch out for

  • Single-reference sentence BLEU is harsh; do not compare absolute values with corpus BLEU from papers.
inputs and example

What to send

required fields
outputexpected
optional fields
none
in suites
PromptfooEverything
result shape

score 0–1 computed in code, label, passed at an evaluator-specific threshold, no probabilities.

POST /api/v1/evaluateopen in console 
{
  "evaluators": [
    "bleu"
  ],
  "output": "The model returns calibrated probabilities, not text.",
  "expected": "The model returns calibrated probabilities rather than text."
}
expected

Most n-grams shared: BLEU near 0.6.

rolling it up

Aggregate with

One result per item is a fact; a dataset of them is a metric. These are the aggregations that fit this evaluator's output shape.

Mean score
Σ score_i / n

Mean score averages the normalised 0–1 score across items.

Pass rate
passed / total

Pass rate is the share of items whose evaluator result met its threshold.

related evaluators
gleucode · free
Sentence GLEU (min of n-gram precision and recall, n = 1..4) against expected. Passes at 0.5. Empty output scores 0.
rouge_ncode · free
ROUGE-1 F1 between output and expected (ROUGE-2 reported alongside). Passes at 0.75, promptfoo's default.
translationjev
Quality of the output as a translation of the input, optionally against an expected translation.
questions people ask
Why is my BLEU low for a correct answer?
BLEU measures wording, not meaning. A correct paraphrase shares few 3- and 4-grams. Use it only where wording should match.