GLEU replaces the geometric mean and brevity penalty with min(precision, recall) over pooled n-grams, so it is symmetric and smoother on single sentences.

code · freepromptfoodeterministicmirrors promptfoo gleu

GLEU

GLEU is Google's sentence-level variant of BLEU: the minimum of n-gram precision and recall over pooled 1- to 4-grams, better behaved on single sentences. It mirrors promptfoo's `gleu` with its 0.5 threshold.

Run this example in the console runs in code · $0

how it works // frame by frame

What the evaluator does

All 1..4-grams are pooled; overlap ÷ output n-grams is precision, overlap ÷ reference n-grams is recall; the score is the smaller of the two.

frame 1/4 · the two inputs

output

the model returns calibrated probabilities not text

expected

the model returns calibrated probabilities rather than text

1.pool 1–4-grams

2.overlap ÷ output = precision

3.overlap ÷ reference = recall

4.take the minimum

0.60score 0.6

runs in code · $0 · ~0 ms · deterministic

when to use it

Reach for it when

Sentence-level comparisons where BLEU's brevity penalty and geometric mean behave badly.
Reinforcement-style reward signals that need a smooth metric.

Not the right tool when

Semantic evaluation (use a Jev judgment).

Watch out for

Empty output scores 0 by definition.

inputs and example

What to send

required fields

outputexpected

optional fields

none

in suites

PromptfooEverything

result shape

score 0–1 computed in code, label, passed at an evaluator-specific threshold, no probabilities.

POST /api/v1/evaluateopen in console

{
  "evaluators": [
    "gleu"
  ],
  "output": "The model returns calibrated probabilities, not text.",
  "expected": "The model returns calibrated probabilities rather than text."
}

expected

GLEU around 0.6.

rolling it up

Aggregate with

One result per item is a fact; a dataset of them is a metric. These are the aggregations that fit this evaluator's output shape.

Mean score

Σ score_i / n

Mean score averages the normalised 0–1 score across items.

Pass rate

passed / total

Pass rate is the share of items whose evaluator result met its threshold.

related evaluators

bleucode · free

Sentence BLEU-4 with add-one smoothing and brevity penalty against expected. Passes at 0.5, promptfoo's default.

rouge_ncode · free

ROUGE-1 F1 between output and expected (ROUGE-2 reported alongside). Passes at 0.75, promptfoo's default.

questions people ask

GLEU vs BLEU?: GLEU replaces the geometric mean and brevity penalty with min(precision, recall) over pooled n-grams, so it is symmetric and smoother on single sentences.