Does Translation need a reference?

No. The reference is optional and used as a guide when present.

jevautoevalsscoremirrors autoevals.Translation

Translation

Translation rates an output as a translation of the input on a 0–4 scale covering accuracy, naturalness and register, optionally against a reference translation. It mirrors autoevals Translation.

Run this example in the console 1 jev question · answered inside one request

how it works // frame by frame

What the evaluator does

A Jev Score question with five ordered levels; the expected level and the distribution are returned. Pass threshold is 0.75 of the scale.

frame 1/4 · the state

state

inputLe modèle renvoie des probabilités calibrées, pas du texte.

outputThe model returns calibrated probabilities, not text.

question → jev

Rate `output` as a translation of `input`.

ordered levels

wrong or unintelligible

meaning partly preserved, significant errors

meaning preserved, awkward or minor errors

accurate, natural phrasing

27%

accurate, natural, faithful to tone

70%

3.7 / 4conf 70%pass · threshold 75%

exactly what jev is asked

The question, verbatim

This is the question the API sends for the example below, generated from the same code path the playground and API use. Jev sees the request fields as state and returns a probability for each outcome. Nothing is generated, so there is nothing to parse.

qscore · ordered rubric

Rate `output` as a translation of `input`. If `expected` is present, use it as a reference translation.

0The translation is wrong or unintelligible
1The meaning is partly preserved with significant errors
2The meaning is preserved with awkward phrasing or minor errors
3Accurate translation with natural phrasing
4Accurate, natural, and faithful to tone and register

when to use it

Reach for it when

Evaluating MT or LLM translation quality at scale without bilingual reviewers for every item.
Comparing translation prompts across languages.

Not the right tool when

Strictly terminology-controlled domains where a glossary decides correctness (add the glossary to criteria and use llm_judge).

Watch out for

English is Jev's strongest language; test on your language pair before relying on absolute levels, and compare relative scores first.

inputs and example

What to send

required fields

inputoutput

optional fields

expected

in suites

CorrectnessEverything

result shape

score 0–1 (expected level ÷ top level), label “x.x / n”, confidence, a probability per level, passed at 75% of the scale.

POST /api/v1/evaluateopen in console

{
  "evaluators": [
    "translation"
  ],
  "input": "Le modèle renvoie des probabilités calibrées, pas du texte.",
  "output": "The model returns calibrated probabilities, not text.",
  "expected": "The model returns calibrated probabilities rather than text."
}

expected

Level 4: accurate and natural.

rolling it up