JEVAL
jevragasscoremirrors ragas.answer_relevancy / autoevals.AnswerRelevancy

Answer relevancy

Answer relevancy rates how directly and completely an output addresses the question asked, penalising padding, drift and non-answers. It mirrors RAGAS answer_relevancy and autoevals AnswerRelevancy as a 0–3 Jev level.

Run this example in the console 1 jev question · answered inside one request
how it works // frame by frame

What the evaluator does

Jev reads `input` and `output` and returns a level from “does not address the question” to “directly and completely, no padding”.

frame 1/4 · the state
state
inputMinimum Node version for the SDK?
outputNode 20 or newer. As background, the SDK ships ESM and CJS builds, includes TS declarations, many teams use pnpm…
question → jev

Rate how directly and completely `output` answers `input`, penalising padding and off-topic content.

ordered levels
0
does not address the question
0%
1
partially related, misses the point
6%
2
addresses it with redundant content
66%
3
direct and complete, no padding
28%
2.2 / 3conf 66%fail · threshold 75%
exactly what jev is asked

The question, verbatim

This is the question the API sends for the example below, generated from the same code path the playground and API use. Jev sees the request fields as state and returns a probability for each outcome. Nothing is generated, so there is nothing to parse.

qscore · ordered rubric

Rate how directly and completely `output` answers the question in `input`, penalising padding and off-topic content.

  1. 0The output does not address the input question at all
  2. 1The output is partially related but misses the main point or is mostly filler
  3. 2The output addresses the question with some redundant or off-topic content
  4. 3The output directly and completely addresses the question with no padding
when to use it

Reach for it when

  • Chat and QA systems where answers wander or hedge.
  • Catching filler and “great question!” openers at scale.
  • Alongside faithfulness: an answer can be grounded yet off-topic.

Not the right tool when

  • Correctness matters more than focus (use answer_correctness).
  • The input is a task, not a question (use task_completion).

Watch out for

  • A concise wrong answer scores high on relevancy; it is not a correctness measure.
  • Multi-part questions need all parts answered to reach the top level.
inputs and example

What to send

required fields
inputoutput
optional fields
none
in suites
RAGCorrectnessEverything
result shape

score 0–1 (expected level ÷ top level), label “x.x / n”, confidence, a probability per level, passed at 75% of the scale.

POST /api/v1/evaluateopen in console 
{
  "evaluators": [
    "answer_relevancy"
  ],
  "input": "What is the minimum Node version for the SDK?",
  "output": "Node 20 or newer. As background, the SDK ships ESM and CommonJS builds, includes TypeScript declarations, and many teams also use pnpm."
}
expected

Level 2: answers, then pads.

rolling it up

Aggregate with

One result per item is a fact; a dataset of them is a metric. These are the aggregations that fit this evaluator's output shape.

Mean score
Σ score_i / n

Mean score averages the normalised 0–1 score across items.

Pass rate
passed / total

Pass rate is the share of items whose evaluator result met its threshold.

related evaluators
faithfulnessjev
Are the claims in the output grounded in the retrieved context?
concisenessjev
Does the output avoid unnecessary length and repetition?
possiblejev
Is the output a genuine attempt at the task rather than a refusal, deflection, or 'I cannot' response?
answer_correctnessjev
Factual agreement between the output and the expected answer.
questions people ask
Does answer relevancy check facts?
No. It checks focus and completeness relative to the question. Use answer_correctness or faithfulness for facts.