JEVAL
jevragasscoremirrors ragas.context_relevancy / autoevals.ContextRelevancy

Context relevancy

Context relevancy rates the overall relevance of the retrieved context to the question on a 0–3 scale, without needing a reference answer. It mirrors RAGAS context_relevancy and autoevals ContextRelevancy.

Run this example in the console 1 jev question · answered inside one request
how it works // frame by frame

What the evaluator does

A single Jev Score over `input` and `context`. It is coarser than chunk_relevance but works when you have no gold answer.

frame 1/4 · the state
state
inputHow do I list available models?
contextGET /v1/models returns the names your account can send in the model field.
question → jev

Rate how relevant `context` as a whole is to `input`.

ordered levels
0
unrelated
0%
1
loosely related
1%
2
relevant, some extraneous material
8%
3
tightly relevant
91%
2.9 / 3conf 91%pass · threshold 75%
exactly what jev is asked

The question, verbatim

This is the question the API sends for the example below, generated from the same code path the playground and API use. Jev sees the request fields as state and returns a probability for each outcome. Nothing is generated, so there is nothing to parse.

qscore · ordered rubric

Rate how relevant the `context` as a whole is to the question in `input`.

  1. 0The context is unrelated to the input
  2. 1The context is loosely related with little that helps answer the input
  3. 2The context is relevant with some extraneous material
  4. 3The context is tightly relevant to the input
when to use it

Reach for it when

  • Quick retrieval sanity checks in production, where no reference exists.
  • Alerting on drift when relevancy drops over time.

Not the right tool when

  • You want per-chunk decisions (use chunk_relevance).
  • You have a reference answer (context_recall is more informative).

Watch out for

  • A single highly relevant chunk among noise can pull the level up; the rubric weighs the whole, so use chunk_relevance for precision.
inputs and example

What to send

required fields
inputcontext
optional fields
none
in suites
RAGEverything
result shape

score 0–1 (expected level ÷ top level), label “x.x / n”, confidence, a probability per level, passed at 75% of the scale.

POST /api/v1/evaluateopen in console 
{
  "evaluators": [
    "context_relevancy"
  ],
  "input": "How do I list available models?",
  "context": [
    "`GET /v1/models` returns the names your account can send in the `model` field."
  ],
  "output": "…"
}
expected

Level 3: tightly relevant.

rolling it up

Aggregate with

One result per item is a fact; a dataset of them is a metric. These are the aggregations that fit this evaluator's output shape.

Mean score
Σ score_i / n

Mean score averages the normalised 0–1 score across items.

related evaluators
context_precisionjev
What share of the retrieved context is actually relevant to the input.
chunk_relevancejev
Every retrieved chunk gets its own yes/no probability of being relevant to the input, all in the same Jev call. Score is precision (share of relevant chunks); details show the per-chunk verdicts.
answer_relevancyjev
How directly and completely the output addresses the input question.
questions people ask
When should I use context relevancy instead of precision?
When you have no reference answer and want a single retrieval-quality number. Precision needs the same inputs but is defined per chunk; chunk_relevance gives you that breakdown.