Chunk relevance
Chunk relevance gives every retrieved context chunk its own yes/no probability of being relevant to the question, all inside one Jev request. The score is precision (relevant chunks over total) and the details list the per-chunk verdicts, which is what RAGAS context_precision computes with one LLM call per chunk.
What the evaluator does
For N chunks jeval adds N Noul questions to the same request: “`context[i]` helps answer `input`, judged on its own”. Each returns p(relevant). Precision, a confidence and the per-chunk list come back together.
`context[i]` contains information that helps answer `input`. Judge this chunk on its own.
Price: $42 per billion input tokens. Output tokens are free.
Rate limits: 250,000 tokens per second / 1,200 requests per minute.
The company was founded in 2024 and is headquartered in San Francisco.
Output tokens are billed at $8 per million on the legacy v0 endpoint, which is deprecated.
The question, verbatim
This is the question the API sends for the example below, generated from the same code path the playground and API use. Jev sees the request fields as state and returns a probability for each outcome. Nothing is generated, so there is nothing to parse.
`context[0]` contains information that helps answer `input` (or supports `expected` if present). Judge this chunk on its own.
`context[1]` contains information that helps answer `input` (or supports `expected` if present). Judge this chunk on its own.
`context[2]` contains information that helps answer `input` (or supports `expected` if present). Judge this chunk on its own.
`context[3]` contains information that helps answer `input` (or supports `expected` if present). Judge this chunk on its own.
Reach for it when
- Debugging retrieval: which chunks were noise?
- Reranker evaluation: precision@k per query.
- Any time you would loop an LLM over chunks.
Not the right tool when
- Whole-document context with no chunk boundaries (use context_relevancy).
Watch out for
- Chunks that are relevant only in combination are judged individually and may score low; that is usually the right signal for a reranker.
- Keep chunk count reasonable (dozens, not hundreds) to stay within Jev's 32k state-plus-question budget.
What to send
score 0–1 (p of the good outcome), label “nn% yes”, confidence |p − 0.5| × 2, probabilities yes/no, passed at 0.5. Plus a details array with one row per chunk.
{
"evaluators": [
"chunk_relevance"
],
"input": "How is Jev priced, and are output tokens billed?",
"output": "…",
"context": [
"Price: $42 per billion input tokens. Output tokens are free.",
"Rate limits: 1,200 rpm.",
"Founded in 2024.",
"Output billed at $8/M on the legacy v0 endpoint, deprecated."
]
}Chunks 1 and 4 relevant, 2 and 3 not: precision 2/4.
Aggregate with
One result per item is a fact; a dataset of them is a metric. These are the aggregations that fit this evaluator's output shape.
Precision is the share of predicted positives that were really positive; recall is the share of real positives the evaluator caught; F1 is their harmonic mean, which is only high when both are.
Mean score averages the normalised 0–1 score across items.
- How many Jev calls does chunk relevance make?
- One. Every chunk's question is packed into the same request, so six chunks cost roughly the tokens of the state plus six short questions.