JEVAL
jevragasbooleanmirrors ragas.noise_sensitivity

Noise sensitivity

Noise sensitivity detects whether irrelevant or misleading context leaked into the output as incorrect claims. It mirrors RAGAS noise_sensitivity and returns the probability that leakage happened.

Run this example in the console 1 jev question · answered inside one request
how it works // frame by frame

What the evaluator does

Jev returns p(yes) for “the output contains claims that come from irrelevant parts of the context and are not needed to answer the question”. jeval inverts it: higher score means less leakage.

frame 1/4 · the state
state
inputWhat is Jev's context limit?
output64k per request; and Jev costs $8/M output tokens.
context64k limit · legacy v0 endpoint billed $8/M output (deprecated)
question → jev

`output` contains claims that come from irrelevant or misleading parts of `context` and are not needed to answer `input`.

p(yes)
88%
noyes
noise leakedconfidence |p − 0.5| × 2 = 76%the 0.5 line is yours to move
exactly what jev is asked

The question, verbatim

This is the question the API sends for the example below, generated from the same code path the playground and API use. Jev sees the request fields as state and returns a probability for each outcome. Nothing is generated, so there is nothing to parse.

leaknoul · p(yes)

`output` contains claims that come from irrelevant or misleading parts of `context` and are not needed to answer `input` (or contradict `expected`).

when to use it

Reach for it when

  • Stress-testing RAG with distractor documents.
  • Measuring robustness of a generator to noisy retrieval.

Not the right tool when

  • The context has no irrelevant material by construction; the signal will be trivially clean.

Watch out for

  • Leakage of correct-but-unneeded facts is still noise sensitivity; decide whether your product cares before thresholding hard.
inputs and example

What to send

required fields
outputcontext
optional fields
inputexpected
in suites
RAGEverything
result shape

score 0–1 (p of the good outcome), label “nn% yes”, confidence |p − 0.5| × 2, probabilities yes/no, passed at 0.5.

POST /api/v1/evaluateopen in console 
{
  "evaluators": [
    "noise_sensitivity"
  ],
  "input": "What is Jev's context limit?",
  "output": "64k tokens per request; and Jev costs $8 per million output tokens.",
  "context": [
    "Context length: 64k tokens per request; 32k tokens for `state` plus the longest question.",
    "Output tokens are billed at $8 per million on the legacy v0 endpoint, which is deprecated."
  ]
}
expected

p(leak) high: the deprecated pricing line leaked into an unrelated answer.

rolling it up

Aggregate with

One result per item is a fact; a dataset of them is a metric. These are the aggregations that fit this evaluator's output shape.

Pass rate
passed / total

Pass rate is the share of items whose evaluator result met its threshold.

Brier score
Σ (p_i − y_i)² / n

The Brier score measures probability quality: the mean squared difference between a predicted probability and what actually happened (1 or 0).

related evaluators
faithfulnessjev
Are the claims in the output grounded in the retrieved context?
chunk_supportjev
For each chunk: does it support the output, contradict it, or say nothing about it? Contradictions are the hallucination signal; score is supporting share minus contradictions.
context_precisionjev
What share of the retrieved context is actually relevant to the input.
questions people ask
What does a high noise sensitivity mean?
The generator repeated irrelevant or misleading context as if it answered the question. It is a robustness problem in the generator, not a retrieval problem.