Faithfulness

Faithfulness measures whether every claim in an output is supported by the retrieved context. It is the core RAG hallucination check (RAGAS faithfulness, autoevals Faithfulness, UiPath FaithfulnessEvaluator), answered by Jev as a 0–4 level with a probability per level.

What the evaluator does

Jev treats `context` as the only source of truth and rates how much of `output` can be traced to it, from “most claims unsupported or contradicted” to “every claim traceable and nothing added”. The expected level, its distribution and a pass at 0.75 are returned.

The question, verbatim

This is the question the API sends for the example below, generated from the same code path the playground and API use. Jev sees the request fields as state and returns a probability for each outcome. Nothing is generated, so there is nothing to parse.

What to send

{
  "evaluators": [
    "faithfulness"
  ],
  "input": "What does Jev cost?",
  "output": "Jev costs $42 per million input tokens.",
  "context": [
    "Context length: 64k tokens per request; 32k tokens for `state` plus the longest question.",
    "Rate limits: 250,000 tokens per second / 1,200 requests per minute.",
    "Price: $42 per billion input tokens ($0.042 per million). Output tokens are free."
  ]
}

Aggregate with

One result per item is a fact; a dataset of them is a metric. These are the aggregations that fit this evaluator's output shape.