Is coherence a correctness check?

No. It judges organisation and internal consistency. A well-organised wrong answer is coherent.

jevragasbooleanmirrors ragas.aspect_critic(coherence)

Coherence

Coherence checks whether an output is logically organised and internally consistent. It mirrors the RAGAS aspect critic for coherence and returns Jev's probability that the text holds together.

Run this example in the console 1 jev question · answered inside one request

how it works // frame by frame

What the evaluator does

Jev returns p(yes) for “logically organised, flows well, no internal contradictions” over the output alone.

frame 1/4 · the state

state

outputRefunds take 5–7 days. Refunds are instant. Contact us if the instant refund takes longer than a week.

question → jev

`output` is logically organised, flows well, and contains no internal contradictions.

p(yes)

noyes

incoherentconfidence |p − 0.5| × 2 = 82%the 0.5 line is yours to move

exactly what jev is asked

The question, verbatim

This is the question the API sends for the example below, generated from the same code path the playground and API use. Jev sees the request fields as state and returns a probability for each outcome. Nothing is generated, so there is nothing to parse.

oknoul · p(yes)

`output` is logically organised, flows well, and contains no internal contradictions.

when to use it

Reach for it when

Long-form generation: reports, emails, articles.
Detecting self-contradiction after retrieval merges conflicting sources.

Not the right tool when

Short factual answers where structure is not the question.

Watch out for

Fluent but wrong text is coherent; this is a form check, not a truth check.

inputs and example

What to send

required fields

output

optional fields

none

in suites

CorrectnessEverything

result shape

score 0–1 (p of the good outcome), label “nn% yes”, confidence |p − 0.5| × 2, probabilities yes/no, passed at 0.5.

POST /api/v1/evaluateopen in console

{
  "evaluators": [
    "coherence"
  ],
  "output": "Refunds take 5–7 days. Refunds are instant. Contact us if the instant refund takes longer than a week."
}

expected

Low p(coherent): the text contradicts itself.

rolling it up