How should I format the conversation?

A string array, oldest turn first, each prefixed with the speaker: "user: …", "assistant: …". The reply under test goes in output, not in the array.

jevpromptfooscoremirrors promptfoo conversation-relevance

Conversation relevance

Conversation relevance judges whether the latest reply stays relevant to the most recent user turn and consistent with the whole conversation. It mirrors promptfoo's `conversation-relevance` as a 0–3 Jev level over the prior turns in `conversation` and the reply in `output`.

Run this example in the console 1 jev question · answered inside one request

how it works // frame by frame

What the evaluator does

The prior turns are the state; one Score question rates the reply from ignoring the conversation to fully relevant and consistent. Pass at 0.67 of the scale.

frame 1/4 · the state

state

conversationuser: How long do refunds take? → assistant: 5–7 business days after we receive the item. → user: Put that in JSON with a summary field?

output{"summary": "Refunds are issued within 5-7 business days after we receive the returned item."}

question → jev

Rate how relevant `output` is to the latest user turn and how consistent it is with the whole conversation.

ordered levels

ignores the conversation

loosely related, drops context

relevant, mostly consistent

12%

relevant and fully consistent

87%

2.9 / 3conf 87%pass · threshold 67%

exactly what jev is asked

The question, verbatim

This is the question the API sends for the example below, generated from the same code path the playground and API use. Jev sees the request fields as state and returns a probability for each outcome. Nothing is generated, so there is nothing to parse.

qscore · ordered rubric

`conversation` is the prior dialogue, oldest first; `output` is the assistant's latest reply. Rate how relevant `output` is to the most recent user turn and how consistent it is with everything said before.

0The reply ignores the conversation or answers a different question
1The reply is loosely related but drops context from earlier turns
2The reply is relevant to the latest turn and mostly consistent with earlier turns
3The reply is relevant to the latest turn and fully consistent with the whole conversation

when to use it

Reach for it when

Multi-turn assistants where context loss shows up several turns in.
Regression tests on transcripts, judging one reply at a time.

Not the right tool when

Single-turn QA (use answer_relevancy).
Correctness of the reply (use factuality or faithfulness).

Watch out for

Pass turns oldest first and prefix speakers (user:, assistant:) so the roles are unambiguous.

inputs and example

What to send

required fields

outputconversation

optional fields

input

in suites

PromptfooEverything

result shape

score 0–1 (expected level ÷ top level), label “x.x / n”, confidence, a probability per level, passed at 75% of the scale.

POST /api/v1/evaluateopen in console

{
  "evaluators": [
    "conversation_relevance"
  ],
  "conversation": [
    "user: How long do refunds take?",
    "assistant: Refunds are issued within 5–7 business days after we receive the returned item.",
    "user: Can you put that in JSON with a single `summary` field?"
  ],
  "output": "{\"summary\": \"Refunds are issued within 5-7 business days after we receive the returned item.\"}"
}

expected

Level 3: answers the latest turn and keeps the earlier facts.

rolling it up

Aggregate with

One result per item is a fact; a dataset of them is a metric. These are the aggregations that fit this evaluator's output shape.

Mean score

Σ score_i / n

Mean score averages the normalised 0–1 score across items.

Pass rate

passed / total

Pass rate is the share of items whose evaluator result met its threshold.

related evaluators

answer_relevancyjev

How directly and completely the output addresses the input question.

coherencejev

Is the output logically organised and internally consistent?

g_evaljev

G-Eval's 1–5 criteria score without the chain-of-thought: Jev returns the probability of each level directly, and the reported score is the probability-weighted mean G-Eval approximates by sampling.

questions people ask

How should I format the conversation?: A string array, oldest turn first, each prefixed with the speaker: "user: …", "assistant: …". The reply under test goes in output, not in the array.