JEVAL
jevragasscoremirrors ragas.answer_similarity / autoevals.AnswerSimilarity

Answer similarity

Answer similarity rates the semantic similarity between an output and a reference on a 0–3 scale, ignoring wording. It mirrors RAGAS answer_similarity and autoevals AnswerSimilarity without needing an embedding model.

Run this example in the console 1 jev question · answered inside one request
how it works // frame by frame

What the evaluator does

A Jev Score question from “mean different things” to “semantically equivalent”. Because it is a judgment rather than a cosine distance, paraphrases with opposite meaning score low.

frame 1/4 · the state
state
outputThe model returns calibrated probabilities, not text.
expectedThe model returns calibrated probabilities rather than text.
question → jev

Rate the semantic similarity between `output` and `expected`, ignoring wording differences.

ordered levels
0
mean different things
0%
1
overlap in topic, differ in meaning
1%
2
largely the same, different emphasis
6%
3
semantically equivalent
93%
2.9 / 3conf 93%pass · threshold 75%
exactly what jev is asked

The question, verbatim

This is the question the API sends for the example below, generated from the same code path the playground and API use. Jev sees the request fields as state and returns a probability for each outcome. Nothing is generated, so there is nothing to parse.

qscore · ordered rubric

Rate the semantic similarity between `output` and `expected`, ignoring wording differences.

  1. 0The output and expected answer mean different things
  2. 1They overlap in topic but differ in key meaning
  3. 2They mean largely the same thing with different emphasis or detail
  4. 3They are semantically equivalent
when to use it

Reach for it when

  • Paraphrase detection and near-duplicate checks.
  • As a lenient companion to answer_correctness when style varies a lot.

Not the right tool when

  • Facts must be right, not just similar (use answer_correctness).
  • You already have embeddings and want a pure geometric similarity.

Watch out for

  • Similar topic, different claim: the rubric's level 1 exists for exactly this, but do not use similarity as a correctness proxy.
inputs and example

What to send

required fields
outputexpected
optional fields
none
in suites
RAGCorrectnessEverything
result shape

score 0–1 (expected level ÷ top level), label “x.x / n”, confidence, a probability per level, passed at 75% of the scale.

POST /api/v1/evaluateopen in console 
{
  "evaluators": [
    "answer_similarity"
  ],
  "output": "The model returns calibrated probabilities, not text.",
  "expected": "The model returns calibrated probabilities rather than text."
}
expected

Level 3: semantically equivalent.

rolling it up

Aggregate with

One result per item is a fact; a dataset of them is a metric. These are the aggregations that fit this evaluator's output shape.

Mean score
Σ score_i / n

Mean score averages the normalised 0–1 score across items.

Mean absolute error
Σ |predicted_i − gold_i| / n

Mean absolute error measures, for graded rubrics, how far the evaluator's level is from the gold level on average.

related evaluators
answer_correctnessjev
Factual agreement between the output and the expected answer.
factualityjev
Compares the output to the expected answer and classifies the relationship (subset, superset, equivalent, disagreement, stylistic).
levenshteincode · free
1 − normalised edit distance between output and expected.
questions people ask
Is this the same as embedding cosine similarity?
No. It is a semantic judgment: two sentences about the same topic with opposite claims score low here and could score high on cosine similarity.