How is jeval's G-Eval different from the paper?

The paper generates chain-of-thought steps and averages sampled 1–5 scores weighted by token probability. Jev is a decision model that returns the level distribution directly, so the weighted score is exact and there is nothing to sample or parse.

jevpromptfooscoremirrors promptfoo g-eval (Liu et al. 2023)

G-Eval

G-Eval scores an output 1–5 against custom criteria. The original method prompts an LLM for evaluation steps and averages sampled scores by their token probabilities; Jev returns the probability of each level directly, so the same probability-weighted score comes from one question with no generation.

Run this example in the console 1 jev question · answered inside one request

how it works // frame by frame

What the evaluator does

A five-level Score question asks Jev to consider the criteria step by step and rate the output. The label reports the expected level on the 1–5 scale; pass at 75% of the scale.

frame 1/4 · the state

state

outputRefunds are issued within 5-7 business days after we receive the returned item.

criteriaone sentence · 5–7 business days · item received first

question → jev

Consider the criteria step by step, then rate how well `output` satisfies them on the 1–5 scale.

ordered levels

1 · fails almost entirely

2 · meets a small part

3 · notable gaps

4 · minor issues

20%

5 · fully meets

75%

3.7 / 4conf 75%pass · threshold 75%

exactly what jev is asked

The question, verbatim

This is the question the API sends for the example below, generated from the same code path the playground and API use. Jev sees the request fields as state and returns a probability for each outcome. Nothing is generated, so there is nothing to parse.

qscore · ordered rubric

You are evaluating `output` (a response to `input`; `expected` is an optional reference). Consider the evaluation criteria in `criteria` step by step — what would a response have to do to satisfy each part — then rate how well `output` satisfies them on the 1–5 scale.

01 — fails the criteria almost entirely
12 — meets a small part of the criteria
23 — meets the criteria with notable gaps
34 — meets the criteria with minor issues
45 — fully meets the criteria

when to use it

Reach for it when

Custom quality criteria for summaries, replies or generated content.
Teams already using G-Eval who want the same scale without the cost and variance of sampling.

Not the right tool when

A reference answer exists (use answer_correctness or factuality).
Binary requirements (use closed_qa).

Watch out for

Multi-dimensional criteria blur the score; run one G-Eval per dimension when you need to know which failed.

inputs and example

What to send

required fields

outputcriteria

optional fields

inputexpected

in suites

PromptfooEverything

result shape

score 0–1 (expected level ÷ top level), label “x.x / n”, confidence, a probability per level, passed at 75% of the scale.

POST /api/v1/evaluateopen in console

{
  "evaluators": [
    "g_eval"
  ],
  "input": "Summarise the refund policy in one sentence.",
  "output": "Refunds are issued within 5-7 business days after we receive the returned item.",
  "criteria": "One sentence, states the 5–7 business day window, mentions that the item must be received first."
}

expected

Level 5: fully meets the criteria.

rolling it up

Aggregate with

One result per item is a fact; a dataset of them is a metric. These are the aggregations that fit this evaluator's output shape.

Mean score

Σ score_i / n

Mean score averages the normalised 0–1 score across items.

Mean absolute error

Σ |predicted_i − gold_i| / n

Mean absolute error measures, for graded rubrics, how far the evaluator's level is from the gold level on average.

Pass rate

passed / total

Pass rate is the share of items whose evaluator result met its threshold.

related evaluators

llm_judgejev

Scores the output against a free-text rubric in `criteria`, the way a generic LLM judge would, but as a calibrated score.

closed_qajev

Does the output correctly answer the input, judged against the supplied criteria?

conversation_relevancejev

Does the latest reply (`output`) stay relevant to the last turn and consistent with the whole `conversation`?

questions people ask

How is jeval's G-Eval different from the paper?: The paper generates chain-of-thought steps and averages sampled 1–5 scores weighted by token probability. Jev is a decision model that returns the level distribution directly, so the weighted score is exact and there is nothing to sample or parse.