LLM-as-judge (rubric)
LLM-as-judge with a rubric scores an output 0–4 against free-text criteria, the way UiPath's LLMJudgeOutputEvaluator and autoevals LLMClassifier do, but as a calibrated Jev distribution over five ordered levels instead of a generated verdict.
What the evaluator does
Your criteria become the rubric; Jev returns a probability per level and the expected level. Nothing is generated, so there is nothing to parse and no rationale tokens to pay for.
Judge `output` strictly against the rubric in `criteria`.
The question, verbatim
This is the question the API sends for the example below, generated from the same code path the playground and API use. Jev sees the request fields as state and returns a probability for each outcome. Nothing is generated, so there is nothing to parse.
Judge `output` (a response to `input`, with `expected` as an optional reference) strictly against the rubric in `criteria`.
- 0The output fails the criteria completely
- 1The output meets a small part of the criteria
- 2The output meets the criteria with notable gaps
- 3The output meets the criteria with minor issues
- 4The output fully meets the criteria
Reach for it when
- Any custom quality bar you can write down.
- Replacing prompt-and-parse judge code with a typed call.
- Pairing with closed_qa (binary) for a pass gate plus a graded score.
Not the right tool when
- You have a reference answer (factuality or answer_correctness are sharper).
- The criteria need external facts the state does not contain.
Watch out for
- Rubrics that mix several dimensions produce middling levels; split them into separate evaluators when you need to know which failed.
- Hard rules (no exclamation marks, one paragraph) are better as their own Noul questions than buried in a rubric.
What to send
score 0–1 (expected level ÷ top level), label “x.x / n”, confidence, a probability per level, passed at 75% of the scale.
{
"evaluators": [
"llm_judge"
],
"input": "Write a one-paragraph product description for a stainless water bottle.",
"output": "Stay refreshed all day! Cold for 24 hours, hot for 12, leak-proof and dishwasher safe. Grab yours now!",
"criteria": "One paragraph, mentions insulation, mentions leak-proofing, mentions a convenience feature, no exclamation marks."
}Level 2: content complete, hard rule violated.
Aggregate with
One result per item is a fact; a dataset of them is a metric. These are the aggregations that fit this evaluator's output shape.
Mean score averages the normalised 0–1 score across items.
Mean absolute error measures, for graded rubrics, how far the evaluator's level is from the gold level on average.
Pass rate is the share of items whose evaluator result met its threshold.
- What is LLM-as-judge?
- Using a model to grade another model's output against a rubric. Traditionally a prompt that returns prose and a number; in jeval it is a Jev Score question that returns a probability per rubric level.
- Why is a Jev rubric judge cheaper than an LLM judge?
- Jev does not generate text. You pay for input tokens only, and every rubric in your suite runs inside the same request.