why jev // decisions, not prose

Why Jev

Evals are decisions. Stop paying for prose. An LLM judge generates a paragraph and then a number. You pay for every token of the paragraph, wait for it, parse it, and get a point estimate with no calibration. Jev never generates text. It reads the state once and returns a probability distribution for every question you asked, in one request. That is what an evaluator should have been all along.

What a Jev answer looks like

No rationale tokens, no parsing, thresholds live in your code

Scorefaithfulness

01234

3.6 / 4, confidence 0.68

Yes / nochunk_relevance[3]

yesno

8% relevant

Choicechunk_support[5]

supportscontradictsunrelated

contradicts

Trajectory evaluation

uipath.eval LLMJudgeTrajectoryEvaluator

With an LLM judge

llm prompt

One prompt: expected trajectory + actual trajectory + rubric.
Model writes a paragraph, then a number 0–100.
Rationale is prose; the number is whatever the model typed.
Rerun → different paragraph, often a different number.

With Jev

trajectory_steps

One Jev call. One yes/no question per expected step: “does the trajectory accomplish this step?”
Plus one question for order.
Each step gets a calibrated probability; score = share performed × order factor.
Same input → same distribution. Threshold it in code.

Trajectory evaluation: per-item numbers, LLM judge against Jev
Per item	LLM judge	Jev
calls	1 LLM completion	1 Jev request
output	about 150 tokens of prose	5 probabilities
latency	about 1.9 s	≈ 150 ms
cost	about $0.003	≈ $0.00003

Tool-call evaluation

uipath.eval ToolCallArgsEvaluator · ToolCallOutputEvaluator

With an LLM judge

exact match

Argument dicts compared literally.
`date: "2025-11-12"` vs `date: "Nov 12, 2025"` → fail.
`from/to` vs `origin/destination` → fail.
So teams fall back to an LLM judge prompt per call, and pay for it.

With Jev

tool_call_semantic

One yes/no question per expected call: “was an equivalent call made?”
Equivalence covers renamed keys and reformatted values.
All expected calls judged in the same request as the trajectory and goal checks.
Keep the exact-match evaluators too; they are free and run alongside.

Tool-call evaluation: per-item numbers, LLM judge against Jev
Per item	LLM judge	Jev
calls	n LLM completions	1 Jev request
signal	pass / fail	probability per call
latency	n × 1.9 s	≈ 150 ms total
cost	n × $0.003	≈ $0.00003 total

Context grounding

ragas context_precision · faithfulness / autoevals ContextPrecision

With an LLM judge

llm per chunk

RAGAS loops the LLM over every chunk, then over every claim.
Six chunks and four claims ≈ ten completions.
Latency stacks; cost stacks; rate limits bite on batches.
The 0/1 verdicts hide how sure the model was.

With Jev

chunk_relevance + chunk_support

Every chunk gets its own relevance probability and a supports / contradicts / unrelated distribution.
All of it in the same request as faithfulness, precision and recall.
Contradicting chunks are the hallucination signal, per chunk, not a paragraph.
A 100-item batch is 100 requests, not a thousand.

Context grounding: per-item numbers, LLM judge against Jev
Per item	LLM judge	Jev
calls	about 10 per item	1 per item
output	10 verdicts	6 + 6 distributions
latency	about 19 s / item	≈ 250 ms / item
cost	about $0.03 / item	≈ $0.00005 / item

Rubric judge

uipath.eval LLMJudgeOutputEvaluator · autoevals LLMClassifier

With an LLM judge

llm-as-judge

Prompt: output + criteria + “score 1–5 and explain”.
Chain-of-thought output you pay for and then throw away.
Position bias, verbosity bias, self-preference.
Calibration requires a second pass.

With Jev

llm_judge (score)

Same criteria, expressed as five ordered levels.
Returns a probability per level and a confidence.
No generated text; nothing to parse.
Add closed_qa, coherence, conciseness in the same call at ~zero marginal latency.

Rubric judge: per-item numbers, LLM judge against Jev
Per item	LLM judge	Jev
calls	1 per evaluator	1 for all evaluators
output	prose + number	distribution over levels
latency	about 1.9 s each	≈ 160 ms total
cost	about $0.003 each	≈ $0.00003 total

Latency and cost figures are typical values from this deployment (Jev, measured) and from a GPT-4.1-class judge at list price with ~600 input and ~150 output tokens per call (estimated). Your prompts will differ; the ratios will not by much.