Every evaluator, explained frame by frame.

Faithfulness measures whether every claim in an output is supported by the retrieved context.

Answer relevancy rates how directly and completely an output addresses the question asked, penalising padding, drift and non-answers.

Answer correctness rates the factual agreement between an output and a reference answer on a 0–4 scale.

Answer similarity rates the semantic similarity between an output and a reference on a 0–3 scale, ignoring wording.

Context precision rates what share of the retrieved context is actually relevant to the question, as a 0–3 level.

Context recall rates how much of the reference answer can be attributed to the retrieved context, as a 0–3 level.

Context relevancy rates the overall relevance of the retrieved context to the question on a 0–3 scale, without needing a reference answer.

Context entity recall rates what share of the named entities, numbers, dates and terms in the reference answer appear in the retrieved context.

Noise sensitivity detects whether irrelevant or misleading context leaked into the output as incorrect claims.

Chunk relevance gives every retrieved context chunk its own yes/no probability of being relevant to the question, all inside one Jev request.

Chunk support asks, for each retrieved chunk, whether it supports the output's claims, contradicts them, or says nothing about them.

Chunk relevance gives every retrieved context chunk its own yes/no probability of being relevant to the question, all inside one Jev request.

Chunk support asks, for each retrieved chunk, whether it supports the output's claims, contradicts them, or says nothing about them.

Factuality compares a submitted answer with an expert (expected) answer and classifies their relationship as subset, superset, equivalent, disagreement, or a difference that does not affect the facts.

Closed QA asks a single yes/no question: does the output correctly answer the input when judged strictly against the supplied criteria? It mirrors autoevals ClosedQA and returns Jev's probability that the answer meets the criteria..

Possible checks whether the output is a genuine attempt at the task rather than a refusal, deflection or “I can't do that”.

Battle is a head-to-head comparison: given the input, is the candidate output a better response than the baseline (expected) response? It mirrors autoevals Battle and returns a three-way choice with probabilities..

Summary judges which of two summaries of a source text is better: the candidate output or the reference.

SQL judges whether two queries are semantically equivalent, meaning they return the same result set for the request, ignoring aliasing, formatting and column order.

Translation rates an output as a translation of the input on a 0–4 scale covering accuracy, naturalness and register, optionally against a reference translation.

Answer relevancy rates how directly and completely an output addresses the question asked, penalising padding, drift and non-answers.

Answer correctness rates the factual agreement between an output and a reference answer on a 0–4 scale.

Answer similarity rates the semantic similarity between an output and a reference on a 0–3 scale, ignoring wording.

Coherence checks whether an output is logically organised and internally consistent.

Conciseness checks whether an output conveys its content without unnecessary length, repetition or filler.

LLM-as-judge with a rubric scores an output 0–4 against free-text criteria, the way UiPath's LLMJudgeOutputEvaluator and autoevals LLMClassifier do, but as a calibrated Jev distribution over five ordered levels instead of a generated verdict..

Security flags outputs that contain malicious or dangerous content: prompt-injection payloads, credential exfiltration, working exploit code or instructions that enable serious harm.

Moderation classifies an output into one content category: none, harassment, hate, sexual, violence, self-harm or dangerous.

Harmfulness estimates whether an output could cause harm to individuals, groups or society if acted upon or distributed.

Maliciousness estimates whether an output is intended to deceive, manipulate, exploit or damage the reader or a third party.

Topic adherence checks whether an output stays within the topic or domain described in the criteria, without wandering into out-of-scope subjects.

Is refusal judges whether the output declines the task rather than attempting it, returning the probability of a refusal.

Topic adherence checks whether an output stays within the topic or domain described in the criteria, without wandering into out-of-scope subjects.

LLM-as-judge with a rubric scores an output 0–4 against free-text criteria, the way UiPath's LLMJudgeOutputEvaluator and autoevals LLMClassifier do, but as a calibrated Jev distribution over five ordered levels instead of a generated verdict..

Trajectory rates how closely an agent's step sequence follows the expected trajectory, as a single 0–3 level.

Tool call accuracy rates how well an agent's tool calls match the expected calls in tool choice and arguments, allowing semantically equivalent values, as a 0–3 level.

Agent goal accuracy asks whether the agent achieved the user's goal as stated in the input, judging from the final output and optionally the trajectory.

Task completion rates how completely an output accomplishes the task requested in the input on a 0–3 scale, using the expected answer as the definition of done when present.

Trajectory steps re-expresses UiPath's trajectory judge as one probability per expected step (“did the agent accomplish this step?”) plus an order judgment, all in one Jev request.

Tool calls (semantic) gives each expected tool call a probability that an equivalent call was made: same tool, and argument values that mean the same thing even if keys are renamed or values reformatted.

Is refusal judges whether the output declines the task rather than attempting it, returning the probability of a refusal.

G-Eval scores an output 1–5 against custom criteria.

Select best compares several candidate responses and picks the one that best satisfies your criteria, returning a probability per candidate.

Classifier assigns the output one of the labels you pass in `values`, such as tones, intents or toxicity classes, with a probability per label.

Conversation relevance judges whether the latest reply stays relevant to the most recent user turn and consistent with the whole conversation.

Humor rates how funny an output is on a 0–3 scale.