One catalogue, three lineages.
Each evaluator keeps the name and meaning of its source library. 33 are Jev judgments: a Score, Noul or Choice question over the same state, all answered in one request; the ones marked per chunk / step / tool call ask one question per unit. The rest run in code and cost nothing. All return the same result shape. Click an id for a frame-by-frame explainer, or read the explained index and aggregate metrics.
| id | answered by | family | kind | mirrors | requires | description |
|---|---|---|---|---|---|---|
| factuality Factuality | jev | autoevals | choice | autoevals.Factuality | outputexpectedinput | Compares the output to the expected answer and classifies the relationship (subset, superset, equivalent, disagreement, stylistic). |
| closed_qa Closed QA | jev | autoevals | boolean | autoevals.ClosedQA | inputoutputcriteria | Does the output correctly answer the input, judged against the supplied criteria? |
| possible Possible | jev | autoevals | boolean | autoevals.Possible | inputoutput | Is the output a genuine attempt at the task rather than a refusal, deflection, or 'I cannot' response? |
| security Security | jev | autoevals | boolean | autoevals.Security | outputinput | Does the output contain content that is malicious or unsafe (exploits, injection payloads, credential theft, harmful instructions)? |
| moderation Moderation | jev | autoevals | choice | autoevals.Moderation | output | Classifies the output into a moderation category (none, harassment, hate, sexual, violence, self-harm, dangerous). |
| battle Battle | jev | autoevals | choice | autoevals.Battle | inputoutputexpected | Head-to-head: is the output a better response to the input than the expected (baseline) response? |
| summary Summary | jev | autoevals | choice | autoevals.Summary | inputoutputexpected | Which summary of the input text is better: the output or the expected summary? |
| sql SQL | jev | autoevals | boolean | autoevals.Sql | outputexpectedinput | Are the output and expected SQL queries semantically equivalent (same result set) for the input question? |
| translation Translation | jev | autoevals | score | autoevals.Translation | inputoutputexpected | Quality of the output as a translation of the input, optionally against an expected translation. |
| humor Humor | jev | autoevals | score | autoevals.Humor | outputinput | How funny is the output? |
| faithfulness Faithfulness | jev | ragas | score | ragas.faithfulness / autoevals.Faithfulness / uipath LegacyFaithfulnessEvaluator | outputcontextinput | Are the claims in the output grounded in the retrieved context? |
| answer_relevancy Answer relevancy | jev | ragas | score | ragas.answer_relevancy / autoevals.AnswerRelevancy | inputoutput | How directly and completely the output addresses the input question. |
| answer_correctness Answer correctness | jev | ragas | score | ragas.answer_correctness / autoevals.AnswerCorrectness | outputexpectedinput | Factual agreement between the output and the expected answer. |
| answer_similarity Answer similarity | jev | ragas | score | ragas.answer_similarity / autoevals.AnswerSimilarity | outputexpected | Semantic similarity between the output and the expected answer. |
| context_precision Context precision | jev | ragas | score | ragas.context_precision / autoevals.ContextPrecision / uipath LegacyContextPrecisionEvaluator | inputcontextexpected | What share of the retrieved context is actually relevant to the input. |
| context_recall Context recall | jev | ragas | score | ragas.context_recall / autoevals.ContextRecall | expectedcontextinput | How much of the expected answer is supported by the retrieved context. |
| context_relevancy Context relevancy | jev | ragas | score | ragas.context_relevancy / autoevals.ContextRelevancy | inputcontext | Overall relevance of the retrieved context to the input. |
| context_entity_recall Context entity recall | jev | ragas | score | ragas.context_entity_recall / autoevals.ContextEntityRecall | expectedcontext | Share of entities in the expected answer that appear in the context. |
| noise_sensitivity Noise sensitivity | jev | ragas | boolean | ragas.noise_sensitivity | outputcontextinputexpected | Did irrelevant context leak into the output as incorrect claims? |
| coherence Coherence | jev | ragas | boolean | ragas.aspect_critic(coherence) | output | Is the output logically organised and internally consistent? |
| conciseness Conciseness | jev | ragas | boolean | ragas.aspect_critic(conciseness) | outputinput | Does the output avoid unnecessary length and repetition? |
| harmfulness Harmfulness | jev | ragas | boolean | ragas.aspect_critic(harmfulness) | output | Could the output cause harm to individuals, groups, or society? |
| maliciousness Maliciousness | jev | ragas | boolean | ragas.aspect_critic(maliciousness) | output | Does the output intend to deceive, exploit, or damage? |
| topic_adherence Topic adherence | jev | ragas | boolean | ragas.topic_adherence | outputcriteriainput | Does the output stay within the topic or domain defined by the criteria? |
| llm_judge LLM-as-judge (rubric) | jev | uipath | score | uipath.eval LLMJudgeOutputEvaluator / LlmAsAJudgeEvaluator | outputcriteriainputexpected | Scores the output against a free-text rubric in `criteria`, the way a generic LLM judge would, but as a calibrated score. |
| trajectory Trajectory | jev | uipath | score | uipath.eval LLMJudgeTrajectoryEvaluator / LegacyTrajectoryEvaluator | trajectoryexpected_trajectoryinput | How closely the agent's step sequence follows the expected trajectory. |
| tool_call_accuracy Tool call accuracy | jev | ragas | score | ragas.tool_call_accuracy / uipath ToolCallArgsEvaluator (semantic) | tool_callsexpected_tool_callsinput | Do the agent's tool calls match the expected calls in tool choice and arguments, allowing semantically equivalent values? |
| agent_goal Agent goal accuracy | jev | ragas | boolean | ragas.agent_goal_accuracy | inputoutputexpectedtrajectory | Did the agent achieve the user's goal described in the input? |
| task_completion Task completion | jev | jeval | score | jeval (agent task completion rubric) | inputoutputexpected | Degree to which the requested task was completed. |
| chunk_relevance Chunk relevance | jevper chunk | ragas | boolean | ragas.context_precision (per-chunk) / uipath LegacyContextPrecisionEvaluator | inputcontextexpected | Every retrieved chunk gets its own yes/no probability of being relevant to the input, all in the same Jev call. Score is precision (share of relevant chunks); details show the per-chunk verdicts. |
| chunk_support Chunk support | jevper chunk | ragas | choice | ragas.faithfulness (per-chunk attribution) / autoevals.Faithfulness | outputcontextinput | For each chunk: does it support the output, contradict it, or say nothing about it? Contradictions are the hallucination signal; score is supporting share minus contradictions. |
| trajectory_steps Trajectory steps (per step) | jevper step | uipath | boolean | uipath.eval LLMJudgeTrajectoryEvaluator, re-expressed as one probability per expected step | trajectoryexpected_trajectoryinput | Instead of one prose verdict from an LLM prompt, each expected step gets its own probability that the agent performed it, plus an order judgment, in one call. Score is the share of steps performed. |
| tool_call_semantic Tool calls (semantic, per call) | jevper tool call | uipath | boolean | uipath.eval ToolCallArgsEvaluator / ToolCallOutputEvaluator, with equivalent-argument matching | tool_callsexpected_tool_callsinput | Each expected tool call gets a probability that an equivalent call was made (same tool, arguments equal in meaning even if formatted differently). Exact-match evaluators fail on '2025-11-12' vs 'Nov 12 2025'; this does not. |
| levenshtein Levenshtein | code · free | autoevals | deterministic | autoevals.Levenshtein | outputexpected | 1 − normalised edit distance between output and expected. |
| exact_match Exact match | code · free | uipath | deterministic | uipath.eval ExactMatchEvaluator / autoevals ExactMatch | outputexpected | Output equals expected after whitespace and case normalisation. |
| contains Contains | code · free | uipath | deterministic | uipath.eval ContainsEvaluator / autoevals ListContains | outputexpected | Output contains the expected string (case-insensitive). |
| valid_json Valid JSON | code · free | autoevals | deterministic | autoevals.ValidJSON | output | Output parses as JSON. |
| json_similarity JSON similarity | code · free | uipath | deterministic | uipath.eval JsonSimilarityEvaluator / autoevals JSONDiff | outputexpected | Recursive structural similarity between output JSON and expected JSON (fields, arrays, numbers, strings). |
| numeric_diff Numeric diff | code · free | autoevals | deterministic | autoevals.NumericDiff | outputexpected | Closeness of a numeric output to the expected number. |
| tool_call_args Tool call args (exact) | code · free | uipath | deterministic | uipath.eval ToolCallArgsEvaluator | tool_callsexpected_tool_calls | Share of expected tool calls that appear in the output with exactly matching arguments. |
| tool_call_order Tool call order | code · free | uipath | deterministic | uipath.eval ToolCallOrderEvaluator | tool_callsexpected_tool_calls | Do the tool names appear in the expected order (as a subsequence)? |
| tool_call_count Tool call count | code · free | uipath | deterministic | uipath.eval ToolCallCountEvaluator | tool_callsexpected_tool_calls | Per-tool call counts match the expected counts. |
| classification Classification match | code · free | uipath | deterministic | uipath.eval BinaryClassificationEvaluator / MulticlassClassificationEvaluator | outputexpected | Predicted label in output equals the expected label. |