# jeval > jeval is a universal evaluation API for LLM outputs. It runs the evaluators from autoevals (Braintrust), RAGAS, UiPath's eval library and promptfoo as calibrated judgments from TypeSafe's Jev model. You send input, output, expected and context once; every selected evaluator's questions are answered in a single Jev request, and you get back a score, a probability distribution, a confidence and pass/fail for each. Key facts: - Evaluators: 60 in total; 38 are Jev judgments and 22 run in code for free. - One request per item: a six-evaluator RAG suite takes about 200 ms and costs about $0.00004. - Price: Jev input tokens cost $0.042 per million; output tokens are free because Jev does not generate text. - Per-unit evaluators judge every context chunk, expected step or expected tool call separately, still in the same request. - Free API keys: 1,000 evaluations a day and 60 requests a minute, no sign-up. - Results are probabilities, so pass thresholds are set in your own code and can change without re-running inference. Limits: jeval is an HTTP API, not a Python or TypeScript library, and is in beta. Scores match the source evaluators' semantics but are not bit-identical ports. ## Product - [Home](https://jeval.dev/): what jeval does, how one Jev call replaces per-evaluator LLM judges, and frequently asked questions - [Playground](https://jeval.dev/app): run any evaluators on your own input and see scores, probabilities and the cost against an LLM judge - [API reference](https://jeval.dev/app?view=api): POST /api/v1/evaluate, POST /api/v1/evaluate/batch, GET /api/v1/evaluators - [Benchmark](https://jeval.dev/app?view=benchmark): Jev against OpenAI, Anthropic and xAI judges on a hand-labelled dataset - [Free API key](https://jeval.dev/app?view=keys): mint a key instantly - [Evaluators](https://jeval.dev/evaluators): every evaluator explained - [Metrics](https://jeval.dev/metrics): the aggregate metrics that roll evaluator results up ## Evaluators - [Factuality](https://jeval.dev/evaluators/factuality): Factuality compares a submitted answer with an expert (expected) answer and classifies their relationship as subset, superset, equivalent, disagreement, or a difference that does not affect the facts. - [Closed QA](https://jeval.dev/evaluators/closed_qa): Closed QA asks a single yes/no question: does the output correctly answer the input when judged strictly against the supplied criteria? - [Possible](https://jeval.dev/evaluators/possible): Possible checks whether the output is a genuine attempt at the task rather than a refusal, deflection or “I can't do that”. - [Security](https://jeval.dev/evaluators/security): Security flags outputs that contain malicious or dangerous content: prompt-injection payloads, credential exfiltration, working exploit code or instructions that enable serious harm. - [Moderation](https://jeval.dev/evaluators/moderation): Moderation classifies an output into one content category: none, harassment, hate, sexual, violence, self-harm or dangerous. - [Battle](https://jeval.dev/evaluators/battle): Battle is a head-to-head comparison: given the input, is the candidate output a better response than the baseline (expected) response? - [Summary](https://jeval.dev/evaluators/summary): Summary judges which of two summaries of a source text is better: the candidate output or the reference. It mirrors autoevals Summary and is a Battle specialised for summarisation. - [SQL](https://jeval.dev/evaluators/sql): SQL judges whether two queries are semantically equivalent, meaning they return the same result set for the request, ignoring aliasing, formatting and column order. - [Translation](https://jeval.dev/evaluators/translation): Translation rates an output as a translation of the input on a 0–4 scale covering accuracy, naturalness and register, optionally against a reference translation. It mirrors autoevals Translation. - [Humor](https://jeval.dev/evaluators/humor): Humor rates how funny an output is on a 0–3 scale. It mirrors autoevals Humor and exists mostly to show that subjective rubrics still yield stable distributions. - [Faithfulness](https://jeval.dev/evaluators/faithfulness): Faithfulness measures whether every claim in an output is supported by the retrieved context. - [Answer relevancy](https://jeval.dev/evaluators/answer_relevancy): Answer relevancy rates how directly and completely an output addresses the question asked, penalising padding, drift and non-answers. - [Answer correctness](https://jeval.dev/evaluators/answer_correctness): Answer correctness rates the factual agreement between an output and a reference answer on a 0–4 scale. - [Answer similarity](https://jeval.dev/evaluators/answer_similarity): Answer similarity rates the semantic similarity between an output and a reference on a 0–3 scale, ignoring wording. - [Context precision](https://jeval.dev/evaluators/context_precision): Context precision rates what share of the retrieved context is actually relevant to the question, as a 0–3 level. - [Context recall](https://jeval.dev/evaluators/context_recall): Context recall rates how much of the reference answer can be attributed to the retrieved context, as a 0–3 level. - [Context relevancy](https://jeval.dev/evaluators/context_relevancy): Context relevancy rates the overall relevance of the retrieved context to the question on a 0–3 scale, without needing a reference answer. - [Context entity recall](https://jeval.dev/evaluators/context_entity_recall): Context entity recall rates what share of the named entities, numbers, dates and terms in the reference answer appear in the retrieved context. - [Noise sensitivity](https://jeval.dev/evaluators/noise_sensitivity): Noise sensitivity detects whether irrelevant or misleading context leaked into the output as incorrect claims. It mirrors RAGAS noise_sensitivity and returns the probability that leakage happened. - [Coherence](https://jeval.dev/evaluators/coherence): Coherence checks whether an output is logically organised and internally consistent. It mirrors the RAGAS aspect critic for coherence and returns Jev's probability that the text holds together. - [Conciseness](https://jeval.dev/evaluators/conciseness): Conciseness checks whether an output conveys its content without unnecessary length, repetition or filler. It mirrors the RAGAS aspect critic for conciseness. - [Harmfulness](https://jeval.dev/evaluators/harmfulness): Harmfulness estimates whether an output could cause harm to individuals, groups or society if acted upon or distributed. It mirrors the RAGAS aspect critic for harmfulness. - [Maliciousness](https://jeval.dev/evaluators/maliciousness): Maliciousness estimates whether an output is intended to deceive, manipulate, exploit or damage the reader or a third party. - [Topic adherence](https://jeval.dev/evaluators/topic_adherence): Topic adherence checks whether an output stays within the topic or domain described in the criteria, without wandering into out-of-scope subjects. - [Tool call accuracy](https://jeval.dev/evaluators/tool_call_accuracy): Tool call accuracy rates how well an agent's tool calls match the expected calls in tool choice and arguments, allowing semantically equivalent values, as a 0–3 level. - [Agent goal accuracy](https://jeval.dev/evaluators/agent_goal): Agent goal accuracy asks whether the agent achieved the user's goal as stated in the input, judging from the final output and optionally the trajectory. - [Chunk relevance](https://jeval.dev/evaluators/chunk_relevance): Chunk relevance gives every retrieved context chunk its own yes/no probability of being relevant to the question, all inside one Jev request. - [Chunk support](https://jeval.dev/evaluators/chunk_support): Chunk support asks, for each retrieved chunk, whether it supports the output's claims, contradicts them, or says nothing about them. Contradictions are the hallucination signal, located to the chunk. - [LLM-as-judge (rubric)](https://jeval.dev/evaluators/llm_judge): LLM-as-judge with a rubric scores an output 0–4 against free-text criteria, the way UiPath's LLMJudgeOutputEvaluator and autoevals LLMClassifier do, but as a calibrated Jev distribution over five… - [Trajectory](https://jeval.dev/evaluators/trajectory): Trajectory rates how closely an agent's step sequence follows the expected trajectory, as a single 0–3 level. - [Trajectory steps (per step)](https://jeval.dev/evaluators/trajectory_steps): ”) plus an order judgment, all in one Jev request. The score is the share of steps performed, weighted by order, and the details name the missing steps. - [Tool calls (semantic, per call)](https://jeval.dev/evaluators/tool_call_semantic): Tool calls (semantic) gives each expected tool call a probability that an equivalent call was made: same tool, and argument values that mean the same thing even if keys are renamed or values… - [Exact match](https://jeval.dev/evaluators/exact_match): Exact match checks whether the output equals the expected value after trimming whitespace and lower-casing. - [Contains](https://jeval.dev/evaluators/contains): Contains checks whether the expected string appears anywhere in the output, case-insensitively. It mirrors UiPath's ContainsEvaluator and autoevals ListContains and is free. - [JSON similarity](https://jeval.dev/evaluators/json_similarity): JSON similarity computes a recursive structural similarity between the output JSON and the expected JSON, field by field, with partial credit for close numbers and strings. - [Valid JSON](https://jeval.dev/evaluators/valid_json): Valid JSON checks that the output parses as JSON. It mirrors autoevals ValidJSON and is the cheapest possible structural gate. - [Levenshtein](https://jeval.dev/evaluators/levenshtein): Levenshtein returns 1 minus the normalised edit distance between the output and the expected string. It mirrors autoevals Levenshtein and is free. - [Numeric diff](https://jeval.dev/evaluators/numeric_diff): Numeric diff measures how close a numeric output is to the expected number, as 1 minus the relative difference. It mirrors autoevals NumericDiff and is free. - [Tool call args (exact)](https://jeval.dev/evaluators/tool_call_args): Tool call args (exact) computes the share of expected tool calls that appear in the output with the same name and exactly matching arguments. It mirrors UiPath's ToolCallArgsEvaluator and is free. - [Tool call order](https://jeval.dev/evaluators/tool_call_order): Tool call order checks that the expected tool names occur in the expected order within the actual calls, as an ordered subsequence. It mirrors UiPath's ToolCallOrderEvaluator and is free. - [Tool call count](https://jeval.dev/evaluators/tool_call_count): Tool call count checks that each tool was called the expected number of times. It mirrors UiPath's ToolCallCountEvaluator and is free. - [Classification match](https://jeval.dev/evaluators/classification): Classification match checks that a predicted label equals the expected label after normalisation. - [Task completion](https://jeval.dev/evaluators/task_completion): Task completion rates how completely an output accomplishes the task requested in the input on a 0–3 scale, using the expected answer as the definition of done when present. - [Regex](https://jeval.dev/evaluators/regex): Regex checks that the output matches a regular expression you supply in `pattern`. It mirrors promptfoo's `regex` assertion, runs in code and costs nothing. - [Starts with](https://jeval.dev/evaluators/starts_with): Starts with checks that the output begins with the expected string, ignoring case and surrounding whitespace. It mirrors promptfoo's `starts-with` and is free. - [Contains any](https://jeval.dev/evaluators/contains_any): Contains any passes when the output contains at least one of the strings in `values`, case-insensitively. It mirrors promptfoo's `contains-any` and `icontains-any` and is free. - [Contains all](https://jeval.dev/evaluators/contains_all): Contains all checks that every string in `values` appears in the output, case-insensitively, and scores the share present. It mirrors promptfoo's `contains-all` and `icontains-all`. - [Contains JSON](https://jeval.dev/evaluators/contains_json): Contains JSON passes when a parseable JSON object or array appears anywhere in the output, even surrounded by prose or code fences. - [Is SQL](https://jeval.dev/evaluators/is_sql): Is SQL checks that the output is a plausible SQL statement, or contains one in a ```sql block: a statement keyword, balanced parentheses and quotes. - [Is XML](https://jeval.dev/evaluators/is_xml): Is XML checks that the output is well-formed XML: tags balance and nest, with self-closing tags allowed. 5 for a well-formed fragment inside prose. It mirrors promptfoo's `is-xml` and `contains-xml`. - [Is HTML](https://jeval.dev/evaluators/is_html): Is HTML checks that the output is HTML with balanced tags, allowing void elements such as br, img and input. 5 for HTML inside prose. It mirrors promptfoo's `is-html` and `contains-html`. - [ROUGE-N](https://jeval.dev/evaluators/rouge_n): ROUGE-N measures n-gram overlap between the output and a reference: the score is ROUGE-1 F1 with ROUGE-2 reported alongside. 75 threshold and runs in code. - [BLEU](https://jeval.dev/evaluators/bleu): BLEU scores the output against a reference with sentence-level BLEU-4: geometric mean of 1- to 4-gram precision with add-one smoothing and a brevity penalty. 5 default threshold. - [GLEU](https://jeval.dev/evaluators/gleu): GLEU is Google's sentence-level variant of BLEU: the minimum of n-gram precision and recall over pooled 1- to 4-grams, better behaved on single sentences. 5 threshold. - [Tool call F1](https://jeval.dev/evaluators/tool_call_f1): Tool call F1 compares the multiset of tool names an agent called with the names expected and reports F1, precision and recall. It mirrors promptfoo's `tool-call-f1` and passes at 0.8. - [Is refusal](https://jeval.dev/evaluators/is_refusal): Is refusal judges whether the output declines the task rather than attempting it, returning the probability of a refusal. - [G-Eval](https://jeval.dev/evaluators/g_eval): G-Eval scores an output 1–5 against custom criteria. The original method prompts an LLM for evaluation steps and averages sampled scores by their token probabilities; Jev returns the probability of… - [Select best](https://jeval.dev/evaluators/select_best): Select best compares several candidate responses and picks the one that best satisfies your criteria, returning a probability per candidate. - [Classifier](https://jeval.dev/evaluators/classifier): Classifier assigns the output one of the labels you pass in `values`, such as tones, intents or toxicity classes, with a probability per label. - [Conversation relevance](https://jeval.dev/evaluators/conversation_relevance): Conversation relevance judges whether the latest reply stays relevant to the most recent user turn and consistent with the whole conversation. ## Metrics - [Confusion matrix](https://jeval.dev/metrics/confusion-matrix): A confusion matrix counts, for a binary or multiclass check, how often each true label was predicted as each label: true positives, false positives, false negatives and true negatives. - [Accuracy](https://jeval.dev/metrics/accuracy): Accuracy is the share of items the evaluator got right. For labels it is exact agreement; for graded rubrics jeval's benchmark counts a judgment correct when it lands within one level of the gold… - [Precision, recall and F1](https://jeval.dev/metrics/precision-recall-f1): Precision is the share of predicted positives that were really positive; recall is the share of real positives the evaluator caught; F1 is their harmonic mean, which is only high when both are. - [Pass rate](https://jeval.dev/metrics/pass-rate): Pass rate is the share of items whose evaluator result met its threshold. It is the number most teams gate releases on, and the one most sensitive to where you set the threshold. - [Mean score](https://jeval.dev/metrics/mean-score): Mean score averages the normalised 0–1 score across items. Because Jev returns an expected level (probability-weighted), the mean of expected levels is itself a calibrated average and moves smoothly… - [Mean absolute error](https://jeval.dev/metrics/mae): Mean absolute error measures, for graded rubrics, how far the evaluator's level is from the gold level on average. - [Brier score](https://jeval.dev/metrics/brier-score): The Brier score measures probability quality: the mean squared difference between a predicted probability and what actually happened (1 or 0). - [Calibration](https://jeval.dev/metrics/calibration): Calibration asks whether a probability means what it says: among all judgments made at 80% confidence, were about 80% right? - [Consistency](https://jeval.dev/metrics/consistency): Consistency is how often an evaluator gives the same verdict when asked the same question again. - [Cohen's kappa](https://jeval.dev/metrics/cohens-kappa): Cohen's kappa measures agreement between two raters (an evaluator and a human, or two evaluators) after subtracting the agreement you would expect by chance. - [Win rate](https://jeval.dev/metrics/win-rate): Win rate is the share of pairwise comparisons a candidate wins against a baseline, counting ties as half.