jeval playground — run 43 evaluators in one Jev call

api // v1 // json over https // free keys

Four endpoints.

Send the fields you have. Evaluators whose required fields are missing are returned as skipped rather than failing the request. Every judgment-based evaluator runs inside one Jev request per item.

POST/api/v1/evaluate

One item, any evaluators or a suite. Returns results, usage and the judge comparison.

POST/api/v1/evaluate/batch

Up to 100 items with shared evaluators/suite. Adds per-evaluator aggregates.

GET/api/v1/evaluators

The catalogue: ids, families, kinds, required fields, suites, pricing assumptions.

POST/api/v1/keys

Mint a free API key instantly (1,000 evaluations/day, 60 rpm). GET with the key as Bearer returns usage.

authentication

Optional. Without a key you get 20 requests a minute and 10 items per request, enough for the playground. With a key: Authorization: Bearer jv_… and 1,000 evaluations a day. Quota headers come back on every response.

get a key

curl -X POST https://jeval.dev/api/v1/keys
→ { "key": "jv_…", "dailyLimit": 1000, "rpm": 60, "remainingToday": 1000, "resetsAt": "…" }

request

{
  "evaluators": ["faithfulness", "answer_relevancy", "factuality", "levenshtein"],
  "input":    "What is Jev's context limit?",
  "output":   "64k tokens per request; 32k for state plus the longest question.",
  "expected": "64k per request, 32k for state + longest question.",
  "context":  ["Context length: 64k tokens per request; 32k tokens for state plus the longest question."]
}

response

{
  "results": [
    { "id": "faithfulness", "family": "ragas", "kind": "score",
      "score": 0.98, "label": "3.9 / 4", "confidence": 0.93,
      "probabilities": { "0": 0, "1": 0, "2": 0.01, "3": 0.08, "4": 0.91 },
      "rationale": "Every claim is traceable and the output adds nothing the context does not state",
      "passed": true },
    { "id": "factuality", "family": "autoevals", "kind": "choice",
      "score": 1, "label": "C", "confidence": 0.88, "probabilities": { "A": 0.04, "B": 0.05, "C": 0.9, "D": 0, "E": 0.01 },
      "rationale": "The submitted answer contains all the same details as the expert answer.", "passed": true },
    { "id": "levenshtein", "family": "autoevals", "kind": "deterministic",
      "score": 0.71, "label": "21 edits", "confidence": null, "probabilities": null,
      "rationale": "Edit distance 21 over 72 chars", "passed": false }
  ],
  "usage": { "model": "jev-1.13.0", "requests": 1, "input_tokens": 611,
             "output_tokens": 54, "cost_usd": 0.0000257, "latency_ms": 212 },
  "comparison": { "baseline": "GPT-4.1-class judge, one call per evaluator", "calls": 3,
                  "est_cost_usd": 0.0095, "est_latency_ms_sequential": 5700,
                  "cost_multiple": 369, "speed_multiple_sequential": 27 }
}

typescript

const res = await fetch("https://jeval.dev/api/v1/evaluate", {
  method: "POST",
  headers: { "content-type": "application/json" },
  body: JSON.stringify({ suite: "rag", input, output, expected, context }),
});
const { results, usage, comparison } = await res.json();
const faithful = results.find(r => r.id === "faithfulness");
if (faithful.score < 0.75 && faithful.confidence > 0.7) flag(item);

python

import requests

r = requests.post("https://jeval.dev/api/v1/evaluate", json={
    "suite": "correctness",
    "input": question, "output": answer, "expected": gold,
})
for res in r.json()["results"]:
    print(res["id"], round(res["score"], 2), res["label"], res["passed"])

batch

POST /api/v1/evaluate/batch
{ "suite": "rag", "items": [ { "input": "...", "output": "...", "context": ["..."] }, ... ] }
→ { "items": [...], "usage": {...}, "comparison": {...},
    "aggregate": [ { "id": "faithfulness", "mean": 0.84, "pass_rate": 0.8 }, ... ] }

request fields

output *: The text under evaluation.
input: Task, question or prompt.
expected: Reference answer / gold label / baseline response.
context: string or string[] of retrieved chunks.
criteria: Free-text rubric for judge-style evaluators.
tool_calls, expected_tool_calls: [{ name, args }] for agent checks.
trajectory, expected_trajectory: string[] of agent steps.
evaluators | suite: Evaluator ids, or one of rag · grounding · correctness · safety · agent · all.
metadata: Echoed back unchanged.

result fields

score: 0–1, normalised. Rubric levels map linearly; choices map to library-equivalent weights (Factuality: A 0.4, B 0.6, C 1, D 0, E 1).
label: Rubric position, chosen option, or match summary.
confidence: Jev's distribution concentration; |p − 0.5| × 2 for yes/no; null for deterministic.
probabilities: Per level/option distribution when available.
rationale: The rubric text or option meaning that was selected.
passed: Library-style default threshold; set your own from score + confidence.
skipped: Present when required fields were missing.

pricing + comparison assumptions

Jev cost is usage.input_tokens × $0.042 / 1M; output tokens are free. The comparison assumes a gpt-4.1-class judge, one call per evaluator at $2 in / $8 out per Mtok, 420 template tokens plus the same state per call, 140 output tokens, and 1900 ms per call (4-way parallel for the parallel figure). It is an estimate to size the gap, not a measurement of any specific vendor. 33 of the 43 evaluators are judgment-based; the rest are free.