Every evaluator.
One call.
jeval runs the evaluators you already use from autoevals, RAGAS, UiPath and promptfoo as calibrated Jev judgments. Send input, output, expected and context once; get every score, probability and confidence back in a single request, in a few hundred milliseconds, for a fraction of a cent.
- Evaluators
- 60
- 38 answered by Jev, 22 in code
- Requests per item
- 1
- every evaluator fans out in one call
- Typical latency
- ~200ms
- measured, 6-evaluator RAG suite
- Typical cost
- $0.00004
- per item, output tokens free
- Cheaper than an LLM judge
- 272×
- on the bundled benchmark
Evals are decisions, not paragraphs.
An LLM judge writes reasoning and then a number. You pay for the reasoning, wait for it, parse it, and get a point estimate with no calibration. Jev never generates text: it reads your state once and returns a probability distribution for every question in the request. jeval maps each evaluator to those questions and combines the answers in code, so the policy is visible and the numbers are comparable.
- input
- What is Jev's context limit?
- context
- Context length: 64k tokens per request; 32k for state plus the longest question.
- output
- 64k tokens per request; 32k for state plus the longest question.
- faithfulness0.00pass
- answer_relevancy0.00pass
- context_precision0.00pass
curl -X POST https://jeval.dev/api/v1/evaluate \
-H 'authorization: Bearer jv_…' \
-H 'content-type: application/json' \
-d '{
"suite": "rag",
"input": "What is Jev's context limit?",
"output": "64k tokens per request; 32k for state plus the longest question.",
"context": ["Context length: 64k tokens per request; 32k for state plus the longest question."]
}'Every chunk gets its own verdict.
Relevant or not, supports or contradicts: one probability per retrieved chunk, in the same request as faithfulness, precision and recall. Contradicting chunks are the hallucination signal, per chunk instead of a paragraph.
Try it in the playgroundTrajectories and tool calls, judged semantically.
UiPath's trajectory and tool-call judges re-expressed as one yes/no probability per expected step and per expected call. Renamed argument keys and reformatted dates stop counting as failures.
See why Jev fitsDecisions you can threshold.
Each result carries a distribution over rubric levels and a confidence. Set your own cut-offs in code, change them without re-running inference, and never parse a paragraph again.
Read the API referenceOne request instead of one judge call per evaluator.
An LLM judge is called once per evaluator, reads a prompt template on top of your data each time and writes its reasoning before the verdict. Jev reads the item once, answers every question in the same request and generates nothing, so you pay for input tokens only.
- Jev requests
- 1
- Judge calls
- 6
- Sample gap
- 494×
Questions, answered first.
Short, direct answers. The same ones an answer engine should be able to quote.
jeval is a universal evaluation API. It exposes the evaluators from autoevals, RAGAS, UiPath's eval library and promptfoo's assertion set through one endpoint, and answers the judgment-based ones with TypeSafe's Jev model instead of an LLM prompt. You send input, output, expected and context; you get back a score, a probability distribution, a confidence and a pass/fail for every evaluator you asked for, in one request.
Run your first evaluation in the next minute.
No sign-up. Free key. Real numbers on screen.