Precision, recall and F1
P = TP/(TP+FP) · R = TP/(TP+FN) · F1 = 2PR/(P+R)
Precision is the share of predicted positives that were really positive; recall is the share of real positives the evaluator caught; F1 is their harmonic mean, which is only high when both are. Together they describe a detector better than accuracy does.
From results to a number
From the confusion matrix: precision divides true positives by everything flagged, recall divides them by everything that should have been flagged. F1 punishes the lopsided case where one is high and the other low. In RAG, context precision and chunk_relevance are precision over retrieved chunks; context recall is recall over the reference's statements.
inputs
Per-item binary decisions plus gold labels, or per-chunk / per-step / per-call decisions inside one item.
live · 40 judgments · move the threshold
p = 0p = 1
gold positive, flagged gold positive, missed gold negative, flagged gold negative, correctly ignored
predicted positivepredicted negativegold positive
TP16hit
FN5miss
gold negativeFP1false alarm
TN18correct rejection
precision0.94
TP / (TP + FP)
recall0.76
TP / (TP + FN)
F10.84
2PR / (P + R)
accuracy0.85
(TP + TN) / n
Reach for it when
- Detectors: hallucination, unsafe content, injection, off-topic. Recall is the cost of missing one; precision is the cost of a false alarm.
- Retrieval: precision@k over retrieved chunks, recall against the needed facts.
- Agent steps: precision over steps taken, recall over steps expected (trajectory_steps).
Not the right number when
- Every error costs the same and classes are balanced (accuracy is enough).
- The output is a continuous score you do not want to threshold (Brier, MAE).
Watch out for
- Moving the threshold trades precision for recall; report the threshold with the numbers or plot the curve.
- Macro vs micro averaging differ on imbalanced multiclass data; say which.
- F1 ignores true negatives entirely, which is right for rare-positive detectors and wrong for balanced tasks.
Evaluators that produce this
chunk_relevancejev
Every retrieved chunk gets its own yes/no probability of being relevant to the input, all in the same Jev call. Score is precision (share of relevant chunks); details show the per-chunk verdicts.chunk_supportjev
For each chunk: does it support the output, contradict it, or say nothing about it? Contradictions are the hallucination signal; score is supporting share minus contradictions.context_precisionjev
What share of the retrieved context is actually relevant to the input.context_recalljev
How much of the expected answer is supported by the retrieved context.trajectory_stepsjev
Instead of one prose verdict from an LLM prompt, each expected step gets its own probability that the agent performed it, plus an order judgment, in one call. Score is the share of steps performed.tool_call_semanticjev
Each expected tool call gets a probability that an equivalent call was made (same tool, arguments equal in meaning even if formatted differently). Exact-match evaluators fail on '2025-11-12' vs 'Nov 12 2025'; this does not.securityjev
Does the output contain content that is malicious or unsafe (exploits, injection payloads, credential theft, harmful instructions)?closed_qajev
Does the output correctly answer the input, judged against the supplied criteria?- What is the difference between precision and recall?
- Precision: of the items you flagged, how many deserved it. Recall: of the items that deserved flagging, how many you flagged. High precision means few false alarms; high recall means few misses.
- Why is F1 a harmonic mean?
- The harmonic mean is dominated by the smaller value, so F1 stays low unless both precision and recall are high. An arithmetic mean would reward flagging everything.
- How does jeval compute context precision?
- chunk_relevance asks Jev one relevance question per retrieved chunk in a single request and reports relevant ÷ retrieved, which is precision over chunks.