What are TP, FP, FN and TN?

True positive: predicted positive and it was. False positive: predicted positive, was not. False negative: predicted negative, was positive. True negative: predicted negative and it was.

aggregate metric

Confusion matrix

TP · FP · FN · TN counts per class

A confusion matrix counts, for a binary or multiclass check, how often each true label was predicted as each label: true positives, false positives, false negatives and true negatives. Every classification metric on this site is a ratio of its cells.

how it is computed

From results to a number

Take a pass/fail or label decision per item (from classification, closed_qa, security, or any thresholded probability) and the gold label. Tally the four cells. For multiclass, build one matrix per class treating that class as positive.

inputs

Per-item binary decisions plus gold labels. Probabilities must be thresholded first (jeval uses 0.5 unless you change it).

live · 40 judgments · move the thresholdthreshold0.50

p = 0p = 1

gold positive, flagged gold positive, missed gold negative, flagged gold negative, correctly ignored

predicted positivepredicted negativegold positive

TP16hit

FN5miss

gold negative

FP1false alarm

TN18correct rejection

precision0.94

TP / (TP + FP)

recall0.76

TP / (TP + FN)

F10.84

2PR / (P + R)

accuracy0.85

(TP + TN) / n

when to use it

Reach for it when

Before quoting any accuracy number: it shows which direction the errors go.
Choosing a threshold for a probability evaluator: move the line, watch FP and FN trade off.
Explaining a safety filter to stakeholders.

Not the right number when

Graded 0–4 outputs where you want distance from the truth (use MAE).
Probabilities you want to score without thresholding (use Brier).

Watch out for

Class imbalance makes accuracy look good while one cell is empty; always look at the matrix, not just the diagonal.
Which class is “positive” changes precision and recall; state it.

feeds from

Evaluators that produce this

classificationcode · free

Predicted label in output equals the expected label.

closed_qajev

Does the output correctly answer the input, judged against the supplied criteria?

securityjev

Does the output contain content that is malicious or unsafe (exploits, injection payloads, credential theft, harmful instructions)?

moderationjev

Classifies the output into a moderation category (none, harassment, hate, sexual, violence, self-harm, dangerous).

possiblejev

Is the output a genuine attempt at the task rather than a refusal, deflection, or 'I cannot' response?

agent_goaljev

Did the agent achieve the user's goal described in the input?

related metrics

Accuracy

(TP + TN) / (TP + TN + FP + FN)

Precision, recall and F1

P = TP/(TP+FP) · R = TP/(TP+FN) · F1 = 2PR/(P+R)

Cohen's kappa

κ = (p_o − p_e) / (1 − p_e)

questions people ask

What are TP, FP, FN and TN?: True positive: predicted positive and it was. False positive: predicted positive, was not. False negative: predicted negative, was positive. True negative: predicted negative and it was.