How do I get precision and recall from classification results?

Aggregate the per-item matches into a confusion matrix; jeval's metrics pages show the formulas with a live widget.

code · freeuipathdeterministicmirrors uipath.eval BinaryClassificationEvaluator / MulticlassClassificationEvaluator

Classification match

Classification match checks that a predicted label equals the expected label after normalisation. It mirrors UiPath's Binary and Multiclass classification evaluators and is the per-item input to accuracy, precision, recall and F1.

Run this example in the console runs in code · $0

how it works // frame by frame

What the evaluator does

Normalise and compare. Aggregate the results into a confusion matrix to get precision, recall and F1 per class.

frame 1/4 · the two inputs

predicted

billing

expected

billing

1.normalise

2.compare

3.feed into the confusion matrix

correctscore 1

runs in code · $0 · ~0 ms · deterministic

when to use it

Reach for it when

Intent routing, sentiment, ticket categorisation, any labelling task.
Building precision/recall dashboards per class.

Not the right tool when

Labels are free text (use answer_similarity).

Watch out for

Label vocabularies must match exactly (Billing vs billing is handled, billing vs payments is not).

inputs and example

What to send

required fields

outputexpected

optional fields

none

in suites

CorrectnessEverything

result shape

score 0–1 computed in code, label, passed at an evaluator-specific threshold, no probabilities.

POST /api/v1/evaluateopen in console

{
  "evaluators": [
    "classification"
  ],
  "input": "Classify: 'My card was charged twice.'",
  "output": "billing",
  "expected": "billing"
}

expected

Correct.

rolling it up

Aggregate with

One result per item is a fact; a dataset of them is a metric. These are the aggregations that fit this evaluator's output shape.

Accuracy

(TP + TN) / (TP + TN + FP + FN)

Accuracy is the share of items the evaluator got right.

Precision, recall and F1

P = TP/(TP+FP) · R = TP/(TP+FN) · F1 = 2PR/(P+R)

Precision is the share of predicted positives that were really positive; recall is the share of real positives the evaluator caught; F1 is their harmonic mean, which is only high when both are.

Confusion matrix

TP · FP · FN · TN counts per class

A confusion matrix counts, for a binary or multiclass check, how often each true label was predicted as each label: true positives, false positives, false negatives and true negatives.

Cohen's kappa

κ = (p_o − p_e) / (1 − p_e)

Cohen's kappa measures agreement between two raters (an evaluator and a human, or two evaluators) after subtracting the agreement you would expect by chance.

related evaluators

exact_matchcode · free

Output equals expected after whitespace and case normalisation.

moderationjev

Classifies the output into a moderation category (none, harassment, hate, sexual, violence, self-harm, dangerous).

factualityjev

Compares the output to the expected answer and classifies the relationship (subset, superset, equivalent, disagreement, stylistic).

questions people ask

How do I get precision and recall from classification results?: Aggregate the per-item matches into a confusion matrix; jeval's metrics pages show the formulas with a live widget.