JEVAL
jevpromptfoochoicemirrors promptfoo classifier

Classifier

Classifier assigns the output one of the labels you pass in `values`, such as tones, intents or toxicity classes, with a probability per label. It mirrors promptfoo's `classifier` without a separate HuggingFace model; when `expected` names a label, the check passes on a match.

Run this example in the console 1 jev question · answered inside one request
how it works // frame by frame

What the evaluator does

One Jev Choice over the labels, optionally guided by `criteria`. Score is the top label's probability; the full distribution is returned.

frame 1/4 · the state
state
outputI've waited three weeks and nobody has replied. This is unacceptable.
valuesneutral · frustrated · grateful · confused
question → jev

Classify `output` into exactly one of the labels in `values`.

options
frustrated
frustrated
93%
neutral
neutral
3%
confused
confused
3%
grateful
grateful
1%
frustrated→ score 1conf 93%
exactly what jev is asked

The question, verbatim

This is the question the API sends for the example below, generated from the same code path the playground and API use. Jev sees the request fields as state and returns a probability for each outcome. Nothing is generated, so there is nothing to parse.

labelchoice · one of n

Classify `output` into exactly one of the labels in `values`.

  1. neutralneutral
  2. frustratedfrustrated
  3. gratefulgrateful
  4. confusedconfused
when to use it

Reach for it when

  • Tone, sentiment, intent or topic tagging with your own label set.
  • Building confusion matrices for a routing model against gold labels.

Not the right tool when

  • Labels are free text (use answer_similarity).
  • Fixed moderation categories (moderation already has them).

Watch out for

  • Overlapping labels split probability; keep the set mutually exclusive and describe edge cases in `criteria`.
inputs and example

What to send

required fields
outputvalues
optional fields
inputexpectedcriteria
in suites
PromptfooEverything
result shape

score mapped from the chosen option, label = option key, confidence, a probability per option, passed by option.

POST /api/v1/evaluateopen in console 
{
  "evaluators": [
    "classifier"
  ],
  "output": "I've waited three weeks and nobody has replied. This is unacceptable.",
  "values": [
    "neutral",
    "frustrated",
    "grateful",
    "confused"
  ],
  "expected": "frustrated"
}
expected

frustrated with high probability: pass.

rolling it up

Aggregate with

One result per item is a fact; a dataset of them is a metric. These are the aggregations that fit this evaluator's output shape.

Accuracy
(TP + TN) / (TP + TN + FP + FN)

Accuracy is the share of items the evaluator got right.

Confusion matrix
TP · FP · FN · TN counts per class

A confusion matrix counts, for a binary or multiclass check, how often each true label was predicted as each label: true positives, false positives, false negatives and true negatives.

Precision, recall and F1
P = TP/(TP+FP) · R = TP/(TP+FN) · F1 = 2PR/(P+R)

Precision is the share of predicted positives that were really positive; recall is the share of real positives the evaluator caught; F1 is their harmonic mean, which is only high when both are.

Cohen's kappa
κ = (p_o − p_e) / (1 − p_e)

Cohen's kappa measures agreement between two raters (an evaluator and a human, or two evaluators) after subtracting the agreement you would expect by chance.

related evaluators
classificationcode · free
Predicted label in output equals the expected label.
moderationjev
Classifies the output into a moderation category (none, harassment, hate, sexual, violence, self-harm, dangerous).
topic_adherencejev
Does the output stay within the topic or domain defined by the criteria?
questions people ask
How is classifier different from classification?
classification compares a predicted label string with an expected one in code. classifier asks Jev to assign the label from your list, so it works on raw text and returns a probability per label.