JEVAL
jevautoevalsscoremirrors autoevals.Humor

Humor

Humor rates how funny an output is on a 0–3 scale. It mirrors autoevals Humor and exists mostly to show that subjective rubrics still yield stable distributions.

Run this example in the console 1 jev question · answered inside one request
how it works // frame by frame

What the evaluator does

A four-level Score question. Expect wide distributions; humour is subjective and the confidence will say so.

frame 1/4 · the state
state
inputTell a one-liner about probabilities.
outputI'd tell you a probability joke, but there's only a 60% chance you'd get it.
question → jev

Rate how funny `output` is as a response to `input`.

ordered levels
0
not funny
5%
1
mildly amusing
30%
2
funny
50%
3
very funny, well-crafted
15%
1.8 / 3conf 50%fail · threshold 75%
exactly what jev is asked

The question, verbatim

This is the question the API sends for the example below, generated from the same code path the playground and API use. Jev sees the request fields as state and returns a probability for each outcome. Nothing is generated, so there is nothing to parse.

qscore · ordered rubric

Rate how funny `output` is as a response to `input`.

  1. 0Not funny; no attempt at humor or a failed one
  2. 1Mildly amusing
  3. 2Funny
  4. 3Very funny and well-crafted
when to use it

Reach for it when

  • Ranking joke or caption candidates.
  • Demonstrating calibrated uncertainty on a subjective dimension.

Not the right tool when

  • Anything where a false positive costs money; this is a taste judgment.

Watch out for

  • Cultural context matters; put the audience in `input`.
inputs and example

What to send

required fields
output
optional fields
input
in suites
Everything
result shape

score 0–1 (expected level ÷ top level), label “x.x / n”, confidence, a probability per level, passed at 75% of the scale.

POST /api/v1/evaluateopen in console 
{
  "evaluators": [
    "humor"
  ],
  "input": "Tell a one-liner about probabilities.",
  "output": "I'd tell you a probability joke, but there's only a 60% chance you'd get it."
}
expected

Level 2: funny, not brilliant.

rolling it up

Aggregate with

One result per item is a fact; a dataset of them is a metric. These are the aggregations that fit this evaluator's output shape.

Mean score
Σ score_i / n

Mean score averages the normalised 0–1 score across items.

Win rate
wins + 0.5 · ties / comparisons

Win rate is the share of pairwise comparisons a candidate wins against a baseline, counting ties as half.

related evaluators
battlejev
Head-to-head: is the output a better response to the input than the expected (baseline) response?
llm_judgejev
Scores the output against a free-text rubric in `criteria`, the way a generic LLM judge would, but as a calibrated score.
questions people ask
Why include a humour evaluator?
It is in autoevals, and it is a good demonstration that even a subjective rubric returns a distribution you can threshold rather than a single opinion.