JEVAL
jevjevalscoremirrors jeval (agent task completion rubric)

Task completion

Task completion rates how completely an output accomplishes the task requested in the input on a 0–3 scale, using the expected answer as the definition of done when present. It is jeval's graded counterpart to agent goal accuracy.

Run this example in the console 1 jev question · answered inside one request
how it works // frame by frame

What the evaluator does

A Jev Score over `input`, `output` and optional `expected`.

frame 1/4 · the state
state
inputRefund order 4471 and notify the customer.
outputNotified customer; refund pending approval.
expectedRefund issued and customer notified.
question → jev

Rate how completely `output` accomplishes the task requested in `input` (use `expected` as the definition of done).

ordered levels
0
not completed
2%
1
a small part
20%
2
most of the task
70%
3
fully completed
8%
1.8 / 3conf 70%fail · threshold 75%
exactly what jev is asked

The question, verbatim

This is the question the API sends for the example below, generated from the same code path the playground and API use. Jev sees the request fields as state and returns a probability for each outcome. Nothing is generated, so there is nothing to parse.

qscore · ordered rubric

Rate how completely `output` accomplishes the task requested in `input` (use `expected` as the definition of done if present).

  1. 0The task was not completed
  2. 1A small part of the task was completed
  3. 2Most of the task was completed
  4. 3The task was fully completed as asked
when to use it

Reach for it when

  • Agent runs where partial progress matters.
  • Multi-part instructions where you want credit for the parts done.

Not the right tool when

  • Binary success is all that matters (use agent_goal).

Watch out for

  • Include evidence of completion in the output; a bare “done” cannot be graded.
inputs and example

What to send

required fields
inputoutput
optional fields
expected
in suites
AgentEverything
result shape

score 0–1 (expected level ÷ top level), label “x.x / n”, confidence, a probability per level, passed at 75% of the scale.

POST /api/v1/evaluateopen in console 
{
  "evaluators": [
    "task_completion"
  ],
  "input": "Refund order 4471 and notify the customer.",
  "output": "Notified customer; refund pending approval.",
  "expected": "Refund issued and customer notified."
}
expected

Level 2: most of the task, refund not final.

rolling it up

Aggregate with

One result per item is a fact; a dataset of them is a metric. These are the aggregations that fit this evaluator's output shape.

Mean score
Σ score_i / n

Mean score averages the normalised 0–1 score across items.

Pass rate
passed / total

Pass rate is the share of items whose evaluator result met its threshold.

related evaluators
agent_goaljev
Did the agent achieve the user's goal described in the input?
trajectory_stepsjev
Instead of one prose verdict from an LLM prompt, each expected step gets its own probability that the agent performed it, plus an order judgment, in one call. Score is the share of steps performed.
answer_relevancyjev
How directly and completely the output addresses the input question.
questions people ask
Task completion vs agent goal accuracy?
Task completion is graded 0–3 and gives partial credit; agent goal accuracy is a single yes/no probability.