Goal accuracy vs task completion?

Goal accuracy is binary: achieved or not. Task completion is graded 0–3 for partial progress.

jevragasbooleanmirrors ragas.agent_goal_accuracy

Agent goal accuracy

Agent goal accuracy asks whether the agent achieved the user's goal as stated in the input, judging from the final output and optionally the trajectory. It mirrors RAGAS agent_goal_accuracy as a yes/no probability.

Run this example in the console 1 jev question · answered inside one request

how it works // frame by frame

What the evaluator does

Jev reads `input` (the goal), `output` (final state or message), optionally `trajectory` and `expected` (definition of success) and returns p(achieved).

frame 1/4 · the state

state

inputRefund order 4471 and notify the customer.

outputNotified customer; refund pending approval.

expectedRefund issued and customer notified.

question → jev

Judging from `output` (and `trajectory` if present), the agent achieved the goal the user set in `input`.

p(yes)

18%

noyes

goal not achievedconfidence |p − 0.5| × 2 = 64%the 0.5 line is yours to move

exactly what jev is asked

The question, verbatim

This is the question the API sends for the example below, generated from the same code path the playground and API use. Jev sees the request fields as state and returns a probability for each outcome. Nothing is generated, so there is nothing to parse.

oknoul · p(yes)

Judging from `output` (and `trajectory` if present), the agent achieved the goal the user set in `input`; `expected` describes success if present.

when to use it

Reach for it when

End-to-end agent evaluation when the exact path does not matter.
Success-rate dashboards.

Not the right tool when

You care how the goal was reached (use trajectory_steps).
Partial credit matters (use task_completion).

Watch out for

Agents that claim success without evidence can fool any judge; include the trajectory or tool results in the state.

inputs and example

What to send

required fields

inputoutput

optional fields

expectedtrajectory

in suites

AgentEverything

result shape

score 0–1 (p of the good outcome), label “nn% yes”, confidence |p − 0.5| × 2, probabilities yes/no, passed at 0.5.

POST /api/v1/evaluateopen in console

{
  "evaluators": [
    "agent_goal"
  ],
  "input": "Refund order 4471 and notify the customer.",
  "output": "Notified customer; refund pending approval.",
  "expected": "Refund issued and customer notified."
}

expected

p(achieved) low: the refund was not issued.

rolling it up