JEVAL
jevuipathscoremirrors uipath.eval LLMJudgeTrajectoryEvaluator / LegacyTrajectoryEvaluator

Trajectory

Trajectory rates how closely an agent's step sequence follows the expected trajectory, as a single 0–3 level. It mirrors UiPath's LLMJudgeTrajectoryEvaluator; for one probability per step use trajectory_steps.

Run this example in the console 1 jev question · answered inside one request
how it works // frame by frame

What the evaluator does

Jev compares `trajectory` with `expected_trajectory` for the task and rates adherence in steps and order.

frame 1/4 · the state
state
trajectorysearch flights → pick cheapest direct → book flight → send confirmation email
expected_trajectorysearch flights → select cheapest direct → book → email confirmation
question → jev

Compare the agent's steps in `trajectory` with `expected_trajectory`. Rate adherence in steps and order.

ordered levels
0
no resemblance
0%
1
some steps, order or key steps wrong
1%
2
most steps, sensible order
12%
3
matches steps and order
87%
2.9 / 3conf 87%pass · threshold 75%
exactly what jev is asked

The question, verbatim

This is the question the API sends for the example below, generated from the same code path the playground and API use. Jev sees the request fields as state and returns a probability for each outcome. Nothing is generated, so there is nothing to parse.

qscore · ordered rubric

Compare the agent's steps in `trajectory` with the reference steps in `expected_trajectory` for the task in `input`. Rate adherence in steps and order.

  1. 0The agent's trajectory bears no resemblance to the expected trajectory
  2. 1Some expected steps appear but order or key steps are wrong
  3. 2Most expected steps appear in a sensible order with minor deviations
  4. 3The trajectory matches the expected steps and order
when to use it

Reach for it when

  • One number per run for a trajectory dashboard.
  • Legacy compatibility with a single trajectory score.

Not the right tool when

  • You need to know which step was skipped (use trajectory_steps).

Watch out for

  • Paraphrased steps are fine; the rubric allows different wording. Extra steps are penalised only if they break the order or intent.
inputs and example

What to send

required fields
trajectoryexpected_trajectory
optional fields
input
in suites
AgentEverything
result shape

score 0–1 (expected level ÷ top level), label “x.x / n”, confidence, a probability per level, passed at 75% of the scale.

POST /api/v1/evaluateopen in console 
{
  "evaluators": [
    "trajectory"
  ],
  "output": "…",
  "trajectory": [
    "search flights",
    "pick cheapest direct",
    "book flight",
    "send confirmation email"
  ],
  "expected_trajectory": [
    "search flights",
    "select cheapest direct option",
    "book",
    "email confirmation"
  ]
}
expected

Level 3: matches in steps and order.

rolling it up

Aggregate with

One result per item is a fact; a dataset of them is a metric. These are the aggregations that fit this evaluator's output shape.

Mean score
Σ score_i / n

Mean score averages the normalised 0–1 score across items.

Pass rate
passed / total

Pass rate is the share of items whose evaluator result met its threshold.

related evaluators
trajectory_stepsjev
Instead of one prose verdict from an LLM prompt, each expected step gets its own probability that the agent performed it, plus an order judgment, in one call. Score is the share of steps performed.
agent_goaljev
Did the agent achieve the user's goal described in the input?
task_completionjev
Degree to which the requested task was completed.
questions people ask
Trajectory vs trajectory_steps?
Trajectory is one graded level for the whole run. trajectory_steps asks one question per expected step and returns a probability for each, plus an order check.