How does trajectory_steps handle paraphrased steps?

Each question asks whether the trajectory accomplishes the expected step allowing different wording, so “issue refund of $89.00” matches “issue refund”.

jevuipathbooleanper stepmirrors uipath.eval LLMJudgeTrajectoryEvaluator, re-expressed as one probability per expected step

Trajectory steps (per step)

Trajectory steps re-expresses UiPath's trajectory judge as one probability per expected step (“did the agent accomplish this step?”) plus an order judgment, all in one Jev request. The score is the share of steps performed, weighted by order, and the details name the missing steps.

Run this example in the console 4 jev questions · answered inside one request

how it works // frame by frame

What the evaluator does

For K expected steps jeval adds K Noul questions and one order question to the request. Each step gets p(performed); the score is (performed / K) × (0.7 + 0.3 × p(order)).

frame 1/4 · expected steps

expected trajectory

1look up order98%

2issue refund96%

3notify customer5%

actual trajectory

1look up order 4471

2issue refund of $89.00

“does the trajectory accomplish expected step i?” → one p(yes) per step, plus one order question

2/3 steps · order 97% → score 0.65 · “notify customer” missing

exactly what jev is asked

The question, verbatim

This is the question the API sends for the example below, generated from the same code path the playground and API use. Jev sees the request fields as state and returns a probability for each outcome. Nothing is generated, so there is nothing to parse.

s0noul · p(yes)

The agent's `trajectory` includes a step that accomplishes `expected_trajectory[0]`, allowing different wording.

s1noul · p(yes)

The agent's `trajectory` includes a step that accomplishes `expected_trajectory[1]`, allowing different wording.

s2noul · p(yes)

The agent's `trajectory` includes a step that accomplishes `expected_trajectory[2]`, allowing different wording.

ordernoul · p(yes)

The steps in `trajectory` occur in the same order as the corresponding steps in `expected_trajectory`.

when to use it

Reach for it when

Agent evaluation where you need to know which step failed.
Comparing agent versions step by step.
Replacing a prose trajectory verdict with numbers you can chart.

Not the right tool when

The expected trajectory is not known (use agent_goal or task_completion).

Watch out for

Write expected steps as outcomes (“book the flight”) rather than tool names; paraphrase tolerance works on intent.

inputs and example

What to send

required fields

trajectoryexpected_trajectory

optional fields

input

in suites

AgentEverything

result shape

score 0–1 (p of the good outcome), label “nn% yes”, confidence |p − 0.5| × 2, probabilities yes/no, passed at 0.5. Plus a details array with one row per step.

POST /api/v1/evaluateopen in console

{
  "evaluators": [
    "trajectory_steps"
  ],
  "input": "Refund order 4471 and notify the customer.",
  "output": "done",
  "trajectory": [
    "look up order 4471",
    "issue refund of $89.00"
  ],
  "expected_trajectory": [
    "look up order",
    "issue refund",
    "notify customer"
  ]
}

expected

Steps 1 and 2 performed, step 3 missing: 2/3.

rolling it up

Aggregate with

One result per item is a fact; a dataset of them is a metric. These are the aggregations that fit this evaluator's output shape.

Precision, recall and F1

P = TP/(TP+FP) · R = TP/(TP+FN) · F1 = 2PR/(P+R)

Precision is the share of predicted positives that were really positive; recall is the share of real positives the evaluator caught; F1 is their harmonic mean, which is only high when both are.

Mean score

Σ score_i / n

Mean score averages the normalised 0–1 score across items.

related evaluators

trajectoryjev

How closely the agent's step sequence follows the expected trajectory.

tool_call_semanticjev

Each expected tool call gets a probability that an equivalent call was made (same tool, arguments equal in meaning even if formatted differently). Exact-match evaluators fail on '2025-11-12' vs 'Nov 12 2025'; this does not.

agent_goaljev

Did the agent achieve the user's goal described in the input?

questions people ask

How does trajectory_steps handle paraphrased steps?: Each question asks whether the trajectory accomplishes the expected step allowing different wording, so “issue refund of $89.00” matches “issue refund”.