Does tool call order penalise extra calls?

No. It only checks that the expected names appear in order. Use tool_call_count for extras.

code · freeuipathdeterministicmirrors uipath.eval ToolCallOrderEvaluator

Tool call order

Tool call order checks that the expected tool names occur in the expected order within the actual calls, as an ordered subsequence. It mirrors UiPath's ToolCallOrderEvaluator and is free.

Run this example in the console runs in code · $0

how it works // frame by frame

What the evaluator does

Walk the actual call names; each time the next expected name appears, advance. Score is advanced / expected.

frame 1/4 · the two inputs

actual names

send_email → lookup_order

expected names

lookup_order → issue_refund → send_email

1.expect lookup_order: found at position 2

2.expect issue_refund: not found after it

3.stop

1/3score 0.33

runs in code · $0 · ~0 ms · deterministic

when to use it

Reach for it when

Workflows where sequence matters: look up before refund, refund before notify.

Not the right tool when

Order is irrelevant (use tool_call_count).

Watch out for

Extra calls between expected ones are allowed; only order of the expected names is checked.

inputs and example

What to send

required fields

tool_callsexpected_tool_calls

optional fields

none

in suites

AgentEverything

result shape

score 0–1 computed in code, label, passed at an evaluator-specific threshold, no probabilities.

POST /api/v1/evaluateopen in console

{
  "evaluators": [
    "tool_call_order"
  ],
  "output": "…",
  "tool_calls": [
    {
      "name": "send_email"
    },
    {
      "name": "lookup_order"
    }
  ],
  "expected_tool_calls": [
    {
      "name": "lookup_order"
    },
    {
      "name": "issue_refund"
    },
    {
      "name": "send_email"
    }
  ]
}

expected

Only lookup_order can be matched in order: 1/3.

rolling it up