What does a count mismatch usually mean?

A retry loop, a duplicated side effect, or a skipped step. It is one of the cheapest agent safety checks.

code · freeuipathdeterministicmirrors uipath.eval ToolCallCountEvaluator

Tool call count

Tool call count checks that each tool was called the expected number of times. It mirrors UiPath's ToolCallCountEvaluator and is free.

Run this example in the console runs in code · $0

how it works // frame by frame

What the evaluator does

Count calls per tool name on both sides; score is the share of tool names whose counts agree.

frame 1/4 · the two inputs

actual counts

issue_refund: 2

expected counts

issue_refund: 1

1.count per tool

2.compare counts per name

3.share of names that agree

0/1score 0

runs in code · $0 · ~0 ms · deterministic

when to use it

Reach for it when

Detecting retry loops and duplicate side effects (two refunds).
Ensuring a required tool was called exactly once.

Not the right tool when

Arguments matter (use tool_call_args or tool_call_semantic).

Watch out for

A tool that should never be called counts as a mismatch only if it appears in one side; include it with count 0 in expected by adding a zero-call entry upstream if needed.

inputs and example

What to send

required fields

tool_callsexpected_tool_calls

optional fields

none

in suites

AgentEverything

result shape

score 0–1 computed in code, label, passed at an evaluator-specific threshold, no probabilities.

POST /api/v1/evaluateopen in console

{
  "evaluators": [
    "tool_call_count"
  ],
  "output": "…",
  "tool_calls": [
    {
      "name": "issue_refund"
    },
    {
      "name": "issue_refund"
    }
  ],
  "expected_tool_calls": [
    {
      "name": "issue_refund"
    }
  ]
}

expected

issue_refund called twice, expected once: 0/1.

rolling it up

Aggregate with

One result per item is a fact; a dataset of them is a metric. These are the aggregations that fit this evaluator's output shape.

Pass rate

passed / total

Pass rate is the share of items whose evaluator result met its threshold.

related evaluators

tool_call_ordercode · free

Do the tool names appear in the expected order (as a subsequence)?

tool_call_argscode · free

Share of expected tool calls that appear in the output with exactly matching arguments.

questions people ask

What does a count mismatch usually mean?: A retry loop, a duplicated side effect, or a skipped step. It is one of the cheapest agent safety checks.