JEVAL
jevragasscoremirrors ragas.tool_call_accuracy / uipath ToolCallArgsEvaluator (semantic)

Tool call accuracy

Tool call accuracy rates how well an agent's tool calls match the expected calls in tool choice and arguments, allowing semantically equivalent values, as a 0–3 level. It mirrors RAGAS tool_call_accuracy as a single graded judgment.

Run this example in the console 1 jev question · answered inside one request
how it works // frame by frame

What the evaluator does

Jev compares `tool_calls` with `expected_tool_calls` and rates the overall match. For one probability per expected call, use tool_call_semantic in the same request.

frame 1/4 · the state
state
tool_callssearch_flights(origin: BLR, destination: SFO, date: Nov 12, 2025)
expected_tool_callssearch_flights(from: BLR, to: SFO, date: 2025-11-12)
question → jev

Compare `tool_calls` with `expected_tool_calls`. Rate the match in tool selection and argument values, allowing equivalent formats.

ordered levels
0
wrong tools or missing
0%
1
right tools, wrong/missing args
2%
2
right tools, mostly right args
15%
3
match in tools, args and intent
83%
2.8 / 3conf 83%pass · threshold 75%
exactly what jev is asked

The question, verbatim

This is the question the API sends for the example below, generated from the same code path the playground and API use. Jev sees the request fields as state and returns a probability for each outcome. Nothing is generated, so there is nothing to parse.

qscore · ordered rubric

Compare `tool_calls` made by the agent with `expected_tool_calls`. Rate the match in tool selection and argument values, allowing equivalent formats.

  1. 0The tool calls are wrong tools or missing entirely
  2. 1Some right tools with wrong or missing arguments
  3. 2Right tools with mostly right arguments; minor mismatches
  4. 3Tool calls match the expected calls in tools, arguments, and intent
when to use it

Reach for it when

  • Agent regression tests where argument formats vary.
  • A single number for a dashboard, with tool_call_semantic for the drill-down.

Not the right tool when

  • Exact argument equality is required (tool_call_args is free and exact).

Watch out for

  • Extra unexpected calls are not penalised by this evaluator; use tool_call_count alongside.
inputs and example

What to send

required fields
tool_callsexpected_tool_calls
optional fields
input
in suites
AgentEverything
result shape

score 0–1 (expected level ÷ top level), label “x.x / n”, confidence, a probability per level, passed at 75% of the scale.

POST /api/v1/evaluateopen in console 
{
  "evaluators": [
    "tool_call_accuracy"
  ],
  "output": "booked",
  "tool_calls": [
    {
      "name": "search_flights",
      "args": {
        "origin": "BLR",
        "destination": "SFO",
        "date": "Nov 12, 2025"
      }
    }
  ],
  "expected_tool_calls": [
    {
      "name": "search_flights",
      "args": {
        "from": "BLR",
        "to": "SFO",
        "date": "2025-11-12"
      }
    }
  ]
}
expected

Level 3: same tool, equivalent arguments despite renamed keys and date format.

rolling it up

Aggregate with

One result per item is a fact; a dataset of them is a metric. These are the aggregations that fit this evaluator's output shape.

Mean score
Σ score_i / n

Mean score averages the normalised 0–1 score across items.

Pass rate
passed / total

Pass rate is the share of items whose evaluator result met its threshold.

related evaluators
tool_call_semanticjev
Each expected tool call gets a probability that an equivalent call was made (same tool, arguments equal in meaning even if formatted differently). Exact-match evaluators fail on '2025-11-12' vs 'Nov 12 2025'; this does not.
tool_call_argscode · free
Share of expected tool calls that appear in the output with exactly matching arguments.
tool_call_ordercode · free
Do the tool names appear in the expected order (as a subsequence)?
tool_call_countcode · free
Per-tool call counts match the expected counts.
questions people ask
Why not just compare tool-call JSON?
Because agents rename keys and reformat values. Exact comparison fails on `date: "Nov 12, 2025"` vs `"2025-11-12"`; a semantic judgment does not. jeval gives you both.