Tool calls (semantic, per call)

Tool calls (semantic) gives each expected tool call a probability that an equivalent call was made: same tool, and argument values that mean the same thing even if keys are renamed or values reformatted. It is the judgment version of UiPath's ToolCallArgsEvaluator, one question per call in one request.

What the evaluator does

For M expected calls jeval adds M Noul questions. Score is the share matched; details list each call with its probability.

The question, verbatim

This is the question the API sends for the example below, generated from the same code path the playground and API use. Jev sees the request fields as state and returns a probability for each outcome. Nothing is generated, so there is nothing to parse.

What to send

{
  "evaluators": [
    "tool_call_semantic"
  ],
  "output": "booked",
  "tool_calls": [
    {
      "name": "search_flights",
      "args": {
        "origin": "BLR",
        "destination": "SFO",
        "date": "Nov 12, 2025"
      }
    }
  ],
  "expected_tool_calls": [
    {
      "name": "search_flights",
      "args": {
        "from": "BLR",
        "to": "SFO",
        "date": "2025-11-12"
      }
    }
  ]
}

Aggregate with

One result per item is a fact; a dataset of them is a metric. These are the aggregations that fit this evaluator's output shape.