Tool call F1
Tool call F1 compares the multiset of tool names an agent called with the names expected and reports F1, precision and recall. It mirrors promptfoo's `tool-call-f1` and passes at 0.8.
What the evaluator does
Counts per tool name on both sides; true positives are the clipped overlap. Precision divides by calls made, recall by calls expected.
lookup_order · issue_refund · issue_refund
lookup_order · issue_refund · send_email
Reach for it when
- Agent runs where the set of tools matters more than argument values or order.
- A single number for tool-selection quality on a dashboard.
Not the right tool when
- Arguments must match (use tool_call_args or tool_call_semantic).
- Order matters (use tool_call_order).
Watch out for
- Extra calls lower precision, missing calls lower recall; the label shows both so you know which.
What to send
score 0–1 computed in code, label, passed at an evaluator-specific threshold, no probabilities.
{
"evaluators": [
"tool_call_f1"
],
"output": "done",
"tool_calls": [
{
"name": "lookup_order"
},
{
"name": "issue_refund"
},
{
"name": "issue_refund"
}
],
"expected_tool_calls": [
{
"name": "lookup_order"
},
{
"name": "issue_refund"
},
{
"name": "send_email"
}
]
}2 matched of 3 called / 3 expected: F1 0.67.
Aggregate with
One result per item is a fact; a dataset of them is a metric. These are the aggregations that fit this evaluator's output shape.
Precision is the share of predicted positives that were really positive; recall is the share of real positives the evaluator caught; F1 is their harmonic mean, which is only high when both are.
Mean score averages the normalised 0–1 score across items.
- How does tool_call_f1 treat repeated calls?
- As a multiset: two issue_refund calls against one expected count as one match and one extra.