How does JSON similarity score numbers?

By relative difference: 1849.5 vs 1849.50 is identical; 100 vs 110 scores 0.9.

code · freeuipathdeterministicmirrors uipath.eval JsonSimilarityEvaluator / autoevals JSONDiff

JSON similarity

JSON similarity computes a recursive structural similarity between the output JSON and the expected JSON, field by field, with partial credit for close numbers and strings. It mirrors UiPath's JsonSimilarityEvaluator and autoevals JSONDiff and is free.

Run this example in the console runs in code · $0

how it works // frame by frame

What the evaluator does

Parse both; for objects average over the union of keys, for arrays average position-wise, numbers by relative difference, strings by normalised edit distance. Score 0–1; pass at 0.9.

frame 1/4 · the two inputs

output

{"invoice_no":"INV-2231",
 "total":1849.5,
 "currency":"USD"}

expected

{"invoice_no":"INV-2231",
 "total":1849.50,
 "currency":"USD",
 "due":"2026-10-15"}

1.parse both documents

2.union of keys: invoice_no, total, currency, due

3.per-key similarity: 1, 1, 1, 0 (missing)

4.average

0.75score 0.75

runs in code · $0 · ~0 ms · deterministic

when to use it

Reach for it when

Structured extraction (invoices, forms, entities).
Tool-call argument objects where exact equality is too strict but semantic judgment is overkill.

Not the right tool when

Key names differ legitimately (use tool_call_semantic or llm_judge).
Either side is not valid JSON (score is 0).

Watch out for

Extra keys in the output reduce the score; decide whether that is desired.
Array order matters.

inputs and example

What to send

required fields

outputexpected

optional fields

none

in suites

CorrectnessAgentEverything

result shape

score 0–1 computed in code, label, passed at an evaluator-specific threshold, no probabilities.

POST /api/v1/evaluateopen in console

{
  "evaluators": [
    "json_similarity"
  ],
  "output": "{\"invoice_no\":\"INV-2231\",\"total\":1849.5,\"currency\":\"USD\"}",
  "expected": "{\"invoice_no\":\"INV-2231\",\"total\":1849.50,\"currency\":\"USD\",\"due\":\"2026-10-15\"}"
}

expected

Three of four fields match exactly: 0.75.

rolling it up