Why divide by max(|a|, |b|, 1)?

So tiny numbers do not produce huge relative errors and zero does not divide by zero.

code · freeautoevalsdeterministicmirrors autoevals.NumericDiff

Numeric diff

Numeric diff measures how close a numeric output is to the expected number, as 1 minus the relative difference. It mirrors autoevals NumericDiff and is free.

Run this example in the console runs in code · $0

how it works // frame by frame

What the evaluator does

Extract numbers from both strings, compute |a − b| / max(|a|, |b|, 1), subtract from 1. Pass at 0.99.

frame 1/4 · the two inputs

output

0.0302

expected

0.03024

1.parse numbers

2.|a − b| = 0.00004

3.÷ max(|a|,|b|,1) = 0.00004

4.1 − 0.00004

0.99996score 1

runs in code · $0 · ~0 ms · deterministic

when to use it

Reach for it when

Calculations, unit conversions, financial figures.
Tolerant comparison where 3.0 and 3 must match.

Not the right tool when

Numbers embedded in prose with units that change meaning (use answer_correctness).

Watch out for

Extracts the first numeric token; keep outputs to the number.

inputs and example

What to send

required fields

outputexpected

optional fields

none

in suites

CorrectnessEverything

result shape

score 0–1 computed in code, label, passed at an evaluator-specific threshold, no probabilities.

POST /api/v1/evaluateopen in console

{
  "evaluators": [
    "numeric_diff"
  ],
  "output": "0.0302",
  "expected": "0.03024"
}

expected

Relative difference 0.1%: 0.999.

rolling it up

Aggregate with

One result per item is a fact; a dataset of them is a metric. These are the aggregations that fit this evaluator's output shape.

Mean score

Σ score_i / n

Mean score averages the normalised 0–1 score across items.

Pass rate

passed / total

Pass rate is the share of items whose evaluator result met its threshold.

Mean absolute error

Σ |predicted_i − gold_i| / n

Mean absolute error measures, for graded rubrics, how far the evaluator's level is from the gold level on average.

related evaluators

exact_matchcode · free

Output equals expected after whitespace and case normalisation.

answer_correctnessjev

Factual agreement between the output and the expected answer.

questions people ask

Why divide by max(|a|, |b|, 1)?: So tiny numbers do not produce huge relative errors and zero does not divide by zero.