What does a 0.5 mean for is_xml?

A balanced XML fragment was found but the output is not entirely XML, for example markup inside an explanation.

code · freepromptfoodeterministicmirrors promptfoo is-xml / contains-xml

Is XML

Is XML checks that the output is well-formed XML: tags balance and nest, with self-closing tags allowed. Scores 1 for a whole document, 0.5 for a well-formed fragment inside prose. It mirrors promptfoo's `is-xml` and `contains-xml`.

Run this example in the console runs in code · $0

how it works // frame by frame

What the evaluator does

A tag scanner skips comments, CDATA and declarations, pushes opening tags and pops on closing tags; any mismatch fails.

frame 1/4 · the two inputs

output

<order id="77812">
  <item sku="A1" qty="2"/>
  <total>3620.00</total>
</order>

scan

push <order> · self-closing <item/> · push/pop <total> · pop </order>

1.walk the tags

2.track the open stack

3.empty stack at the end

well-formedscore 1

runs in code · $0 · ~0 ms · deterministic

when to use it

Reach for it when

Pipelines that feed model output to an XML parser.
Configuration or markup generation checks.

Not the right tool when

Schema conformance (validate against your XSD downstream).
HTML, which allows void elements (use is_html).

Watch out for

Well-formed is not valid: the right tags in the wrong places still pass.

inputs and example

What to send

required fields

output

optional fields

none

in suites

PromptfooEverything

result shape

score 0–1 computed in code, label, passed at an evaluator-specific threshold, no probabilities.

POST /api/v1/evaluateopen in console

{
  "evaluators": [
    "is_xml"
  ],
  "output": "<order id=\"77812\"><item sku=\"A1\" qty=\"2\"/><total>3620.00</total></order>"
}

expected

Balanced document: score 1.

rolling it up

Aggregate with

One result per item is a fact; a dataset of them is a metric. These are the aggregations that fit this evaluator's output shape.

Pass rate

passed / total

Pass rate is the share of items whose evaluator result met its threshold.

related evaluators

is_htmlcode · free

Output is HTML: tags balance (void elements allowed), scored 1 for a document or full fragment and 0.5 when HTML is embedded in prose.

valid_jsoncode · free

Output parses as JSON.

contains_jsoncode · free

Output contains a parseable JSON object or array somewhere in the text (prose or code fences around it are fine).

questions people ask

What does a 0.5 mean for is_xml?: A balanced XML fragment was found but the output is not entirely XML, for example markup inside an explanation.