contains_json vs valid_json?

valid_json requires the entire output to parse. contains_json accepts JSON embedded in text.

code · freepromptfoodeterministicmirrors promptfoo contains-json

Contains JSON

Contains JSON passes when a parseable JSON object or array appears anywhere in the output, even surrounded by prose or code fences. It mirrors promptfoo's `contains-json` and is the lenient sibling of valid_json.

Run this example in the console runs in code · $0

how it works // frame by frame

What the evaluator does

The first balanced `{…}` or `[…]` span is located with a small scanner that respects strings and escapes, then parsed. Score 1 if any span parses.

frame 1/4 · the two inputs

output

Here you go:
```json
{"status": "booked", "pnr": "X7K9QF"}
```

scan

first balanced { … } → JSON.parse

1.find the first balanced brace span

2.parse it

3.any success passes

object foundscore 1

runs in code · $0 · ~0 ms · deterministic

when to use it

Reach for it when

Models that wrap JSON in explanation or Markdown fences.
Extracting a structured payload from a chatty answer before validating it.

Not the right tool when

The whole output must be JSON (use valid_json).
You need schema validation (pair with json_similarity against an expected shape).

Watch out for

A JSON value inside prose can still be the wrong shape; this only checks parseability.

inputs and example

What to send

required fields

output

optional fields

none

in suites

PromptfooEverything

result shape

score 0–1 computed in code, label, passed at an evaluator-specific threshold, no probabilities.

POST /api/v1/evaluateopen in console

{
  "evaluators": [
    "contains_json"
  ],
  "output": "Here you go:\n```json\n{\"status\": \"booked\", \"pnr\": \"X7K9QF\"}\n```"
}

expected

Object found inside the fence: pass.

rolling it up

Aggregate with

One result per item is a fact; a dataset of them is a metric. These are the aggregations that fit this evaluator's output shape.

Pass rate

passed / total

Pass rate is the share of items whose evaluator result met its threshold.

related evaluators

valid_jsoncode · free

Output parses as JSON.

json_similaritycode · free

Recursive structural similarity between output JSON and expected JSON (fields, arrays, numbers, strings).

regexcode · free

Output matches the regular expression in `pattern` (JavaScript syntax; `/…/flags` form allowed).

questions people ask

contains_json vs valid_json?: valid_json requires the entire output to parse. contains_json accepts JSON embedded in text.