Does contains_all give partial credit?

The score is the fraction of values found, so yes for averaging; the pass flag requires all of them.

code · freepromptfoodeterministicmirrors promptfoo contains-all / icontains-all

Contains all

Contains all checks that every string in `values` appears in the output, case-insensitively, and scores the share present. It mirrors promptfoo's `contains-all` and `icontains-all`.

Run this example in the console runs in code · $0

how it works // frame by frame

What the evaluator does

Each value is searched for in the normalised output. Score is present ÷ total; pass requires all of them.

frame 1/4 · the two inputs

output

Refunds are issued within 5-7 business days after we receive the returned item.

values

refund · business days · receive

1.normalise

2.search each value

3.share present

3/3score 1

runs in code · $0 · ~0 ms · deterministic

when to use it

Reach for it when

Checklists: every required disclaimer, field name or keyword must appear.
Coverage of key facts in a summary when a reference list exists.

Not the right tool when

Paraphrase is acceptable (use faithfulness against the facts as context, or llm_judge).

Watch out for

Long value lists make the check brittle; keep values to the words that truly must appear.

inputs and example

What to send

required fields

outputvalues

optional fields

none

in suites

PromptfooEverything

result shape

score 0–1 computed in code, label, passed at an evaluator-specific threshold, no probabilities.

POST /api/v1/evaluateopen in console

{
  "evaluators": [
    "contains_all"
  ],
  "output": "Refunds are issued within 5-7 business days after we receive the returned item.",
  "values": [
    "refund",
    "business days",
    "receive"
  ]
}

expected

3/3 present: pass.

rolling it up