Does is_sql parse the SQL?

No. It is a conservative syntax heuristic. For correctness use execution, or the sql evaluator for equivalence against a reference.

code · freepromptfoodeterministicmirrors promptfoo is-sql / contains-sql

Is SQL

Is SQL checks that the output is a plausible SQL statement, or contains one in a ```sql block: a statement keyword, balanced parentheses and quotes. It mirrors promptfoo's `is-sql` and `contains-sql` as a syntax heuristic rather than a full parser.

Run this example in the console runs in code · $0

how it works // frame by frame

What the evaluator does

The text (or the fenced block) must start with WITH, SELECT, INSERT, UPDATE, DELETE, CREATE, ALTER, DROP, MERGE, TRUNCATE, GRANT, REVOKE or EXPLAIN, and have balanced parentheses and an even number of single quotes.

frame 1/4 · the two inputs

output

SELECT customer_id FROM orders GROUP BY customer_id HAVING COUNT(*) > 5;

checks

keyword · parens · quotes

1.starts with a statement keyword

2.parentheses balance

3.quotes pair up

sqlscore 1

runs in code · $0 · ~0 ms · deterministic

when to use it

Reach for it when

Text-to-SQL pipelines, as a free gate before execution.
Catching models that answer in prose instead of a query.

Not the right tool when

You can execute the query; execution is the real test.
Semantic equivalence to a reference (use sql).

Watch out for

A heuristic accepts some invalid SQL and could reject unusual dialect constructs; treat it as a smoke test.

inputs and example

What to send

required fields

output

optional fields

none

in suites

PromptfooEverything

result shape

score 0–1 computed in code, label, passed at an evaluator-specific threshold, no probabilities.

POST /api/v1/evaluateopen in console

{
  "evaluators": [
    "is_sql"
  ],
  "output": "SELECT customer_id FROM orders GROUP BY customer_id HAVING COUNT(*) > 5;"
}

expected

Statement keyword, balanced: pass.

rolling it up

Aggregate with

One result per item is a fact; a dataset of them is a metric. These are the aggregations that fit this evaluator's output shape.

Pass rate

passed / total

Pass rate is the share of items whose evaluator result met its threshold.

related evaluators

sqljev

Are the output and expected SQL queries semantically equivalent (same result set) for the input question?

contains_jsoncode · free

Output contains a parseable JSON object or array somewhere in the text (prose or code fences around it are fine).

regexcode · free

Output matches the regular expression in `pattern` (JavaScript syntax; `/…/flags` form allowed).

questions people ask

Does is_sql parse the SQL?: No. It is a conservative syntax heuristic. For correctness use execution, or the sql evaluator for equivalence against a reference.