SQL

SQL judges whether two queries are semantically equivalent, meaning they return the same result set for the request, ignoring aliasing, formatting and column order. It mirrors autoevals Sql and returns the probability of equivalence.

What the evaluator does

Jev reads the request, the candidate query and the reference query and returns p(equivalent). It reasons about joins, predicates and aggregation rather than string similarity.

The question, verbatim

This is the question the API sends for the example below, generated from the same code path the playground and API use. Jev sees the request fields as state and returns a probability for each outcome. Nothing is generated, so there is nothing to parse.

What to send

{
  "evaluators": [
    "sql"
  ],
  "input": "Customers with more than 5 orders.",
  "output": "SELECT customer_id FROM orders WHERE COUNT(*) > 5 GROUP BY customer_id",
  "expected": "SELECT customer_id FROM orders GROUP BY customer_id HAVING COUNT(*) > 5"
}

Aggregate with

One result per item is a fact; a dataset of them is a metric. These are the aggregations that fit this evaluator's output shape.