From 23da66d09b83655e37237e7b33793cc485168b33 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Wed, 15 Jul 2026 13:33:40 -0600 Subject: [PATCH 1/5] docs(sql): add 5.2 SQL engine performance guidance Document the new Resource-API SQL engine landing in 5.2 (harper #1285): which queries run index-served/streaming vs. which fall back to the legacy full-scan path, and the auto-fallback contract. Replace the blanket "SQL not recommended" warning with accurate, nuanced guidance. Co-Authored-By: Claude Opus 4.8 (1M context) --- reference/operations-api/sql.md | 56 +++++++++++++++++++++++++++++++-- 1 file changed, 54 insertions(+), 2 deletions(-) diff --git a/reference/operations-api/sql.md b/reference/operations-api/sql.md index d27fec67..b1dd6ad1 100644 --- a/reference/operations-api/sql.md +++ b/reference/operations-api/sql.md @@ -11,8 +11,10 @@ title: SQL -:::warning -SQL querying is not recommended for production use or on large tables. SQL queries often do not utilize indexes and are not optimized for performance. Use the [REST interface](../rest/overview.md) for production data access — it provides a more stable, secure, and performant interface. SQL is intended for ad-hoc data investigation and administrative queries. +:::info Harper 5.2 introduces a new SQL engine +As of Harper 5.2, SQL runs on a new engine built directly on Harper's indexed storage and Resource API. Queries that can be planned against an index now execute as streaming, index-driven operations with memory bounded by the size of the result — not the size of the table. Queries the engine can't plan against an index automatically fall back to the legacy execution path, returning the same results but with the older full-scan cost. See [Query Performance](#query-performance) for how to keep your queries on the fast path. + +For your most performance-critical production read paths, the [REST interface](../rest/overview.md) remains the recommended choice — it provides a stable, cacheable, index-driven contract. SQL is well suited to ad-hoc investigation, administrative queries, joins, and reporting. ::: Harper includes a SQL interface supporting SELECT, INSERT, UPDATE, and DELETE operations. Tables are referenced using `database.table` notation (e.g., `dev.dog`). @@ -97,6 +99,56 @@ ORDER BY d.dog_name --- +## Query Performance + +Harper 5.2's SQL engine maps SQL clauses onto Harper's native indexed storage. When a query can be planned against an index, it runs as a streaming, index-driven operation with memory bounded by the size of the result — not the size of the table. When it can't, the engine falls back to the legacy path, which fetches candidate rows and evaluates the query in memory, with cost and memory growing with the number of rows scanned. + +On any non-trivial table the difference between these two paths is large. The rules below keep your queries on the fast path. + +### Index your predicates, sorts, and join keys + +The single most important factor is whether the attributes you filter, sort, and join on are indexed. An equality or range filter on an indexed attribute becomes an index lookup; the same filter on an un-indexed attribute forces a full scan. Define indexes on the attributes your queries actually constrain. + +### Queries that run efficiently (index-served) + +These map directly to indexed storage operations: + +- **Primary-key lookups** — `WHERE id = ...`, `WHERE id IN (...)`. +- **Equality and `IN` on an indexed attribute** — `WHERE status = 'active'`, `WHERE breed_id IN (1, 2, 3)`. +- **Range predicates on an indexed attribute** — `<`, `<=`, `>`, `>=`, and `BETWEEN` (compiled to a bounded index range). +- **Prefix `LIKE`** — `WHERE name LIKE 'Har%'` compiles to an indexed `starts_with` scan. +- **`ORDER BY` on an indexed attribute** — served directly from index order, with no separate in-memory sort; `LIMIT`/`OFFSET` push into the scan so only the rows you return are read. +- **`GROUP BY` on an indexed attribute** — aggregates stream group-by-group with O(1) memory per group. +- **`INNER`/`LEFT JOIN` on an indexed join key** — executed as an index-nested-loop join (stream the outer table, probe the inner by index). Keep the join key indexed on the inner (probed) table. +- **Column projections** — selecting a subset of columns pushes down so only those attributes are read. + +### Queries that fall back to a full scan (avoid on large tables) + +These can't be served from an index. They still return correct results, but the engine falls back to the legacy full-scan path, so cost and memory grow with table size: + +- **Filters on un-indexed attributes** — any equality, range, or `IN` on an attribute with no index. +- **`OR` across different attributes** — e.g. `WHERE a = 1 OR b = 2`. (An `IN` list on a _single_ indexed attribute is fine.) +- **Suffix / substring `LIKE`** — `LIKE '%x'` and `LIKE '%x%'` (only prefix `LIKE 'x%'` is index-served). A suffix/contains `LIKE` combined with an indexed predicate is applied as a cheap residual filter on the indexed result, so pair it with an indexed condition. +- **`!=` / `<>` and `NOT` negations** — matching "everything except" a value can't use an index. +- **`ORDER BY` with no indexed filter to drive it** — a table-wide ordered scan on an un-indexed sort key. +- **`SEARCH_JSON`** — evaluates JSONata over each candidate row's nested JSON; it is not index-backed. Narrow the candidate set with an indexed `WHERE` condition first. +- **`RIGHT`/`FULL OUTER JOIN`, no-`WHERE` `UPDATE`/`DELETE`, `COUNT(DISTINCT ...)`** — the new engine doesn't plan these yet, so they run on the legacy engine (correct results, legacy cost). + +Note that `UNION`, sub-`SELECT`, and `INSERT … SELECT` are not supported by either engine — see the [Features Matrix](#features-matrix). + +### The fallback contract + +The engine runs in `auto` mode by default: it executes every query it can plan on the new engine and transparently falls back to the legacy engine for anything it can't, so **no query changes its results** — only its performance profile. Fallbacks are logged (`sql-engine v2 fallback: ...`) with the reason and the offending column, which is a useful signal for finding queries worth an added index or a reshape. + +### Practical guidance + +- Index every attribute you filter, sort, or join on. +- Prefer prefix `LIKE 'x%'` over `LIKE '%x%'`; back a substring search with a separate indexed predicate, or model it as a dedicated indexed attribute. +- Constrain the row set with an indexed `WHERE` before leaning on `ORDER BY`, `GROUP BY`, joins, or `SEARCH_JSON`. +- Reach for the [REST interface](../rest/overview.md) on your hottest production read paths for a cacheable, index-driven contract; keep SQL for ad-hoc investigation, joins, and reporting. + +--- + ## Features Matrix | INSERT | | From 14d45235c835fa62ad84fa2d265794ee96688349 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Wed, 15 Jul 2026 14:13:30 -0600 Subject: [PATCH 2/5] =?UTF-8?q?docs(sql):=20correct=20OR=20handling=20?= =?UTF-8?q?=E2=80=94=20indexed=20OR=20is=20an=20index=20union,=20not=20a?= =?UTF-8?q?=20full=20scan?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Table.search executes a top-level OR as a union of per-condition index searches (resources/search.ts), and the new engine pushes an OR group down when every branch is index-driving (whereToConditions conditionUsesIndex). So `a = 1 OR b = 2` with both attributes indexed is index-served; only an OR with an un-indexed / non-index-comparator branch falls back to a scan. Co-Authored-By: Claude Opus 4.8 (1M context) --- reference/operations-api/sql.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/reference/operations-api/sql.md b/reference/operations-api/sql.md index b1dd6ad1..4ed66c8c 100644 --- a/reference/operations-api/sql.md +++ b/reference/operations-api/sql.md @@ -116,6 +116,7 @@ These map directly to indexed storage operations: - **Primary-key lookups** — `WHERE id = ...`, `WHERE id IN (...)`. - **Equality and `IN` on an indexed attribute** — `WHERE status = 'active'`, `WHERE breed_id IN (1, 2, 3)`. - **Range predicates on an indexed attribute** — `<`, `<=`, `>`, `>=`, and `BETWEEN` (compiled to a bounded index range). +- **`OR` of indexed predicates** — `WHERE a = 1 OR b = 2` is served as a union of index scans, provided **every** branch is an index-driving predicate on an indexed attribute. - **Prefix `LIKE`** — `WHERE name LIKE 'Har%'` compiles to an indexed `starts_with` scan. - **`ORDER BY` on an indexed attribute** — served directly from index order, with no separate in-memory sort; `LIMIT`/`OFFSET` push into the scan so only the rows you return are read. - **`GROUP BY` on an indexed attribute** — aggregates stream group-by-group with O(1) memory per group. @@ -127,8 +128,8 @@ These map directly to indexed storage operations: These can't be served from an index. They still return correct results, but the engine falls back to the legacy full-scan path, so cost and memory grow with table size: - **Filters on un-indexed attributes** — any equality, range, or `IN` on an attribute with no index. -- **`OR` across different attributes** — e.g. `WHERE a = 1 OR b = 2`. (An `IN` list on a _single_ indexed attribute is fine.) -- **Suffix / substring `LIKE`** — `LIKE '%x'` and `LIKE '%x%'` (only prefix `LIKE 'x%'` is index-served). A suffix/contains `LIKE` combined with an indexed predicate is applied as a cheap residual filter on the indexed result, so pair it with an indexed condition. +- **`OR` where a branch isn't index-driving** — e.g. `WHERE a = 1 OR b = 2` when `b` is un-indexed, or `WHERE a = 1 OR name LIKE '%x%'`. A single un-indexed (or non-index-comparator) branch taints the whole disjunction and forces a scan. When **every** branch is an indexed, index-driving predicate, the `OR` stays on the fast path (see above). +- **Suffix / substring `LIKE`** — `LIKE '%x'` and `LIKE '%x%'` (only prefix `LIKE 'x%'` is index-served). A suffix/contains `LIKE` combined (via `AND`) with an indexed predicate is applied as a cheap residual filter on the indexed result, so pair it with an indexed condition. - **`!=` / `<>` and `NOT` negations** — matching "everything except" a value can't use an index. - **`ORDER BY` with no indexed filter to drive it** — a table-wide ordered scan on an un-indexed sort key. - **`SEARCH_JSON`** — evaluates JSONata over each candidate row's nested JSON; it is not index-backed. Narrow the candidate set with an indexed `WHERE` condition first. From a19e1a39d499f7cc6087e42e38430c401caeca07 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Wed, 15 Jul 2026 15:28:48 -0600 Subject: [PATCH 3/5] docs(sql): document 5.2 engine behavior changes + reserved-word aliases Adds a 'Behavior changes from the legacy engine' subsection covering the standard-SQL corrections the new engine applies to queries it plans (empty-set aggregates -> NULL, string MIN/MAX, NULL!=NULL join keys, explicit-null expression results), and scopes the fallback-contract 'results don't change' promise to genuine fallbacks. Adds TOTAL to the reserved-word list and notes that reserved words used as AS aliases must be quoted (e.g. AS `total`). Sourced from the sql-engine-v2 QA differential wave (findings D-217, F-143). Co-Authored-By: Claude Opus 4.8 (1M context) --- reference/operations-api/sql.md | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/reference/operations-api/sql.md b/reference/operations-api/sql.md index 4ed66c8c..312bfc2f 100644 --- a/reference/operations-api/sql.md +++ b/reference/operations-api/sql.md @@ -139,7 +139,18 @@ Note that `UNION`, sub-`SELECT`, and `INSERT … SELECT` are not supported by ei ### The fallback contract -The engine runs in `auto` mode by default: it executes every query it can plan on the new engine and transparently falls back to the legacy engine for anything it can't, so **no query changes its results** — only its performance profile. Fallbacks are logged (`sql-engine v2 fallback: ...`) with the reason and the offending column, which is a useful signal for finding queries worth an added index or a reshape. +The engine runs in `auto` mode by default: it executes every query it can plan on the new engine and transparently falls back to the legacy engine for anything it can't. A query that falls back returns exactly what the legacy engine would — **its results don't change, only its performance profile.** Fallbacks are logged (`sql-engine v2 fallback: ...`) with the reason and the offending column, which is a useful signal for finding queries worth an added index or a reshape. (For queries the new engine _does_ plan, it corrects a few long-standing legacy quirks — see [Behavior changes from the legacy engine](#behavior-changes-from-the-legacy-engine).) + +### Behavior changes from the legacy engine + +For queries the new engine plans (the default under `auto`), a handful of non-standard legacy behaviors are corrected, so you may see different — standard-SQL-conformant — results after upgrading: + +- **Aggregates over an empty result set** return `NULL` for `SUM`, `AVG`, `MIN`, and `MAX` (and `0` for `COUNT`), per the SQL standard. The legacy engine returned `0` for `SUM` and omitted `MIN`/`MAX` from the response entirely. +- **`MIN`/`MAX` over text columns** return the lexicographically smallest/largest value. The legacy engine produced no result for string `MIN`/`MAX`. +- **`NULL` never equals `NULL` in a join key.** Rows whose join key is `NULL` on either side no longer match each other (three-valued logic). The legacy engine incorrectly joined `NULL` to `NULL`. +- **A `NULL`-valued expression result is returned as an explicit `null`** rather than omitted from the row. A computed column or `COALESCE` that evaluates to `NULL` — or `UPPER()` — now yields `null` (the legacy engine dropped the key, and `UPPER` of a `NULL` returned the literal string `"NULL"`). + +These are deliberate corrections toward standard SQL, applied when the new engine serves the query — not fallbacks. If your application depended on a legacy quirk, adjust for the standard behavior. ### Practical guidance @@ -383,16 +394,17 @@ Geospatial data must be stored using the [GeoJSON standard](https://geojson.org/ ## Reserved Words -If a database, table, or attribute name conflicts with a reserved word, wrap it in backticks or brackets: +If a database, table, attribute, or column-alias name conflicts with a reserved word, wrap it in backticks or brackets. This applies to `AS` aliases too: an unquoted reserved word such as `AS total` fails to parse — quote it, as in the last example below. ```sql SELECT * FROM data.`ASSERT` SELECT * FROM data.[ASSERT] +SELECT SUM(qty) AS `total` FROM data.dog ```
Full reserved word list -ABSOLUTE, ACTION, ADD, AGGR, ALL, ALTER, AND, ANTI, ANY, APPLY, ARRAY, AS, ASSERT, ASC, ATTACH, AUTOINCREMENT, AUTO_INCREMENT, AVG, BEGIN, BETWEEN, BREAK, BY, CALL, CASE, CAST, CHECK, CLASS, CLOSE, COLLATE, COLUMN, COLUMNS, COMMIT, CONSTRAINT, CONTENT, CONTINUE, CONVERT, CORRESPONDING, COUNT, CREATE, CROSS, CUBE, CURRENT_TIMESTAMP, CURSOR, DATABASE, DECLARE, DEFAULT, DELETE, DELETED, DESC, DETACH, DISTINCT, DOUBLEPRECISION, DROP, ECHO, EDGE, END, ENUM, ELSE, EXCEPT, EXISTS, EXPLAIN, FALSE, FETCH, FIRST, FOREIGN, FROM, GO, GRAPH, GROUP, GROUPING, HAVING, HDB_HASH, HELP, IF, IDENTITY, IS, IN, INDEX, INNER, INSERT, INSERTED, INTERSECT, INTO, JOIN, KEY, LAST, LET, LEFT, LIKE, LIMIT, LOOP, MATCHED, MATRIX, MAX, MERGE, MIN, MINUS, MODIFY, NATURAL, NEXT, NEW, NOCASE, NO, NOT, NULL, OFF, ON, ONLY, OFFSET, OPEN, OPTION, OR, ORDER, OUTER, OVER, PATH, PARTITION, PERCENT, PLAN, PRIMARY, PRINT, PRIOR, QUERY, READ, RECORDSET, REDUCE, REFERENCES, RELATIVE, REPLACE, REMOVE, RENAME, REQUIRE, RESTORE, RETURN, RETURNS, RIGHT, ROLLBACK, ROLLUP, ROW, SCHEMA, SCHEMAS, SEARCH, SELECT, SEMI, SET, SETS, SHOW, SOME, SOURCE, STRATEGY, STORE, SYSTEM, SUM, TABLE, TABLES, TARGET, TEMP, TEMPORARY, TEXTSTRING, THEN, TIMEOUT, TO, TOP, TRAN, TRANSACTION, TRIGGER, TRUE, TRUNCATE, UNION, UNIQUE, UPDATE, USE, USING, VALUE, VERTEX, VIEW, WHEN, WHERE, WHILE, WITH, WORK +ABSOLUTE, ACTION, ADD, AGGR, ALL, ALTER, AND, ANTI, ANY, APPLY, ARRAY, AS, ASSERT, ASC, ATTACH, AUTOINCREMENT, AUTO_INCREMENT, AVG, BEGIN, BETWEEN, BREAK, BY, CALL, CASE, CAST, CHECK, CLASS, CLOSE, COLLATE, COLUMN, COLUMNS, COMMIT, CONSTRAINT, CONTENT, CONTINUE, CONVERT, CORRESPONDING, COUNT, CREATE, CROSS, CUBE, CURRENT_TIMESTAMP, CURSOR, DATABASE, DECLARE, DEFAULT, DELETE, DELETED, DESC, DETACH, DISTINCT, DOUBLEPRECISION, DROP, ECHO, EDGE, END, ENUM, ELSE, EXCEPT, EXISTS, EXPLAIN, FALSE, FETCH, FIRST, FOREIGN, FROM, GO, GRAPH, GROUP, GROUPING, HAVING, HDB_HASH, HELP, IF, IDENTITY, IS, IN, INDEX, INNER, INSERT, INSERTED, INTERSECT, INTO, JOIN, KEY, LAST, LET, LEFT, LIKE, LIMIT, LOOP, MATCHED, MATRIX, MAX, MERGE, MIN, MINUS, MODIFY, NATURAL, NEXT, NEW, NOCASE, NO, NOT, NULL, OFF, ON, ONLY, OFFSET, OPEN, OPTION, OR, ORDER, OUTER, OVER, PATH, PARTITION, PERCENT, PLAN, PRIMARY, PRINT, PRIOR, QUERY, READ, RECORDSET, REDUCE, REFERENCES, RELATIVE, REPLACE, REMOVE, RENAME, REQUIRE, RESTORE, RETURN, RETURNS, RIGHT, ROLLBACK, ROLLUP, ROW, SCHEMA, SCHEMAS, SEARCH, SELECT, SEMI, SET, SETS, SHOW, SOME, SOURCE, STRATEGY, STORE, SYSTEM, SUM, TABLE, TABLES, TARGET, TEMP, TEMPORARY, TEXTSTRING, THEN, TIMEOUT, TO, TOP, TOTAL, TRAN, TRANSACTION, TRIGGER, TRUE, TRUNCATE, UNION, UNIQUE, UPDATE, USE, USING, VALUE, VERTEX, VIEW, WHEN, WHERE, WHILE, WITH, WORK
From e16f88bb1565a4863c5913556ec99ecd42269b6b Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Wed, 15 Jul 2026 16:48:19 -0600 Subject: [PATCH 4/5] docs(sql): correct fallback-log string; ORDER/GROUP BY WHERE-driver requirement; OFFSET behavior change MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit QA verification against the engine (sql-engine-v2 @ 8a46e0302) found three inaccuracies: - The fallback log line is 'SQL engine v2 fallback:' (router.ts), not 'sql-engine v2 fallback:' — the documented grep string matched nothing. - ORDER BY / GROUP BY on an indexed attribute run on the legacy path unless an index-driving WHERE condition is present (validateScannable requires a scan driver) — state the requirement instead of promising them unconditionally. - Behavior changes: OFFSET past the end of a result now correctly returns no rows (legacy could still return rows). Co-Authored-By: Claude Fable 5 --- reference/operations-api/sql.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/reference/operations-api/sql.md b/reference/operations-api/sql.md index 312bfc2f..a376e15d 100644 --- a/reference/operations-api/sql.md +++ b/reference/operations-api/sql.md @@ -118,8 +118,8 @@ These map directly to indexed storage operations: - **Range predicates on an indexed attribute** — `<`, `<=`, `>`, `>=`, and `BETWEEN` (compiled to a bounded index range). - **`OR` of indexed predicates** — `WHERE a = 1 OR b = 2` is served as a union of index scans, provided **every** branch is an index-driving predicate on an indexed attribute. - **Prefix `LIKE`** — `WHERE name LIKE 'Har%'` compiles to an indexed `starts_with` scan. -- **`ORDER BY` on an indexed attribute** — served directly from index order, with no separate in-memory sort; `LIMIT`/`OFFSET` push into the scan so only the rows you return are read. -- **`GROUP BY` on an indexed attribute** — aggregates stream group-by-group with O(1) memory per group. +- **`ORDER BY` on an indexed attribute, combined with an indexed `WHERE`** — served directly from index order, with no separate in-memory sort; `LIMIT`/`OFFSET` push into the scan so only the rows you return are read. Without an index-driving `WHERE` condition, an `ORDER BY` currently runs on the legacy path. +- **`GROUP BY` on an indexed attribute, combined with an indexed `WHERE`** — aggregates stream group-by-group with O(1) memory per group (memory is bounded by the group count; every matching row is still read). Without an index-driving `WHERE` condition, a `GROUP BY` currently runs on the legacy path. - **`INNER`/`LEFT JOIN` on an indexed join key** — executed as an index-nested-loop join (stream the outer table, probe the inner by index). Keep the join key indexed on the inner (probed) table. - **Column projections** — selecting a subset of columns pushes down so only those attributes are read. @@ -139,7 +139,7 @@ Note that `UNION`, sub-`SELECT`, and `INSERT … SELECT` are not supported by ei ### The fallback contract -The engine runs in `auto` mode by default: it executes every query it can plan on the new engine and transparently falls back to the legacy engine for anything it can't. A query that falls back returns exactly what the legacy engine would — **its results don't change, only its performance profile.** Fallbacks are logged (`sql-engine v2 fallback: ...`) with the reason and the offending column, which is a useful signal for finding queries worth an added index or a reshape. (For queries the new engine _does_ plan, it corrects a few long-standing legacy quirks — see [Behavior changes from the legacy engine](#behavior-changes-from-the-legacy-engine).) +The engine runs in `auto` mode by default: it executes every query it can plan on the new engine and transparently falls back to the legacy engine for anything it can't. A query that falls back returns exactly what the legacy engine would — **its results don't change, only its performance profile.** Fallbacks are logged (`SQL engine v2 fallback: ...`) with the reason, which is a useful signal for finding queries worth an added index or a reshape. (For queries the new engine _does_ plan, it corrects a few long-standing legacy quirks — see [Behavior changes from the legacy engine](#behavior-changes-from-the-legacy-engine).) ### Behavior changes from the legacy engine @@ -148,6 +148,7 @@ For queries the new engine plans (the default under `auto`), a handful of non-st - **Aggregates over an empty result set** return `NULL` for `SUM`, `AVG`, `MIN`, and `MAX` (and `0` for `COUNT`), per the SQL standard. The legacy engine returned `0` for `SUM` and omitted `MIN`/`MAX` from the response entirely. - **`MIN`/`MAX` over text columns** return the lexicographically smallest/largest value. The legacy engine produced no result for string `MIN`/`MAX`. - **`NULL` never equals `NULL` in a join key.** Rows whose join key is `NULL` on either side no longer match each other (three-valued logic). The legacy engine incorrectly joined `NULL` to `NULL`. +- **`OFFSET` past the end of a result returns no rows.** The legacy engine could still return rows when the `OFFSET` exceeded the number of matches. - **A `NULL`-valued expression result is returned as an explicit `null`** rather than omitted from the row. A computed column or `COALESCE` that evaluates to `NULL` — or `UPPER()` — now yields `null` (the legacy engine dropped the key, and `UPPER` of a `NULL` returned the literal string `"NULL"`). These are deliberate corrections toward standard SQL, applied when the new engine serves the query — not fallbacks. If your application depended on a legacy quirk, adjust for the standard behavior. From 82e074c37d8ac749e0a9977c01e13efeb1349116 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 20 Jul 2026 08:17:33 -0600 Subject: [PATCH 5/5] =?UTF-8?q?docs(sql):=20address=20review=20feedback=20?= =?UTF-8?q?=E2=80=94=20ORDER=20BY=20PK=20exception,=20AND-residual=20filte?= =?UTF-8?q?rs,=20over-indexing,=20observability,=20callout?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - ORDER BY bullet: correct the WHERE-driver requirement to note the primary-key exception (validateScannable/sortDrivesIndex only checks isPrimaryKey; a no-WHERE ORDER BY on the PK is index-served, only a secondary indexed attribute falls back) — cb1kenobi - Un-indexed-filter bullet: clarify that ANDed with an indexed predicate it's a cheap residual filter and the query stays index-served; only a standalone or OR-joined un-indexed filter forces a scan — kylebernhardy - Practical guidance: note the storage/write-cost tradeoff of indexing every attribute — kylebernhardy - Intro info box: point to the `SQL engine v2 fallback: ...` log line as the observability signal for query planning — kylebernhardy - Query Performance intro: add a callout emphasizing that fallback means reading every row in the table, not just "slower" — kylebernhardy Verified against harper sqlEngine/optimizer/{validateScannable,whereToConditions}.ts and unitTests/sqlEngine/pipeline.test.js (D-219 PK exception tests). Co-Authored-By: Claude Sonnet 5 --- reference/operations-api/sql.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/reference/operations-api/sql.md b/reference/operations-api/sql.md index a376e15d..12a5ce71 100644 --- a/reference/operations-api/sql.md +++ b/reference/operations-api/sql.md @@ -12,7 +12,7 @@ title: SQL :::info Harper 5.2 introduces a new SQL engine -As of Harper 5.2, SQL runs on a new engine built directly on Harper's indexed storage and Resource API. Queries that can be planned against an index now execute as streaming, index-driven operations with memory bounded by the size of the result — not the size of the table. Queries the engine can't plan against an index automatically fall back to the legacy execution path, returning the same results but with the older full-scan cost. See [Query Performance](#query-performance) for how to keep your queries on the fast path. +As of Harper 5.2, SQL runs on a new engine built directly on Harper's indexed storage and Resource API. Queries that can be planned against an index now execute as streaming, index-driven operations with memory bounded by the size of the result — not the size of the table. Queries the engine can't plan against an index automatically fall back to the legacy execution path, returning the same results but with the older full-scan cost — each fallback is logged (`SQL engine v2 fallback: ...`) with the reason, so you have direct visibility into which queries aren't being planned. See [Query Performance](#query-performance) for how to keep your queries on the fast path. For your most performance-critical production read paths, the [REST interface](../rest/overview.md) remains the recommended choice — it provides a stable, cacheable, index-driven contract. SQL is well suited to ad-hoc investigation, administrative queries, joins, and reporting. ::: @@ -103,7 +103,11 @@ ORDER BY d.dog_name Harper 5.2's SQL engine maps SQL clauses onto Harper's native indexed storage. When a query can be planned against an index, it runs as a streaming, index-driven operation with memory bounded by the size of the result — not the size of the table. When it can't, the engine falls back to the legacy path, which fetches candidate rows and evaluates the query in memory, with cost and memory growing with the number of rows scanned. -On any non-trivial table the difference between these two paths is large. The rules below keep your queries on the fast path. +On any non-trivial table the difference between these two paths is large. + +:::caution Full-scan fallback cost +A fallback isn't just "slower" — it means every row in the table is read and evaluated in memory for that query, with no bound from the result size. On a large table this can be the difference between a sub-millisecond lookup and a scan that reads millions of rows. The rules below keep your queries on the fast path; see [Queries that fall back to a full scan](#queries-that-fall-back-to-a-full-scan-avoid-on-large-tables) for what to avoid. +::: ### Index your predicates, sorts, and join keys @@ -118,7 +122,7 @@ These map directly to indexed storage operations: - **Range predicates on an indexed attribute** — `<`, `<=`, `>`, `>=`, and `BETWEEN` (compiled to a bounded index range). - **`OR` of indexed predicates** — `WHERE a = 1 OR b = 2` is served as a union of index scans, provided **every** branch is an index-driving predicate on an indexed attribute. - **Prefix `LIKE`** — `WHERE name LIKE 'Har%'` compiles to an indexed `starts_with` scan. -- **`ORDER BY` on an indexed attribute, combined with an indexed `WHERE`** — served directly from index order, with no separate in-memory sort; `LIMIT`/`OFFSET` push into the scan so only the rows you return are read. Without an index-driving `WHERE` condition, an `ORDER BY` currently runs on the legacy path. +- **`ORDER BY` on an indexed attribute, combined with an indexed `WHERE`** — served directly from index order, with no separate in-memory sort; `LIMIT`/`OFFSET` push into the scan so only the rows you return are read. Without an index-driving `WHERE` condition, an `ORDER BY` on a _secondary_ indexed attribute runs on the legacy path — except when sorting on the table's **primary key**, which drives the scan from the primary key's natural index order even with no `WHERE` clause. - **`GROUP BY` on an indexed attribute, combined with an indexed `WHERE`** — aggregates stream group-by-group with O(1) memory per group (memory is bounded by the group count; every matching row is still read). Without an index-driving `WHERE` condition, a `GROUP BY` currently runs on the legacy path. - **`INNER`/`LEFT JOIN` on an indexed join key** — executed as an index-nested-loop join (stream the outer table, probe the inner by index). Keep the join key indexed on the inner (probed) table. - **Column projections** — selecting a subset of columns pushes down so only those attributes are read. @@ -127,7 +131,7 @@ These map directly to indexed storage operations: These can't be served from an index. They still return correct results, but the engine falls back to the legacy full-scan path, so cost and memory grow with table size: -- **Filters on un-indexed attributes** — any equality, range, or `IN` on an attribute with no index. +- **Filters on un-indexed attributes** — any equality, range, or `IN` on an attribute with no index, _when it's the only predicate_. Combined via `AND` with an indexed predicate, it's applied as a cheap residual filter after the index narrows the scan, and the query stays index-served overall — it's only a standalone un-indexed filter (or one joined by `OR`) that forces a full scan. - **`OR` where a branch isn't index-driving** — e.g. `WHERE a = 1 OR b = 2` when `b` is un-indexed, or `WHERE a = 1 OR name LIKE '%x%'`. A single un-indexed (or non-index-comparator) branch taints the whole disjunction and forces a scan. When **every** branch is an indexed, index-driving predicate, the `OR` stays on the fast path (see above). - **Suffix / substring `LIKE`** — `LIKE '%x'` and `LIKE '%x%'` (only prefix `LIKE 'x%'` is index-served). A suffix/contains `LIKE` combined (via `AND`) with an indexed predicate is applied as a cheap residual filter on the indexed result, so pair it with an indexed condition. - **`!=` / `<>` and `NOT` negations** — matching "everything except" a value can't use an index. @@ -155,7 +159,7 @@ These are deliberate corrections toward standard SQL, applied when the new engin ### Practical guidance -- Index every attribute you filter, sort, or join on. +- Index every attribute you filter, sort, or join on — but no more than that. Each index adds storage and write-time maintenance cost, so index for your actual query patterns rather than every attribute in the table. - Prefer prefix `LIKE 'x%'` over `LIKE '%x%'`; back a substring search with a separate indexed predicate, or model it as a dedicated indexed attribute. - Constrain the row set with an indexed `WHERE` before leaning on `ORDER BY`, `GROUP BY`, joins, or `SEARCH_JSON`. - Reach for the [REST interface](../rest/overview.md) on your hottest production read paths for a cacheable, index-driven contract; keep SQL for ad-hoc investigation, joins, and reporting.