diff --git a/.changeset/index-optimization-partial-and-or.md b/.changeset/index-optimization-partial-and-or.md new file mode 100644 index 0000000000..b5b6067360 --- /dev/null +++ b/.changeset/index-optimization-partial-and-or.md @@ -0,0 +1,17 @@ +--- +'@tanstack/db': patch +--- + +Fix incorrect results from index-optimized `where` clauses that combine indexed and non-indexed conditions. + +- `OR` expressions are now only served from indexes when every disjunct can use an index; otherwise the query falls back to a full scan. Previously, rows matched only by a non-indexed disjunct were missing from the result. +- `AND` expressions still use indexes for the conditions that have them, but the remaining conditions are now enforced by re-checking each candidate row against the full expression. Previously, non-indexed conditions were silently dropped, returning rows that did not match the query. +- Compound range conditions (e.g. `age > 5 AND age < 10`) combined with conditions on other fields no longer ignore those other conditions. +- Compound range conditions sharing the same boundary value (e.g. `age >= 5 AND age > 5`) now apply the strictest bound regardless of the order the conditions appear in, using the same value comparison semantics as the indexes (dates, locale strings, ...). +- Compound range conditions that only bound one side (e.g. `age > 5 AND age >= 8`) no longer return an empty result. +- Strict range comparisons (`gt`/`lt`) on BTree-indexed fields holding normalized values such as dates now correctly exclude the boundary value. +- Compound range conditions with a `null`/`undefined` bound (e.g. `gt(score, undefined)`) now re-filter against the full expression instead of returning index-ordered rows, matching the semantics of a full scan (a comparison against `null`/`undefined` is never true). +- Index-optimized `eq`, `IN`, and range queries on a field that has rows with `null`/`undefined` values no longer leak those rows into results. BTree indexes store and return such rows (they sort as the smallest key), but a comparison against `null`/`undefined` is never true, so these results are now re-filtered against the full expression to stay equivalent to a full scan. +- String range conditions (`gt`/`gte`/`lt`/`lte`) on a collection using locale string collation (the default) are no longer served by the index. The index orders strings with `localeCompare` while the `where` evaluator compares them with standard relational operators, so an index range lookup could omit matching rows; these conditions now fall back to a full scan. +- Range conditions whose operand is not ordered the same way by the index and the `where` evaluator (arrays, plain objects, Temporal values) now fall back to a full scan instead of using the index, which could otherwise omit matching rows. +- Range conditions on an index created with a custom comparator now fall back to a full scan, since the comparator's ordering may not match the `where` evaluator's relational operators. diff --git a/.changeset/nan-postgres-semantics.md b/.changeset/nan-postgres-semantics.md new file mode 100644 index 0000000000..028a372523 --- /dev/null +++ b/.changeset/nan-postgres-semantics.md @@ -0,0 +1,15 @@ +--- +'@tanstack/db': patch +--- + +Adopt PostgreSQL float semantics for `NaN` in `where` clauses and ordering. + +`NaN` (and invalid `Date` values, whose timestamp is `NaN`) previously had no consistent order — `NaN === NaN` is `false` in JavaScript, so `NaN` compared unequal to everything and could not be sorted or indexed deterministically. Following PostgreSQL, `NaN` is now treated as **equal to itself** and **greater than every other non-null value**: + +- `eq(row.value, NaN)` matches rows whose value is `NaN`; `inArray(row.value, [NaN, ...])` matches them too. +- Range comparisons treat `NaN` as the greatest value: `gt`/`gte` include it, `lt`/`lte` exclude it. +- Ordering by a field containing `NaN` is now deterministic, with `NaN` sorting last (and `null` still ordered by `NULLS FIRST`/`NULLS LAST`). + +`null`/`undefined` are unaffected: they continue to use three-valued logic (a comparison with `null` yields `UNKNOWN`). + +This makes results independent of whether a query is served from an index or a full scan. diff --git a/docs/guides/live-queries.md b/docs/guides/live-queries.md index 0780871a3b..4e8ec8dcbf 100644 --- a/docs/guides/live-queries.md +++ b/docs/guides/live-queries.md @@ -728,6 +728,13 @@ not(condition) For a complete reference of all available functions, see the [Expression Functions Reference](#expression-functions-reference) section. +### Comparison semantics + +Comparisons follow SQL/PostgreSQL conventions rather than raw JavaScript: + +- **`null` / `undefined` use three-valued logic.** Any comparison involving `null` or `undefined` evaluates to `UNKNOWN`, so the row is not matched. For example `eq(user.score, null)` matches nothing — use a dedicated null check (e.g. `isUndefined`) to match missing values. +- **`NaN` follows PostgreSQL float semantics.** `NaN` is treated as equal to itself and greater than every other (non-null) value. So `eq(row.value, NaN)` matches `NaN` rows, `gt(row.value, x)` includes `NaN`, and ordering by such a field places `NaN` last. (Invalid `Date` values, whose timestamp is `NaN`, behave the same way.) This differs from JavaScript, where `NaN === NaN` is `false`, and matches how PostgreSQL orders and indexes floating-point values. + ## Select Use `select` to specify which fields to include in your results and transform your data. Without `select`, you get the full schema. diff --git a/packages/db/src/collection/change-events.ts b/packages/db/src/collection/change-events.ts index 6afac412ce..3f4977b7bd 100644 --- a/packages/db/src/collection/change-events.ts +++ b/packages/db/src/collection/change-events.ts @@ -138,11 +138,16 @@ export function currentStateAsChanges< ) if (optimizationResult.canOptimize) { - // Use index optimization + // Use index optimization. When the index lookup is inexact, the keys + // are a superset of the true result (some conditions could not be + // served by an index), so re-check each row against the full expression. + const filterFn = optimizationResult.isExact + ? undefined + : createFilterFunctionFromExpression(expression) const result: Array, TKey>> = [] for (const key of optimizationResult.matchingKeys) { const value = collection.get(key) - if (value !== undefined) { + if (value !== undefined && (filterFn?.(value) ?? true)) { result.push({ type: `insert`, key, diff --git a/packages/db/src/indexes/base-index.ts b/packages/db/src/indexes/base-index.ts index 4346450f90..945221e6fa 100644 --- a/packages/db/src/indexes/base-index.ts +++ b/packages/db/src/indexes/base-index.ts @@ -68,6 +68,15 @@ export interface IndexInterface< supports: (operation: IndexOperation) => boolean + /** + * Whether range lookups (gt/gte/lt/lte) on this index can be trusted to + * return every matching key. Range traversal relies on the index ordering, so + * it is unsafe when the index uses a custom comparator, whose order may not + * match the WHERE evaluator's relational operators. Callers must fall back to + * a full scan when this is `false`. + */ + get supportsRangeOptimization(): boolean + matchesField: (fieldPath: Array) => boolean matchesCompareOptions: (compareOptions: CompareOptions) => boolean matchesDirection: (direction: OrderByDirection) => boolean @@ -90,6 +99,11 @@ export abstract class BaseIndex< protected totalLookupTime = 0 protected lastUpdated = new Date() protected compareOptions: CompareOptions + /** + * Set by subclasses when constructed with a user-supplied comparator, whose + * ordering may not match the WHERE evaluator's relational operators. + */ + protected hasCustomComparator = false constructor( id: number, @@ -144,6 +158,10 @@ export abstract class BaseIndex< return this.supportedOperations.has(operation) } + get supportsRangeOptimization(): boolean { + return !this.hasCustomComparator + } + matchesField(fieldPath: Array): boolean { return ( this.expression.type === `ref` && diff --git a/packages/db/src/indexes/basic-index.ts b/packages/db/src/indexes/basic-index.ts index b80f7fb439..b9b06d1925 100644 --- a/packages/db/src/indexes/basic-index.ts +++ b/packages/db/src/indexes/basic-index.ts @@ -65,6 +65,7 @@ export class BasicIndex< ) { super(id, expression, name, options) this.compareFn = options?.compareFn ?? defaultComparator + this.hasCustomComparator = options?.compareFn != null if (options?.compareOptions) { this.compareOptions = options!.compareOptions } diff --git a/packages/db/src/indexes/btree-index.ts b/packages/db/src/indexes/btree-index.ts index 17608950a7..8b92095f01 100644 --- a/packages/db/src/indexes/btree-index.ts +++ b/packages/db/src/indexes/btree-index.ts @@ -62,6 +62,7 @@ export class BTreeIndex< // Get the base compare function const baseCompareFn = options?.compareFn ?? defaultComparator + this.hasCustomComparator = options?.compareFn != null // Wrap it to denormalize sentinels before comparison // This ensures UNDEFINED_SENTINEL is converted back to undefined @@ -247,7 +248,16 @@ export class BTreeIndex< toKey, toInclusive, (indexedValue, _) => { - if (!fromInclusive && this.compareFn(indexedValue, from) === 0) { + // Only exclude the boundary when an exclusive lower bound was + // actually provided. Without a `from` bound, `fromKey` defaults to + // the minimum key and must not be dropped. Compare against the + // normalized key since indexed values are stored normalized + // (e.g. dates as timestamps), so the raw `from` would never match. + if ( + hasFrom && + !fromInclusive && + this.compareFn(indexedValue, fromKey) === 0 + ) { // the B+ tree `forRange` method does not support exclusive lower bounds // so we need to exclude it manually return diff --git a/packages/db/src/indexes/reverse-index.ts b/packages/db/src/indexes/reverse-index.ts index 8999b2801c..6ca61636e1 100644 --- a/packages/db/src/indexes/reverse-index.ts +++ b/packages/db/src/indexes/reverse-index.ts @@ -73,6 +73,10 @@ export class ReverseIndex< return this.originalIndex.supports(operation) } + get supportsRangeOptimization(): boolean { + return this.originalIndex.supportsRangeOptimization + } + matchesField(fieldPath: Array): boolean { return this.originalIndex.matchesField(fieldPath) } diff --git a/packages/db/src/query/compiler/evaluators.ts b/packages/db/src/query/compiler/evaluators.ts index 929ac56dfb..fa2e90725d 100644 --- a/packages/db/src/query/compiler/evaluators.ts +++ b/packages/db/src/query/compiler/evaluators.ts @@ -3,7 +3,11 @@ import { UnknownExpressionTypeError, UnknownFunctionError, } from '../../errors.js' -import { areValuesEqual, normalizeValue } from '../../utils/comparison.js' +import { + areValuesEqual, + isUnorderable, + normalizeValue, +} from '../../utils/comparison.js' import type { BasicExpression, Func, PropRef } from '../ir.js' import type { NamespacedRow } from '../../types.js' @@ -14,6 +18,19 @@ function isUnknown(value: any): boolean { return value === null || value === undefined } +/** + * Equality that follows PostgreSQL float semantics for `NaN`/invalid Dates: + * such values are equal to one another and unequal to anything else. For all + * other values it defers to {@link areValuesEqual}. Operands must not be + * null/undefined (callers handle UNKNOWN first). + */ +function valuesEqual(a: any, b: any): boolean { + if (isUnorderable(a) || isUnorderable(b)) { + return isUnorderable(a) && isUnorderable(b) + } + return areValuesEqual(a, b) +} + function toDateValue(value: any): Date | null { if (value instanceof Date) { return Number.isNaN(value.getTime()) ? null : value @@ -233,8 +250,9 @@ function compileFunction(func: Func, isSingleRow: boolean): (data: any) => any { if (isUnknown(a) || isUnknown(b)) { return null } - // Use areValuesEqual for proper Uint8Array/Buffer comparison - return areValuesEqual(a, b) + // NaN/invalid Dates are equal to one another (PostgreSQL semantics); + // otherwise use areValuesEqual for proper Uint8Array/Buffer comparison + return valuesEqual(a, b) } } case `gt`: { @@ -247,6 +265,11 @@ function compileFunction(func: Func, isSingleRow: boolean): (data: any) => any { if (isUnknown(a) || isUnknown(b)) { return null } + // NaN/invalid Dates sort greater than every other value, and are equal + // to one another (PostgreSQL semantics) + if (isUnorderable(a) || isUnorderable(b)) { + return isUnorderable(a) && !isUnorderable(b) + } return a > b } } @@ -260,6 +283,9 @@ function compileFunction(func: Func, isSingleRow: boolean): (data: any) => any { if (isUnknown(a) || isUnknown(b)) { return null } + if (isUnorderable(a) || isUnorderable(b)) { + return isUnorderable(a) + } return a >= b } } @@ -273,6 +299,9 @@ function compileFunction(func: Func, isSingleRow: boolean): (data: any) => any { if (isUnknown(a) || isUnknown(b)) { return null } + if (isUnorderable(a) || isUnorderable(b)) { + return isUnorderable(b) && !isUnorderable(a) + } return a < b } } @@ -286,6 +315,9 @@ function compileFunction(func: Func, isSingleRow: boolean): (data: any) => any { if (isUnknown(a) || isUnknown(b)) { return null } + if (isUnorderable(a) || isUnorderable(b)) { + return isUnorderable(b) + } return a <= b } } @@ -370,7 +402,7 @@ function compileFunction(func: Func, isSingleRow: boolean): (data: any) => any { if (!Array.isArray(array)) { return false } - return array.some((item) => normalizeValue(item) === value) + return array.some((item) => valuesEqual(normalizeValue(item), value)) } } diff --git a/packages/db/src/utils/comparison.ts b/packages/db/src/utils/comparison.ts index bf5ac1a913..992f0098c5 100644 --- a/packages/db/src/utils/comparison.ts +++ b/packages/db/src/utils/comparison.ts @@ -17,6 +17,21 @@ function getObjectId(obj: object): number { return id } +/** + * Whether a value has no IEEE-754 natural order: `NaN`, or an invalid Date + * (whose timestamp is `NaN`). The query engine follows PostgreSQL float + * semantics for these values — they are all equal to one another and greater + * than every other (non-null) value — so the comparator and the WHERE + * evaluator treat them explicitly instead of letting `NaN` compare unequal to + * everything (which has no consistent order and cannot be indexed or sorted). + */ +export function isUnorderable(value: any): boolean { + return ( + (typeof value === `number` && Number.isNaN(value)) || + (value instanceof Date && Number.isNaN(value.getTime())) + ) +} + /** * Universal comparison function for all data types * Handles null/undefined, strings, arrays, dates, objects, and primitives @@ -30,6 +45,16 @@ export const ascComparator = (a: any, b: any, opts: CompareOptions): number => { if (a == null) return nulls === `first` ? -1 : 1 if (b == null) return nulls === `first` ? 1 : -1 + // Handle NaN / invalid Dates. Following PostgreSQL float semantics, they are + // all equal and sort greater than every other non-null value. This keeps the + // order total (NaN would otherwise compare equal to everything), so such + // values can be sorted and stored in tree-based indexes. + const aUnordered = isUnorderable(a) + const bUnordered = isUnorderable(b) + if (aUnordered && bUnordered) return 0 + if (aUnordered) return 1 + if (bUnordered) return -1 + // if a and b are both strings, compare them based on locale if (typeof a === `string` && typeof b === `string`) { if (opts.stringSort === `locale`) { diff --git a/packages/db/src/utils/index-optimization.ts b/packages/db/src/utils/index-optimization.ts index 81b111af56..5a52a5ec54 100644 --- a/packages/db/src/utils/index-optimization.ts +++ b/packages/db/src/utils/index-optimization.ts @@ -18,6 +18,7 @@ import { DEFAULT_COMPARE_OPTIONS } from '../utils.js' import { ReverseIndex } from '../indexes/reverse-index.js' import { hasVirtualPropPath } from '../virtual-props.js' +import { makeComparator } from './comparison.js' import type { CompareOptions } from '../query/builder/types.js' import type { IndexInterface, IndexOperation } from '../indexes/base-index.js' import type { BasicExpression } from '../query/ir.js' @@ -29,6 +30,13 @@ import type { CollectionLike } from '../types.js' export interface OptimizationResult { canOptimize: boolean matchingKeys: Set + /** + * Whether `matchingKeys` is exactly the set of keys matching the expression. + * When `false`, the keys are a superset of the true result (some conditions + * could not be served by an index) and each row must be re-checked against + * the full expression before being included in the result. + */ + isExact: boolean } /** @@ -94,6 +102,94 @@ export function unionSets(sets: Array>): Set { return result } +/** + * Whether a value can be matched exactly by an index lookup, i.e. the index + * result for it is not a superset that the caller must re-filter. + * + * Only `null`/`undefined` are inexact: the WHERE evaluator's three-valued logic + * makes any comparison against them UNKNOWN, yet a BTree index stores and + * returns rows with nullish keys (they sort to the nulls end), so a result that + * could include such rows must be re-filtered. + * + * `NaN` and invalid Dates are exact: under the engine's PostgreSQL float + * semantics they are equal to themselves and ordered (greatest non-null value), + * so the evaluator and the index agree on them and no re-filtering is needed. + */ +function isExactComparisonValue(value: unknown): boolean { + return value != null +} + +/** + * Whether the collection orders strings using locale collation. + * + * Under `stringSort: 'locale'` a BTree string index orders values with + * `localeCompare`, but the WHERE evaluator compares strings with JS relational + * operators (code-point order). For range predicates these orders disagree + * (e.g. `'ö' > 'z'` is true in JS but `'ö'` sorts before `'z'` under locale + * `en`), so an index range lookup can omit matching rows. Such omissions cannot + * be recovered by re-filtering, so locale-backed string range predicates must + * not be index-optimized. + */ +function usesLocaleStringSort(collection: CollectionLike): boolean { + const opts = { ...DEFAULT_COMPARE_OPTIONS, ...collection.compareOptions } + return opts.stringSort === `locale` +} + +/** + * Whether a range predicate on this operand would use an index ordering that + * differs from the WHERE evaluator's relational operators, so an index range + * lookup could omit genuine matches that re-filtering cannot recover. + * + * The evaluator compares with JS relational operators (extended with + * PostgreSQL float semantics for `NaN`/invalid Dates). That order matches the + * index comparator for numbers, booleans, bigints, lexically-sorted strings, + * Dates (valid, ordered by time; invalid, ordered as the greatest value) and + * `NaN`. It diverges for locale-sorted strings (localeCompare vs code-point + * order) and for arrays, plain objects, Temporal values and typed arrays + * (recursive/identity ordering vs string coercion). + * + * Note: `null`/`undefined` operands are not handled here — those are superset + * cases handled by re-filtering ({@link isExactComparisonValue}). + */ +function isRangeOrderingDivergent( + value: unknown, + collection: CollectionLike, +): boolean { + switch (typeof value) { + case `number`: + case `bigint`: + case `boolean`: + return false + case `string`: + return usesLocaleStringSort(collection) + case `object`: { + if (value === null) return false + // Dates order consistently with the evaluator: valid Dates by time, and + // invalid Dates as the greatest value under PostgreSQL float semantics. + return !(value instanceof Date) + } + default: + return false + } +} + +/** + * Whether a range predicate (gt/gte/lt/lte) on this operand can be safely + * served by the given index: the operand's domain must order the same way the + * index does, and the index itself must support trustworthy range traversal + * (no custom comparator). + */ +function canRangeOptimize( + value: unknown, + index: IndexInterface, + collection: CollectionLike, +): boolean { + return ( + !isRangeOrderingDivergent(value, collection) && + index.supportsRangeOptimization + ) +} + /** * Optimizes a query expression using available indexes to find matching keys */ @@ -134,7 +230,7 @@ function optimizeQueryRecursive( } } - return { canOptimize: false, matchingKeys: new Set() } + return { canOptimize: false, matchingKeys: new Set(), isExact: false } } /** @@ -167,6 +263,14 @@ export function canOptimizeExpression< return false } +/** + * Result of compound range optimization, including which AND arguments + * were covered by the range query so the caller can process the rest. + */ +interface CompoundRangeResult extends OptimizationResult { + coveredArgIndices: Set +} + /** * Optimizes compound range queries on the same field * Example: WHERE age > 5 AND age < 10 @@ -177,9 +281,14 @@ function optimizeCompoundRangeQuery< >( expression: BasicExpression, collection: CollectionLike, -): OptimizationResult { +): CompoundRangeResult { if (expression.type !== `func` || expression.args.length < 2) { - return { canOptimize: false, matchingKeys: new Set() } + return { + canOptimize: false, + matchingKeys: new Set(), + isExact: false, + coveredArgIndices: new Set(), + } } // Group range operations by field @@ -188,11 +297,12 @@ function optimizeCompoundRangeQuery< Array<{ operation: `gt` | `gte` | `lt` | `lte` value: any + argIndex: number }> >() // Collect all range operations from AND arguments - for (const arg of expression.args) { + for (const [argIndex, arg] of expression.args.entries()) { if (arg.type === `func` && [`gt`, `gte`, `lt`, `lte`].includes(arg.name)) { const rangeOp = arg as any if (rangeOp.args.length === 2) { @@ -238,7 +348,7 @@ function optimizeCompoundRangeQuery< if (!fieldOperations.has(fieldKey)) { fieldOperations.set(fieldKey, []) } - fieldOperations.get(fieldKey)!.push({ operation, value }) + fieldOperations.get(fieldKey)!.push({ operation, value, argIndex }) } } } @@ -250,55 +360,114 @@ function optimizeCompoundRangeQuery< const fieldPath = fieldKey.split(`.`) const index = findIndexForField(collection, fieldPath) + // Only collapse this field into a range query when every bound's domain + // orders the same way the index does and the index supports trustworthy + // range traversal. Otherwise the index may omit matching rows that + // re-filtering cannot recover, so leave the field for a full scan. + if ( + index && + operations.some((op) => !canRangeOptimize(op.value, index, collection)) + ) { + continue + } + if (index && index.supports(`gt`) && index.supports(`lt`)) { - // Build range query options + // Compare values with the same semantics the index uses (dates, + // locale strings, ...), in ascending order since bounds are about + // value order regardless of the index direction + const compare = makeComparator({ + ...DEFAULT_COMPARE_OPTIONS, + ...collection.compareOptions, + direction: `asc`, + }) + + // Build range query options, keeping the strictest bound on each + // side: a larger lower bound (or smaller upper bound) wins, and at + // equal values the exclusive operation wins over the inclusive one. + // `hasFromBound`/`hasToBound` track whether a bound was selected, + // separately from the bound value (which may legitimately be falsy). let from: any = undefined let to: any = undefined + let hasFromBound = false + let hasToBound = false let fromInclusive = true let toInclusive = true + // A comparison against null/undefined is never true, but in an index + // nullish values sort to the nulls end, so a range query cannot + // represent such a bound. Track it and force a re-filter instead of + // claiming the result is exact. (NaN/invalid Dates are ordered and + // comparable under PostgreSQL semantics, so they are real bounds.) + let hasNonComparableBound = false for (const { operation, value } of operations) { + if (!isExactComparisonValue(value)) { + hasNonComparableBound = true + continue + } switch (operation) { case `gt`: - if (from === undefined || value > from) { + case `gte`: { + const cmp = hasFromBound ? compare(value, from) : 1 + if (cmp > 0) { from = value + hasFromBound = true + fromInclusive = operation === `gte` + } else if (cmp === 0 && operation === `gt`) { fromInclusive = false } break - case `gte`: - if (from === undefined || value > from) { - from = value - fromInclusive = true - } - break + } case `lt`: - if (to === undefined || value < to) { + case `lte`: { + const cmp = hasToBound ? compare(value, to) : -1 + if (cmp < 0) { to = value + hasToBound = true + toInclusive = operation === `lte` + } else if (cmp === 0 && operation === `lt`) { toInclusive = false } break - case `lte`: - if (to === undefined || value < to) { - to = value - toInclusive = true - } - break + } } } - const matchingKeys = (index as any).rangeQuery({ - from, - to, - fromInclusive, - toInclusive, - }) - - return { canOptimize: true, matchingKeys } + // Only pass the bounds that were selected: rangeQuery distinguishes + // an absent bound (open-ended) from an explicitly provided one + const rangeOptions: Record = {} + if (hasFromBound) { + rangeOptions.from = from + rangeOptions.fromInclusive = fromInclusive + } + if (hasToBound) { + rangeOptions.to = to + rangeOptions.toInclusive = toInclusive + } + const matchingKeys = (index as any).rangeQuery(rangeOptions) + + return { + canOptimize: true, + matchingKeys, + // The range result is exact only when it cannot include rows with a + // nullish indexed value (which a comparison would reject but the + // index returns, as they sort as the smallest key). That requires a + // non-nullish lower bound to exclude them: without `hasFromBound` + // the range is open at the bottom and captures those rows, and a + // non-comparable bound value (`hasNonComparableBound`) can never + // bound them out. + isExact: hasFromBound && !hasNonComparableBound, + coveredArgIndices: new Set(operations.map((op) => op.argIndex)), + } } } } - return { canOptimize: false, matchingKeys: new Set() } + return { + canOptimize: false, + matchingKeys: new Set(), + isExact: false, + coveredArgIndices: new Set(), + } } /** @@ -312,7 +481,7 @@ function optimizeSimpleComparison< collection: CollectionLike, ): OptimizationResult { if (expression.type !== `func` || expression.args.length !== 2) { - return { canOptimize: false, matchingKeys: new Set() } + return { canOptimize: false, matchingKeys: new Set(), isExact: false } } const leftArg = expression.args[0]! @@ -362,15 +531,46 @@ function optimizeSimpleComparison< // Check if the index supports this operation if (!index.supports(indexOperation)) { - return { canOptimize: false, matchingKeys: new Set() } + return { canOptimize: false, matchingKeys: new Set(), isExact: false } + } + + // A range op can only use the index when the operand's domain orders the + // same way the index does and the index supports trustworthy traversal. + // Otherwise the index may omit matching rows, which re-filtering cannot + // recover, so fall back to a full scan. + if ( + (operation === `gt` || + operation === `gte` || + operation === `lt` || + operation === `lte`) && + !canRangeOptimize(queryValue, index, collection) + ) { + return { canOptimize: false, matchingKeys: new Set(), isExact: false } } const matchingKeys = index.lookup(indexOperation, queryValue) - return { canOptimize: true, matchingKeys } + + // A comparison against a nullish value is never true, but BTree indexes + // store and return rows with nullish keys (they sort to the nulls end). + // Determine whether the index result is exact or a superset that the + // caller must re-filter: + // - eq/gt/gte: a nullish query value matches nothing while the index + // still returns nullish-keyed rows -> inexact. A non-nullish lower + // bound (gt/gte) excludes those bottom-sorted rows, so they stay exact. + // - lt/lte: the open lower bound always includes nullish-keyed rows, + // so the result is conservatively inexact. + // NaN/invalid Dates are exact here: under PostgreSQL float semantics the + // evaluator and the index agree on them (equal to self, greatest). + const isExact = + operation === `lt` || operation === `lte` + ? false + : isExactComparisonValue(queryValue) + + return { canOptimize: true, matchingKeys, isExact } } } - return { canOptimize: false, matchingKeys: new Set() } + return { canOptimize: false, matchingKeys: new Set(), isExact: false } } /** @@ -412,22 +612,41 @@ function optimizeAndExpression( collection: CollectionLike, ): OptimizationResult { if (expression.type !== `func` || expression.args.length < 2) { - return { canOptimize: false, matchingKeys: new Set() } + return { canOptimize: false, matchingKeys: new Set(), isExact: false } } // First, try to optimize compound range queries on the same field + // (e.g. age > 5 AND age < 10 becomes a single range query) const compoundRangeResult = optimizeCompoundRangeQuery(expression, collection) - if (compoundRangeResult.canOptimize) { - return compoundRangeResult - } + const coveredArgIndices = compoundRangeResult.canOptimize + ? compoundRangeResult.coveredArgIndices + : new Set() const results: Array> = [] + if (compoundRangeResult.canOptimize) { + results.push(compoundRangeResult) + } - // Try to optimize each part, keep the optimizable ones - for (const arg of expression.args) { + // Try to optimize the remaining conjuncts, keep the optimizable ones. + // Conjuncts that cannot use an index make the result inexact: the + // intersection is then a superset of the true result and must be + // re-filtered against the full expression by the caller. The compound + // range result may itself be inexact (e.g. a null/undefined bound). + let allConjunctsExact = !compoundRangeResult.canOptimize + ? true + : compoundRangeResult.isExact + for (const [argIndex, arg] of expression.args.entries()) { + if (coveredArgIndices.has(argIndex)) { + continue + } const result = optimizeQueryRecursive(arg, collection) if (result.canOptimize) { results.push(result) + if (!result.isExact) { + allConjunctsExact = false + } + } else { + allConjunctsExact = false } } @@ -435,10 +654,14 @@ function optimizeAndExpression( // Use intersectSets utility for AND logic const allMatchingSets = results.map((r) => r.matchingKeys) const intersectedKeys = intersectSets(allMatchingSets) - return { canOptimize: true, matchingKeys: intersectedKeys } + return { + canOptimize: true, + matchingKeys: intersectedKeys, + isExact: allConjunctsExact, + } } - return { canOptimize: false, matchingKeys: new Set() } + return { canOptimize: false, matchingKeys: new Set(), isExact: false } } /** @@ -464,27 +687,31 @@ function optimizeOrExpression( collection: CollectionLike, ): OptimizationResult { if (expression.type !== `func` || expression.args.length < 2) { - return { canOptimize: false, matchingKeys: new Set() } + return { canOptimize: false, matchingKeys: new Set(), isExact: false } } const results: Array> = [] - // Try to optimize each part, keep the optimizable ones + // Every disjunct must be optimizable: rows matched only by a disjunct + // that cannot use an index would be missing from the union, and no + // post-filtering can recover them. In that case fall back to a full scan. for (const arg of expression.args) { const result = optimizeQueryRecursive(arg, collection) - if (result.canOptimize) { - results.push(result) + if (!result.canOptimize) { + return { canOptimize: false, matchingKeys: new Set(), isExact: false } } + results.push(result) } - if (results.length > 0) { - // Use unionSets utility for OR logic - const allMatchingSets = results.map((r) => r.matchingKeys) - const unionedKeys = unionSets(allMatchingSets) - return { canOptimize: true, matchingKeys: unionedKeys } + // Use unionSets utility for OR logic + const allMatchingSets = results.map((r) => r.matchingKeys) + const unionedKeys = unionSets(allMatchingSets) + return { + canOptimize: true, + matchingKeys: unionedKeys, + // An inexact (superset) disjunct makes the union a superset as well + isExact: results.every((r) => r.isExact), } - - return { canOptimize: false, matchingKeys: new Set() } } /** @@ -498,8 +725,9 @@ function canOptimizeOrExpression< return false } - // If any argument can be optimized, we can gain some speedup - return expression.args.some((arg) => canOptimizeExpression(arg, collection)) + // Every disjunct must be optimizable, otherwise the union would miss + // rows matched only by the non-optimizable disjuncts + return expression.args.every((arg) => canOptimizeExpression(arg, collection)) } /** @@ -513,7 +741,7 @@ function optimizeInArrayExpression< collection: CollectionLike, ): OptimizationResult { if (expression.type !== `func` || expression.args.length !== 2) { - return { canOptimize: false, matchingKeys: new Set() } + return { canOptimize: false, matchingKeys: new Set(), isExact: false } } const fieldArg = expression.args[0]! @@ -528,11 +756,17 @@ function optimizeInArrayExpression< const values = (arrayArg as any).value const index = findIndexForField(collection, fieldPath) + // A nullish or NaN member can never be matched by `IN` (a comparison + // against null/undefined/NaN is never true), but the index would still + // return rows with such an indexed value. When the list contains one of + // those the result is a superset that the caller must re-filter. + const isExact = values.every((value: any) => isExactComparisonValue(value)) + if (index) { // Check if the index supports IN operation if (index.supports(`in`)) { const matchingKeys = index.lookup(`in`, values) - return { canOptimize: true, matchingKeys } + return { canOptimize: true, matchingKeys, isExact } } else if (index.supports(`eq`)) { // Fallback to multiple equality lookups const matchingKeys = new Set() @@ -542,12 +776,12 @@ function optimizeInArrayExpression< matchingKeys.add(key) } } - return { canOptimize: true, matchingKeys } + return { canOptimize: true, matchingKeys, isExact } } } } - return { canOptimize: false, matchingKeys: new Set() } + return { canOptimize: false, matchingKeys: new Set(), isExact: false } } /** diff --git a/packages/db/tests/btree-index-undefined-values.test.ts b/packages/db/tests/btree-index-undefined-values.test.ts index 11e29690c6..1510c02e3c 100644 --- a/packages/db/tests/btree-index-undefined-values.test.ts +++ b/packages/db/tests/btree-index-undefined-values.test.ts @@ -248,6 +248,24 @@ describe(`BTreeIndex - undefined value handling`, () => { expect(withoutFrom.size).toBe(3) }) + it(`should not drop the minimum key when an upper-only range is exclusive on the (absent) lower bound`, () => { + // When no `from` bound is provided, `fromInclusive` must not cause the + // smallest key to be excluded: there is no lower bound to exclude + // against. Only an explicitly provided exclusive lower bound should + // drop its boundary value. + const index = createIndex(`value`) + index.add(`a`, { value: 1 }) + index.add(`b`, { value: 5 }) + index.add(`c`, { value: 10 }) + + const result = index.rangeQuery({ to: 10, fromInclusive: false }) + + expect(result.size).toBe(3) + expect(result).toContain(`a`) + expect(result).toContain(`b`) + expect(result).toContain(`c`) + }) + it(`should handle range query from undefined to undefined`, () => { const index = createIndex(`value`) index.add(`a`, { value: undefined }) diff --git a/packages/db/tests/collection-indexes.test.ts b/packages/db/tests/collection-indexes.test.ts index a441a5520d..bd5f4868c7 100644 --- a/packages/db/tests/collection-indexes.test.ts +++ b/packages/db/tests/collection-indexes.test.ts @@ -625,6 +625,19 @@ describe(`Collection Indexes`, () => { }) }) + it(`should exclude the boundary value from greater than queries on dates`, () => { + // gt must be strict for date fields: Bob was created exactly on + // 2023-01-02, so only rows created strictly later may be returned. + collection.createIndex((row) => row.createdAt) + + const result = collection.currentStateAsChanges({ + where: gt(new PropRef([`createdAt`]), new Date(`2023-01-02`)), + })! + + const names = result.map((r) => r.value.name).sort() + expect(names).toEqual([`Charlie`, `Diana`, `Eve`]) + }) + it(`should perform greater than or equal queries`, () => { withIndexTracking(collection, (tracker) => { const result = collection.currentStateAsChanges({ @@ -1179,6 +1192,538 @@ describe(`Collection Indexes`, () => { }) }) }) + + it(`should include rows matched by any OR condition when conditions mix indexed and non-indexed expressions`, () => { + // An OR query must return the union of rows matching each condition: + // eq(age, 25) matches Alice (age 25) + // gt(length(name), 6) matches Charlie (name length 7) + // `age` has an index while `length(name)` is a computed expression + // without one, but the chosen execution strategy must not change the + // result: both Alice and Charlie satisfy the OR and must be returned. + const result = collection.currentStateAsChanges({ + where: or( + eq(new PropRef([`age`]), 25), + gt(length(new PropRef([`name`])), 6), + ), + })! + + const names = result.map((r) => r.value.name).sort() + expect(names).toEqual([`Alice`, `Charlie`]) + }) + + it(`should only return rows matching every AND condition when conditions mix indexed and non-indexed expressions`, () => { + // An AND query must return only the rows matching all conditions: + // eq(status, 'active') matches Alice, Charlie and Eve + // gt(length(name), 6) matches only Charlie (name length 7) + // `status` has an index while `length(name)` is a computed expression + // without one, but every condition must still be enforced: only + // Charlie satisfies both. + const result = collection.currentStateAsChanges({ + where: and( + eq(new PropRef([`status`]), `active`), + gt(length(new PropRef([`name`])), 6), + ), + })! + + const names = result.map((r) => r.value.name).sort() + expect(names).toEqual([`Charlie`]) + }) + + it(`should apply the strictest lower bound when range conditions share the same value`, () => { + // gte(age, 25) AND gt(age, 25) reduces to age > 25: the strict + // comparison wins at the shared boundary, so Alice (age 25) must be + // excluded regardless of the order the conditions appear in. + const result = collection.currentStateAsChanges({ + where: and(gte(new PropRef([`age`]), 25), gt(new PropRef([`age`]), 25)), + })! + + const names = result.map((r) => r.value.name).sort() + expect(names).toEqual([`Bob`, `Charlie`, `Diana`]) + }) + + it(`should apply the strictest upper bound when range conditions share the same value`, () => { + // lte(age, 30) AND lt(age, 30) reduces to age < 30: the strict + // comparison wins at the shared boundary, so Bob (age 30) must be + // excluded. + const result = collection.currentStateAsChanges({ + where: and(lte(new PropRef([`age`]), 30), lt(new PropRef([`age`]), 30)), + })! + + const names = result.map((r) => r.value.name).sort() + expect(names).toEqual([`Alice`, `Diana`, `Eve`]) + }) + + it(`should apply the strictest bound for date ranges sharing the same value`, () => { + // Distinct Date instances representing the same point in time must be + // treated as equal values: gte(createdAt, jan2) AND gt(createdAt, jan2) + // reduces to createdAt > jan2, so Bob (created 2023-01-02) must be + // excluded. + collection.createIndex((row) => row.createdAt) + + const result = collection.currentStateAsChanges({ + where: and( + gte(new PropRef([`createdAt`]), new Date(`2023-01-02`)), + gt(new PropRef([`createdAt`]), new Date(`2023-01-02`)), + ), + })! + + const names = result.map((r) => r.value.name).sort() + expect(names).toEqual([`Charlie`, `Diana`, `Eve`]) + }) + + it(`should enforce every AND condition when a range on one field is combined with conditions on other fields`, () => { + // An AND query that contains a compound range on one field plus a + // condition on another field must enforce all of them: + // gt(age, 24) AND lt(age, 36) matches Alice (25), Bob (30), + // Charlie (35) and Diana (28) + // eq(status, 'active') matches Alice, Charlie and Eve + // Only Alice and Charlie satisfy the full conjunction. + const result = collection.currentStateAsChanges({ + where: and( + gt(new PropRef([`age`]), 24), + lt(new PropRef([`age`]), 36), + eq(new PropRef([`status`]), `active`), + ), + })! + + const names = result.map((r) => r.value.name).sort() + expect(names).toEqual([`Alice`, `Charlie`]) + }) + + it(`should match a full scan when a range condition uses an undefined bound`, () => { + // A comparison against `undefined` matches no rows (a comparison with + // null/undefined is never true), so `gt(score, undefined)` excludes + // every row and the whole AND must return nothing. The index-optimized + // path must agree with a plain full scan and not leak rows. + collection.createIndex((row) => row.score) + + const result = collection.currentStateAsChanges({ + where: and( + gt(new PropRef([`score`]), undefined), + lt(new PropRef([`score`]), 90), + ), + })! + + expect(result).toEqual([]) + }) + + it(`should not match rows with a missing value for an equality on undefined`, () => { + // An equality comparison against `undefined` is never true, so + // `eq(score, undefined)` must return no rows even though Eve has an + // undefined score. The index-optimized path must agree with a full + // predicate scan. + collection.createIndex((row) => row.score) + + const result = collection.currentStateAsChanges({ + where: eq(new PropRef([`score`]), undefined), + })! + + expect(result).toEqual([]) + }) + + it(`should ignore an undefined member when matching an IN list`, () => { + // A row only matches `IN` when its value equals one of the listed + // values; a comparison with `undefined` is never true. So + // `inArray(score, [undefined, 80])` must match only Bob (score 80) + // and must not match Eve (undefined score). + collection.createIndex((row) => row.score) + + const result = collection.currentStateAsChanges({ + where: inArray(new PropRef([`score`]), [undefined, 80]), + })! + + const names = result.map((r) => r.value.name).sort() + expect(names).toEqual([`Bob`]) + }) + + it(`should not match rows with a missing value for a range comparison`, () => { + // A range comparison against a row with an undefined value is never + // true, so `lt(score, 85)` must match only Bob (score 80) and must + // not match Eve (undefined score). + collection.createIndex((row) => row.score) + + const result = collection.currentStateAsChanges({ + where: lt(new PropRef([`score`]), 85), + })! + + const names = result.map((r) => r.value.name).sort() + expect(names).toEqual([`Bob`]) + }) + + it(`should not match rows with a missing value for an upper-bounded compound range`, () => { + // A compound range with only upper bounds (e.g. score <= 90) must not + // match a row with an undefined value, since a comparison against + // undefined is never true. Only Bob (80), Charlie (90) and Diana (85) + // satisfy `score <= 90`; Eve (undefined) must be excluded. + collection.createIndex((row) => row.score) + + const result = collection.currentStateAsChanges({ + where: and( + lte(new PropRef([`score`]), 90), + lte(new PropRef([`score`]), 95), + ), + })! + + const names = result.map((r) => r.value.name).sort() + expect(names).toEqual([`Bob`, `Charlie`, `Diana`]) + }) + + it(`should match a string range predicate using the same ordering as a full scan`, async () => { + // String comparisons in the WHERE evaluator use JS relational operators + // (code-point order), where `'ö' > 'z'` is true. A row named `ö` must + // therefore be returned by `name > 'z'`, even though a locale-collated + // index orders `ö` before `z`. The index-optimized result must agree + // with a full predicate scan. + const stringCollection = createCollection< + { id: string; name: string }, + string + >({ + getKey: (row) => row.id, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `1`, name: `apple` } }) + write({ type: `insert`, value: { id: `2`, name: `ö` } }) + commit() + markReady() + }, + }, + }) + await stringCollection.stateWhenReady() + stringCollection.createIndex((row) => row.name) + + const result = stringCollection.currentStateAsChanges({ + where: gt(new PropRef([`name`]), `z`), + })! + + const names = result.map((r) => r.value.name).sort() + expect(names).toEqual([`ö`]) + }) + + it(`should match a row with a NaN value for an equality on NaN`, async () => { + // Under PostgreSQL float semantics NaN is equal to itself, so + // `eq(score, NaN)` matches the NaN-valued row (and the index, which + // stores and returns it, agrees with a full scan). + const nanCollection = createCollection< + { id: string; score: number }, + string + >({ + getKey: (row) => row.id, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `1`, score: 5 } }) + write({ type: `insert`, value: { id: `2`, score: NaN } }) + commit() + markReady() + }, + }, + }) + await nanCollection.stateWhenReady() + nanCollection.createIndex((row) => row.score) + + const result = nanCollection.currentStateAsChanges({ + where: eq(new PropRef([`score`]), NaN), + })! + + const ids = result.map((r) => r.value.id).sort() + expect(ids).toEqual([`2`]) + }) + + it(`should match a row with a NaN value for an IN list containing NaN`, async () => { + // A row matches `IN` when its value equals a listed value. Under + // PostgreSQL float semantics NaN is equal to itself, so + // `inArray(score, [NaN, 5])` matches both the score-5 row and the + // NaN-valued row. + const nanCollection = createCollection< + { id: string; score: number }, + string + >({ + getKey: (row) => row.id, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `1`, score: 5 } }) + write({ type: `insert`, value: { id: `2`, score: NaN } }) + commit() + markReady() + }, + }, + }) + await nanCollection.stateWhenReady() + nanCollection.createIndex((row) => row.score) + + const result = nanCollection.currentStateAsChanges({ + where: inArray(new PropRef([`score`]), [NaN, 5]), + })! + + const ids = result.map((r) => r.value.id).sort() + expect(ids).toEqual([`1`, `2`]) + }) + + it(`should return array-valued rows for a range predicate consistently with a full scan`, async () => { + // Range predicates are evaluated with standard relational comparison, + // under which `[2] > [10]` is true (arrays compare as their string + // form). An index on an array-valued field must return the same rows as + // a full scan and must not drop this match. + const arrayCollection = createCollection< + { id: string; value: Array }, + string + >({ + getKey: (row) => row.id, + startSync: true, + autoIndex: `off`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `1`, value: [2] } }) + commit() + markReady() + }, + }, + }) + await arrayCollection.stateWhenReady() + arrayCollection.createIndex((row) => row.value) + + const result = arrayCollection.currentStateAsChanges({ + where: gt(new PropRef([`value`]), [10]), + })! + + const ids = result.map((r) => r.value.id).sort() + expect(ids).toEqual([`1`]) + }) + + it(`should return all matching rows for a range predicate on a custom-comparator index`, async () => { + // A range predicate must return every row that satisfies it regardless + // of the comparator the index was created with. With scores 5 and 20, + // `score > 10` matches only the row with score 20. + const customCollection = createCollection< + { id: string; score: number }, + string + >({ + getKey: (row) => row.id, + startSync: true, + autoIndex: `off`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `low`, score: 5 } }) + write({ type: `insert`, value: { id: `high`, score: 20 } }) + commit() + markReady() + }, + }, + }) + await customCollection.stateWhenReady() + customCollection.createIndex((row) => row.score, { + options: { compareFn: (a: number, b: number) => b - a }, + }) + + const result = customCollection.currentStateAsChanges({ + where: gt(new PropRef([`score`]), 10), + })! + + const ids = result.map((r) => r.value.id).sort() + expect(ids).toEqual([`high`]) + }) + + it(`should return all matching rows for a range predicate when the field also contains NaN`, async () => { + // A range predicate must return every matching row even when other rows + // hold a NaN value for the field. Under PostgreSQL float semantics NaN is + // the greatest value, so with scores NaN, 1, 3, 5 and 7, `score > 2` + // matches the rows with scores 3, 5, 7 and NaN. + const nanCollection = createCollection< + { id: string; score: number }, + string + >({ + getKey: (row) => row.id, + startSync: true, + autoIndex: `off`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `nan`, score: NaN } }) + write({ type: `insert`, value: { id: `one`, score: 1 } }) + write({ type: `insert`, value: { id: `three`, score: 3 } }) + write({ type: `insert`, value: { id: `five`, score: 5 } }) + write({ type: `insert`, value: { id: `seven`, score: 7 } }) + commit() + markReady() + }, + }, + }) + await nanCollection.stateWhenReady() + nanCollection.createIndex((row) => row.score) + + const result = nanCollection.currentStateAsChanges({ + where: gt(new PropRef([`score`]), 2), + })! + + const ids = result.map((r) => r.value.id).sort() + expect(ids).toEqual([`five`, `nan`, `seven`, `three`]) + }) + + it(`should use the index for a range query on a field that also contains NaN`, async () => { + // A NaN value has a well-defined sort position (greatest, under + // PostgreSQL float semantics), so a range query on the field can still be + // served by the index and does not need to fall back to a full scan. + const nanCollection = createCollection< + { id: string; score: number }, + string + >({ + getKey: (row) => row.id, + startSync: true, + autoIndex: `off`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `nan`, score: NaN } }) + write({ type: `insert`, value: { id: `one`, score: 1 } }) + write({ type: `insert`, value: { id: `three`, score: 3 } }) + write({ type: `insert`, value: { id: `five`, score: 5 } }) + write({ type: `insert`, value: { id: `seven`, score: 7 } }) + commit() + markReady() + }, + }, + }) + await nanCollection.stateWhenReady() + nanCollection.createIndex((row) => row.score) + + withIndexTracking(nanCollection, (tracker) => { + const result = nanCollection.currentStateAsChanges({ + where: gt(new PropRef([`score`]), 2), + })! + + const ids = result.map((r) => r.value.id).sort() + expect(ids).toEqual([`five`, `nan`, `seven`, `three`]) + + expectIndexUsage(tracker.stats, { + shouldUseIndex: true, + shouldUseFullScan: false, + }) + }) + }) + + it(`should exclude NaN from a less-than range query`, async () => { + // Under PostgreSQL float semantics NaN is the greatest value, so + // `score < 4` matches the rows with scores 1 and 3 but never the + // NaN-valued row. + const nanCollection = createCollection< + { id: string; score: number }, + string + >({ + getKey: (row) => row.id, + startSync: true, + autoIndex: `off`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `nan`, score: NaN } }) + write({ type: `insert`, value: { id: `one`, score: 1 } }) + write({ type: `insert`, value: { id: `three`, score: 3 } }) + write({ type: `insert`, value: { id: `five`, score: 5 } }) + write({ type: `insert`, value: { id: `seven`, score: 7 } }) + commit() + markReady() + }, + }, + }) + await nanCollection.stateWhenReady() + nanCollection.createIndex((row) => row.score) + + const result = nanCollection.currentStateAsChanges({ + where: lt(new PropRef([`score`]), 4), + })! + + const ids = result.map((r) => r.value.id).sort() + expect(ids).toEqual([`one`, `three`]) + }) + + // Invalid Dates have a NaN timestamp, so they follow the same PostgreSQL + // float semantics as NaN: equal to one another and greater than every valid + // Date. The index-served and full-scan results must agree. + const makeInvalidDateCollection = async () => { + const dateCollection = createCollection< + { id: string; createdAt: Date }, + string + >({ + getKey: (row) => row.id, + startSync: true, + autoIndex: `off`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ + type: `insert`, + value: { id: `invalid`, createdAt: new Date(`not a date`) }, + }) + write({ + type: `insert`, + value: { id: `valid`, createdAt: new Date(`2023-01-01`) }, + }) + commit() + markReady() + }, + }, + }) + await dateCollection.stateWhenReady() + dateCollection.createIndex((row) => row.createdAt) + return dateCollection + } + + it(`should match an invalid-Date row for an equality on an invalid Date`, async () => { + const dateCollection = await makeInvalidDateCollection() + + const result = dateCollection.currentStateAsChanges({ + where: eq(new PropRef([`createdAt`]), new Date(`not a date`)), + })! + + const ids = result.map((r) => r.value.id).sort() + expect(ids).toEqual([`invalid`]) + }) + + it(`should match an invalid-Date member of an IN list`, async () => { + const dateCollection = await makeInvalidDateCollection() + + const result = dateCollection.currentStateAsChanges({ + where: inArray(new PropRef([`createdAt`]), [ + new Date(`not a date`), + new Date(`2023-01-01`), + ]), + })! + + const ids = result.map((r) => r.value.id).sort() + expect(ids).toEqual([`invalid`, `valid`]) + }) + + it(`should treat an invalid Date as greater than valid Dates in a range query`, async () => { + // `createdAt > 2022` matches the valid Date and the invalid Date (which + // is the greatest value under PostgreSQL float semantics). + const dateCollection = await makeInvalidDateCollection() + + const result = dateCollection.currentStateAsChanges({ + where: gt(new PropRef([`createdAt`]), new Date(`2022-01-01`)), + })! + + const ids = result.map((r) => r.value.id).sort() + expect(ids).toEqual([`invalid`, `valid`]) + }) }) describe(`Index Usage Verification`, () => { diff --git a/packages/db/tests/comparison.test.ts b/packages/db/tests/comparison.test.ts new file mode 100644 index 0000000000..ae40488f43 --- /dev/null +++ b/packages/db/tests/comparison.test.ts @@ -0,0 +1,41 @@ +import { describe, expect, it } from 'vitest' +import { ascComparator, defaultComparator } from '../src/utils/comparison' +import { DEFAULT_COMPARE_OPTIONS } from '../src/utils' + +describe(`ascComparator - PostgreSQL float semantics for NaN`, () => { + const opts = DEFAULT_COMPARE_OPTIONS // nulls: `first` + + it(`orders NaN greater than every number`, () => { + expect(ascComparator(NaN, 5, opts)).toBeGreaterThan(0) + expect(ascComparator(5, NaN, opts)).toBeLessThan(0) + }) + + it(`treats NaN as equal to NaN`, () => { + expect(ascComparator(NaN, NaN, opts)).toBe(0) + }) + + it(`produces a stable total order with NaN sorting last`, () => { + const sorted = [3, NaN, 1, 5, NaN].sort((a, b) => defaultComparator(a, b)) + + expect(sorted.slice(0, 3)).toEqual([1, 3, 5]) + expect(sorted.slice(3).every((v) => Number.isNaN(v))).toBe(true) + }) + + it(`keeps null before non-null values regardless of NaN`, () => { + // nulls still sort first by default; NaN sorts last (greatest non-null) + const sorted = [5, NaN, null, 1].sort((a, b) => defaultComparator(a, b)) + + expect(sorted[0]).toBe(null) + expect(sorted[1]).toBe(1) + expect(sorted[2]).toBe(5) + expect(Number.isNaN(sorted[3])).toBe(true) + }) + + it(`orders an invalid Date greater than valid Dates`, () => { + const invalid = new Date(`not a date`) + const valid = new Date(`2023-01-01`) + + expect(ascComparator(invalid, valid, opts)).toBeGreaterThan(0) + expect(ascComparator(valid, invalid, opts)).toBeLessThan(0) + }) +}) diff --git a/packages/db/tests/deterministic-ordering.test.ts b/packages/db/tests/deterministic-ordering.test.ts index 9ce9a326d6..160d03284b 100644 --- a/packages/db/tests/deterministic-ordering.test.ts +++ b/packages/db/tests/deterministic-ordering.test.ts @@ -489,5 +489,38 @@ describe(`Deterministic Ordering`, () => { const keys = changes?.map((c) => c.key) expect(keys).toEqual([`a`, `b`, `c`]) }) + + it(`should place NaN values consistently when ordering`, () => { + type Item = { id: string; score: number } + + const options = mockSyncCollectionOptions({ + id: `test-collection-changes-nan`, + getKey: (item) => item.id, + initialData: [], + }) + + const collection = createCollection(options) + + options.utils.begin() + options.utils.write({ type: `insert`, value: { id: `a`, score: 5 } }) + options.utils.write({ type: `insert`, value: { id: `nan`, score: NaN } }) + options.utils.write({ type: `insert`, value: { id: `b`, score: 1 } }) + options.utils.write({ type: `insert`, value: { id: `c`, score: 3 } }) + options.utils.commit() + + const changes = collection.currentStateAsChanges({ + orderBy: [ + { + expression: new PropRef([`score`]), + compareOptions: { direction: `asc`, nulls: `first` }, + }, + ], + }) + + // Under PostgreSQL float semantics NaN is the greatest value, so the + // numbers sort ascending first and NaN sorts last. + const keys = changes?.map((c) => c.key) + expect(keys).toEqual([`b`, `c`, `a`, `nan`]) + }) }) }) diff --git a/packages/db/tests/query/compiler/evaluators.test.ts b/packages/db/tests/query/compiler/evaluators.test.ts index 69969de18a..4c5acb78c1 100644 --- a/packages/db/tests/query/compiler/evaluators.test.ts +++ b/packages/db/tests/query/compiler/evaluators.test.ts @@ -730,6 +730,87 @@ describe(`evaluators`, () => { expect(compiled({})).toBe(null) }) }) + + describe(`NaN (PostgreSQL float semantics)`, () => { + // Following PostgreSQL, NaN is equal to itself and greater than every + // other (non-null) value, so it has a well-defined order. + it(`treats NaN as equal to NaN`, () => { + const func = new Func(`eq`, [new Value(NaN), new Value(NaN)]) + expect(compileExpression(func)({})).toBe(true) + }) + + it(`treats NaN as not equal to a number`, () => { + const func = new Func(`eq`, [new Value(NaN), new Value(5)]) + expect(compileExpression(func)({})).toBe(false) + }) + + it(`still returns UNKNOWN when comparing NaN with null`, () => { + const func = new Func(`eq`, [new Value(NaN), new Value(null)]) + expect(compileExpression(func)({})).toBe(null) + }) + + it(`treats NaN as greater than every number`, () => { + expect( + compileExpression(new Func(`gt`, [new Value(NaN), new Value(5)]))( + {}, + ), + ).toBe(true) + expect( + compileExpression(new Func(`gt`, [new Value(5), new Value(NaN)]))( + {}, + ), + ).toBe(false) + expect( + compileExpression( + new Func(`gt`, [new Value(NaN), new Value(NaN)]), + )({}), + ).toBe(false) + }) + + it(`orders NaN with gte/lt/lte consistently`, () => { + // NaN >= anything (including NaN); nothing finite >= NaN + expect( + compileExpression( + new Func(`gte`, [new Value(NaN), new Value(5)]), + )({}), + ).toBe(true) + expect( + compileExpression( + new Func(`gte`, [new Value(NaN), new Value(NaN)]), + )({}), + ).toBe(true) + // NaN < nothing; a finite value < NaN + expect( + compileExpression(new Func(`lt`, [new Value(NaN), new Value(5)]))( + {}, + ), + ).toBe(false) + expect( + compileExpression(new Func(`lt`, [new Value(5), new Value(NaN)]))( + {}, + ), + ).toBe(true) + // NaN <= NaN; a finite value <= NaN + expect( + compileExpression( + new Func(`lte`, [new Value(NaN), new Value(NaN)]), + )({}), + ).toBe(true) + expect( + compileExpression( + new Func(`lte`, [new Value(5), new Value(NaN)]), + )({}), + ).toBe(true) + }) + + it(`matches NaN inside an IN list`, () => { + const func = new Func(`in`, [ + new Value(NaN), + new Value([NaN, 1, 2]), + ]) + expect(compileExpression(func)({})).toBe(true) + }) + }) }) describe(`boolean operators`, () => {