Hybrid index implementation

[kevin-dp]

This PR modifies the index such that it is a composition of two indexes:

• a single value index
• a multi value hash index

As long as a key only has a single value associated to it, it is stored in the single value index and no hashes are computed. Once a key has two or more distinct values, it is moved to the multi value index where they are distinguished based on their hash.

This ensures that objects are distinguished by structural equality (as is needed by differential dataflow) and not by referential equality (as is the case with objects in JS). Thus guaranteeing correctness while also minimizing the amount of hashes that are computed.

[changeset-bot]

🦋 Changeset detected

Latest commit: 1afdf00

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 9 packages

Name	Type
@tanstack/db-ivm	Patch
@tanstack/db	Patch
@tanstack/electric-db-collection	Patch
@tanstack/query-db-collection	Patch
@tanstack/react-db	Patch
@tanstack/solid-db	Patch
@tanstack/svelte-db	Patch
@tanstack/trailbase-db-collection	Patch
@tanstack/vue-db	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[pkg-pr-new]

More templates

• tanstack-start-db-projects-example
• @tanstack/db-example-react-todo
• @tanstack/db-example-solid-todo

@tanstack/db

npm i https://pkg.pr.new/@tanstack/db@489

@tanstack/db-ivm

npm i https://pkg.pr.new/@tanstack/db-ivm@489

@tanstack/electric-db-collection

npm i https://pkg.pr.new/@tanstack/electric-db-collection@489

@tanstack/query-db-collection

npm i https://pkg.pr.new/@tanstack/query-db-collection@489

@tanstack/react-db

npm i https://pkg.pr.new/@tanstack/react-db@489

@tanstack/solid-db

npm i https://pkg.pr.new/@tanstack/solid-db@489

@tanstack/svelte-db

npm i https://pkg.pr.new/@tanstack/svelte-db@489

@tanstack/trailbase-db-collection

npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@489

@tanstack/vue-db

npm i https://pkg.pr.new/@tanstack/vue-db@489

commit: 1afdf00

[github-actions]

Size Change: +16 B (+0.02%)

Total Size: 65 kB

Filename	Size	Change
./packages/db/dist/esm/query/compiler/joins.js	2.36 kB	+16 B (+0.68%)

ℹ️ View Unchanged

Filename	Size
./packages/db/dist/esm/change-events.js	1.13 kB
./packages/db/dist/esm/collection.js	10.5 kB
./packages/db/dist/esm/deferred.js	230 B
./packages/db/dist/esm/errors.js	3.1 kB
./packages/db/dist/esm/index.js	1.52 kB
./packages/db/dist/esm/indexes/auto-index.js	745 B
./packages/db/dist/esm/indexes/base-index.js	605 B
./packages/db/dist/esm/indexes/btree-index.js	1.74 kB
./packages/db/dist/esm/indexes/lazy-index.js	1.25 kB
./packages/db/dist/esm/local-only.js	827 B
./packages/db/dist/esm/local-storage.js	2.03 kB
./packages/db/dist/esm/optimistic-action.js	294 B
./packages/db/dist/esm/proxy.js	3.87 kB
./packages/db/dist/esm/query/builder/functions.js	575 B
./packages/db/dist/esm/query/builder/index.js	3.83 kB
./packages/db/dist/esm/query/builder/ref-proxy.js	890 B
./packages/db/dist/esm/query/compiler/evaluators.js	1.48 kB
./packages/db/dist/esm/query/compiler/expressions.js	631 B
./packages/db/dist/esm/query/compiler/group-by.js	2.09 kB
./packages/db/dist/esm/query/compiler/index.js	2.16 kB
./packages/db/dist/esm/query/compiler/order-by.js	1.23 kB
./packages/db/dist/esm/query/compiler/select.js	655 B
./packages/db/dist/esm/query/ir.js	466 B
./packages/db/dist/esm/query/live-query-collection.js	333 B
./packages/db/dist/esm/query/live/collection-config-builder.js	2.58 kB
./packages/db/dist/esm/query/live/collection-subscriber.js	2.4 kB
./packages/db/dist/esm/query/optimizer.js	2.63 kB
./packages/db/dist/esm/SortedMap.js	1.24 kB
./packages/db/dist/esm/transactions.js	2.29 kB
./packages/db/dist/esm/utils.js	943 B
./packages/db/dist/esm/utils/btree.js	6.02 kB
./packages/db/dist/esm/utils/comparison.js	718 B
./packages/db/dist/esm/utils/index-optimization.js	1.62 kB

_{compressed-size-action::db-package-size}

[github-actions]

Size Change: 0 B

Total Size: 1.18 kB

ℹ️ View Unchanged

Filename	Size
./packages/react-db/dist/esm/index.js	152 B
./packages/react-db/dist/esm/useLiveQuery.js	1.02 kB

_{compressed-size-action::react-db-package-size}

[samwillis]

This is great.

My main comment is that hashIndex.ts and valueIndex.ts are small files, and indexes.ts isn't that large, could we fold them into indexes.ts?

If you would prefer to keep them separate can we name them hash-index.ts and value-index.ts.

Other than that it's great. let get it merged!

[kevin-dp]

@samwillis i would prefer to keep them as separate files. Regarding the naming of the files, it seems that the db-ivm package uses camelCase also for file names (see for instance topKWithFractionalIndex.ts), probably a remnant from the past. Do you want me to fix all file names to be hyphenated instead of camelCased?

[samwillis]

ah, ok lets leave as is 👍