Implement Adaptive Radix Tree (ART) Indexes

[adsharma]

Details in the included docs.

du -sh /tmp/test1*.db
634M    /tmp/test1-hash.db
512M    /tmp/test1-noindex.db
604M    /tmp/test1-art.db

➜  ladybug git:(art_index) ✗ ./build/release/tools/shell/lbug -r /tmp/test1-hash.db
Opening the database at path: /tmp/test1-hash.db in read-only mode.
Enter ":help" for usage hints.
lbug> CALL show_indexes() return *;
┌────────────┬──────────────┬────────────┬────────────────┬──────────────────┬──────────────────────────────────────────────────────────────┐
│ table_name │ index_name   │ index_type │ property_names │ extension_loaded │ index_definition                                             │
│ STRING     │ STRING       │ STRING     │ STRING[]       │ BOOL             │ STRING                                                       │
├────────────┼──────────────┼────────────┼────────────────┼──────────────────┼──────────────────────────────────────────────────────────────┤
│ User       │ user_hash_pk │ HASH       │ [id]           │ True             │ CREATE HASH INDEX `user_hash_pk` FOR (n:`User`) ON (n.`id`); │
└────────────┴──────────────┴────────────┴────────────────┴──────────────────┴──────────────────────────────────────────────────────────────┘
(1 tuple)
(6 columns)
Time: 5.56ms (compiling), 1.69ms (executing)
lbug>
➜  ladybug git:(art_index) ✗ ./build/release/tools/shell/lbug -r /tmp/test1-art.db
Opening the database at path: /tmp/test1.db in read-only mode.
Enter ":help" for usage hints.
lbug> CALL show_indexes() return *;
┌────────────┬───────────────┬────────────┬────────────────┬──────────────────┬──────────────────────────────────────────────────────────────┐
│ table_name │ index_name    │ index_type │ property_names │ extension_loaded │ index_definition                                             │
│ STRING     │ STRING        │ STRING     │ STRING[]       │ BOOL             │ STRING                                                       │
├────────────┼───────────────┼────────────┼────────────────┼──────────────────┼──────────────────────────────────────────────────────────────┤
│ User       │ idx_person_pk │ ART        │ [id]           │ True             │ CREATE ART INDEX `idx_person_pk` FOR (n:`User`) ON (n.`id`); │
└────────────┴───────────────┴────────────┴────────────────┴──────────────────┴──────────────────────────────────────────────────────────────┘
(1 tuple)
(6 columns)
Time: 2.08ms (compiling), 0.21ms (executing)
lbug>
➜  ladybug git:(art_index) ✗ ./build/release/tools/shell/lbug -r /tmp/test1-noindex.db
Opening the database at path: /tmp/test1-save.db in read-only mode.
Enter ":help" for usage hints.
lbug> CALL show_indexes() return *;
┌────────────┬────────────┬────────────┬────────────────┬──────────────────┬──────────────────┐
│ table_name │ index_name │ index_type │ property_names │ extension_loaded │ index_definition │
│ STRING     │ STRING     │ STRING     │ STRING[]       │ BOOL             │ STRING           │
├────────────┼────────────┼────────────┼────────────────┼──────────────────┼──────────────────┤
└────────────┴────────────┴────────────┴────────────────┴──────────────────┴──────────────────┘
(0 tuples)
(6 columns)
Time: 1.50ms (compiling), 0.16ms (executing)

[adsharma]

random_lookup_bench.py --backend pybind --literal --lookups 500 --warmup 50


  Literal benchmark cross-check, 500 lookups:

  base  1996.8/s  avg=0.501ms p95=0.615ms
  hash  1227.3/s  avg=0.815ms p95=1.802ms
  art  12029.0/s  avg=0.083ms p95=0.095ms

Without --literal  
  
  base  1405.7/s avg=0.711ms p95=0.842ms
  hash  1350.4/s avg=0.740ms p95=0.959ms
  art   1424.0/s avg=0.702ms p95=0.817ms

So with literal constants, ART is clearly being used and is much faster. The prepared benchmark is still mostly tied, which suggests the prepared/parameter path or Python execution overhead is masking the index difference. Hash being slower than base in the literal run is also notable.

https://gist.github.com/adsharma/83a1b7c9c320d829e349135d396ab4d3

[adsharma]

ART index also enables range scans. Previously ladybug/kuzu didn't support range scans via indexes.

[aheev]

queries after COPY FROM are failing

-CASE ArtIndexCopyFrom
-STATEMENT CALL enable_default_hash_index=false;
---- ok
-STATEMENT CREATE NODE TABLE art_copy_person (ID INT64, name STRING, PRIMARY KEY(ID));
---- 1
Table art_copy_person has been created.
-STATEMENT CREATE ART INDEX art_copy_person_pk FOR (p:art_copy_person) ON (p.ID);
---- 1
Index art_copy_person_pk has been created.
-STATEMENT COPY art_copy_person FROM "${LBUG_ROOT_DIRECTORY}/dataset/art-index-test/person.csv";
---- 1
4 tuples have been copied to the art_copy_person table.
-STATEMENT MATCH (p:art_copy_person) WHERE p.ID = 2 RETURN p.name;
---- 1
Grace
-STATEMENT MATCH (p:art_copy_person) WHERE p.ID >= 1 AND p.ID <= 10 RETURN p.ID, p.name ORDER BY p.ID;
---- 3
1|Ada
2|Grace
10|Barbara
-STATEMENT MATCH (p:art_copy_person) WHERE p.ID < 2 RETURN p.ID, p.name ORDER BY p.ID;
---- 2
-1|Edsger
1|Ada
-STATEMENT CALL enable_default_hash_index=true;
---- ok

person.csv

We should also look at thread safety

[adsharma]

Key changes:

• ART index creation now validates the physical contract: built-in primary-key index, exactly one indexed column, exactly one key type, supported scalar key
  type.
• Range pushdown now only uses ART range scan for validated simple shapes: at most one constant lower bound and one constant upper bound. Duplicate same-side
  bounds or complex/non-constant predicates stay as normal filters.
• Consumed valid range predicates are now removed from the residual predicate set.
• ART public lookup/scan/insert/discard/checkpoint paths now take a coarse mutex for thread safety.
• Moved initInsertState out of the header and marked ArtPrimaryKeyIndex::load with LBUG_API.
• Reworked range traversal to iterate actual children instead of probing all 256 byte values through getChild.
• Rollback/discard cleanup now prunes empty child nodes; it does not shrink node kind layouts, which I’d keep as a separate memory-layout refactor.
• Updated docs/art_index.md with unsupported shapes and optimizer limitations.
• Added e2e coverage for unsupported ART creation, duplicate/complex range predicate behavior, and kept the COPY regression.

[adsharma]

Need refactor the pathological unique_ptr<> deallocation pattern in this PR. Eventually we should use a more efficient purpose designed memory allocator for the nodes. But for now, I'll try to avoid the long pause on shell exit.