perf(scan): intra-file decode parallelism — sub-split large chunk spans

[lukekim]

Summary

SplitBy::Layout now sub-divides any span between adjacent chunk boundaries wider than IDEAL_SPLIT_SIZE (100k rows) into evenly sized row-range splits, so a file with few large chunks (e.g. a single flat layout, or byte-targeted int columns that coalesce to ~262k rows/chunk) decodes across multiple cores instead of one.

Correctness

Subdivision only inserts points strictly between existing adjacent boundaries — it never moves or removes one — so the half-open ranges consumers derive (tuple_windows) remain a contiguous, non-overlapping, exact partition of the same rows. Spans at or below the cap pass through untouched (fast-path no-op). All boundary consumers (RepeatedScan, VortexFile::splits(), the DataFusion repartitioner) operate on arbitrary ranges; sub-chunk ranges were already exercised by SplitBy::RowCount. The arithmetic saturates at u64::MAX.

API/Observable behavior

For files whose merged (projected-column) chunk boundaries leave spans > 100k rows, VortexFile::splits() (incl. Python bindings) returns more, smaller ranges; scans emit smaller batches; DataFusion gets real repartitioning where a single-chunk file previously collapsed to one partition. Fine-grained files (e.g. ~8k-row string chunks from the default 1 MiB block target) are untouched.

Testing

• Unit/property/overflow tests for subdivide_large_spans (no-op, large single chunk, mixed gaps, exact-coverage property, u64::MAX boundary).
• E2E: 250k-row single flat chunk → splits all ≤ the cap, contiguous, exact endpoints; full + filtered scans match the unsplit data.
• E2E (rstest): fixed-size SplitBy::RowCount scans (unaligned 33,333 and exceeds-file 300,000 cases).
• E2E: ~120-byte string column via the default write strategy keeps its natural fine-grained chunk splits (bounded relative to the cap, not to writer defaults).
• cargo nextest run -p vortex-layout -p vortex-file — 174 passed on this branch.

[codspeed-hq]

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 4 improved benchmarks
❌ 1 regressed benchmark
✅ 1584 untouched benchmarks
⏩ 4 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	bitwise_not_vortex_buffer_mut[128]	244.4 ns	273.6 ns	-10.66%
⚡	Simulation	encode_varbin[(1000, 4)]	157 µs	139.8 µs	+12.34%
⚡	Simulation	encode_varbin[(1000, 32)]	162.5 µs	144.8 µs	+12.22%
⚡	Simulation	encode_varbin[(1000, 8)]	157.3 µs	140.4 µs	+12.05%
⚡	Simulation	encode_varbin[(1000, 2)]	156.1 µs	140.7 µs	+10.98%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing spiceai:lukim/scan-split-large-chunks-develop (ca2aa8b) with develop (e2478aa)}

Footnotes

1. 4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[robert3005]

Sorry for the delay reviewing it. I think this is good to go. You will need to rebase/merge develop