Skip to content

Use aggregate descriptors for zoned stats#7938

Merged
gatesn merged 6 commits into
developfrom
ngates/stats-7707/aggregate-zoned-stats
Jun 19, 2026
Merged

Use aggregate descriptors for zoned stats#7938
gatesn merged 6 commits into
developfrom
ngates/stats-7707/aggregate-zoned-stats

Conversation

@gatesn

@gatesn gatesn commented May 15, 2026

Copy link
Copy Markdown
Contributor

This PR adds a new layout type: vortex.zoned

During deserialization, we read the old zone map layout vortex.stats into a vortex.zoned in memory.

The new zoned layout now stores aggregate descriptors instead of legacy Stat bits, enabling arbitrary aggregate functions to be used as zone statistics.

FLUPs:

  • Move FooterStats over to using AggregateFns. Or even just a root-level ZonedLayout with a single zone?

@gatesn gatesn added the changelog/feature A new feature label May 15, 2026
@gatesn gatesn mentioned this pull request May 15, 2026
42 tasks
@gatesn gatesn force-pushed the ngates/stats-7707/aggregate-zoned-stats branch from 772d188 to 6e1ab90 Compare May 15, 2026 03:51
@gatesn gatesn force-pushed the ngates/stats-7707/zoned-statfn-pruning branch from c69b282 to f9cf88c Compare May 15, 2026 03:51
@codspeed-hq

codspeed-hq Bot commented May 15, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 11 improved benchmarks
❌ 7 regressed benchmarks
✅ 1563 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation decompress_rd[f64, (10000, 0.01)] 108.7 µs 138.8 µs -21.73%
Simulation decompress_rd[f64, (10000, 0.1)] 109 µs 139.2 µs -21.71%
Simulation decompress_rd[f64, (10000, 0.0)] 108.7 µs 138.8 µs -21.7%
Simulation decompress_rd[f32, (100000, 0.0)] 496 µs 583.5 µs -15%
Simulation decompress_rd[f32, (10000, 0.1)] 78.1 µs 90.3 µs -13.56%
Simulation decompress_rd[f32, (10000, 0.01)] 78.1 µs 90.3 µs -13.56%
Simulation decompress_rd[f32, (10000, 0.0)] 78.5 µs 90.8 µs -13.52%
Simulation take_10k_first_chunk_only 253.1 µs 208.2 µs +21.55%
Simulation decompress_rd[f64, (100000, 0.1)] 1,020.7 µs 842.5 µs +21.15%
Simulation decompress_rd[f64, (100000, 0.01)] 1,020.7 µs 842.5 µs +21.15%
Simulation take_10k_dispersed 266.6 µs 221.8 µs +20.18%
Simulation decompress_rd[f32, (100000, 0.1)] 582.9 µs 495 µs +17.75%
Simulation decompress_rd[f32, (100000, 0.01)] 582.9 µs 495 µs +17.75%
Simulation bitwise_not_vortex_buffer_mut[128] 244.4 ns 215.3 ns +13.55%
Simulation patched_take_10k_adversarial 260.9 µs 230.5 µs +13.16%
Simulation patched_take_10k_first_chunk_only 284.7 µs 254.4 µs +11.92%
Simulation patched_take_10k_dispersed 298.9 µs 268.6 µs +11.28%
Simulation bitwise_not_vortex_buffer_mut[1024] 304.7 ns 275.6 ns +10.58%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ngates/stats-7707/aggregate-zoned-stats (02e7409) with develop (5d55fbc)

Open in CodSpeed

@gatesn gatesn force-pushed the ngates/stats-7707/aggregate-zoned-stats branch 2 times, most recently from bc8f53c to 6ceb393 Compare May 15, 2026 08:46
@gatesn gatesn force-pushed the ngates/stats-7707/zoned-statfn-pruning branch from f9cf88c to ed63b11 Compare May 15, 2026 08:46
@gatesn gatesn force-pushed the ngates/stats-7707/aggregate-zoned-stats branch from 6ceb393 to 549e240 Compare May 15, 2026 09:16
@gatesn gatesn force-pushed the ngates/stats-7707/zoned-statfn-pruning branch from ed63b11 to 0d3003c Compare May 15, 2026 09:16
@gatesn gatesn force-pushed the ngates/stats-7707/aggregate-zoned-stats branch from 549e240 to 8b3da0d Compare May 15, 2026 09:23
@gatesn gatesn force-pushed the ngates/stats-7707/zoned-statfn-pruning branch from 0d3003c to 096128a Compare May 15, 2026 09:23
@tl-dr-review tl-dr-review Bot added the tldr label May 15, 2026
@gatesn gatesn added tldr and removed tldr labels May 15, 2026
@gatesn gatesn force-pushed the ngates/stats-7707/zoned-statfn-pruning branch from 096128a to 68e19bb Compare May 19, 2026 20:46
@gatesn gatesn force-pushed the ngates/stats-7707/aggregate-zoned-stats branch from 8b3da0d to 6abd91b Compare May 19, 2026 20:46
@gatesn gatesn force-pushed the ngates/stats-7707/zoned-statfn-pruning branch from 68e19bb to 0473953 Compare May 27, 2026 00:40
@gatesn gatesn force-pushed the ngates/stats-7707/aggregate-zoned-stats branch from 6abd91b to c04ebf9 Compare May 27, 2026 00:51
@gatesn gatesn force-pushed the ngates/stats-7707/zoned-statfn-pruning branch from 0473953 to f14f6a1 Compare May 27, 2026 00:56
@gatesn gatesn force-pushed the ngates/stats-7707/aggregate-zoned-stats branch from c04ebf9 to e3de60b Compare May 27, 2026 00:58
@gatesn gatesn force-pushed the ngates/stats-7707/zoned-statfn-pruning branch from f14f6a1 to fc97c20 Compare May 27, 2026 01:41
@gatesn gatesn force-pushed the ngates/stats-7707/aggregate-zoned-stats branch 3 times, most recently from c112569 to 95145a1 Compare May 28, 2026 15:51
Base automatically changed from ngates/stats-7707/zoned-statfn-pruning to develop May 31, 2026 23:14
@github-actions

Copy link
Copy Markdown
Contributor

This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days

@github-actions github-actions Bot added the stale This PR is stale and will be auto-closed soon label Jun 15, 2026
@gatesn gatesn removed the stale This PR is stale and will be auto-closed soon label Jun 17, 2026
Signed-off-by: "Nicholas Gates" <nick@nickgates.com>

Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn gatesn force-pushed the ngates/stats-7707/aggregate-zoned-stats branch from eb8f266 to 1f13453 Compare June 18, 2026 20:37
@gatesn gatesn marked this pull request as ready for review June 18, 2026 20:51
@gatesn gatesn requested a review from a team June 18, 2026 20:51
@gatesn gatesn added changelog/break A breaking API change and removed changelog/feature A new feature tldr labels Jun 18, 2026

@robert3005 robert3005 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can delete some of the old write path? We no longer seem to be using old stats accumulators

Comment thread vortex-file/src/footer/mod.rs Outdated
Comment thread vortex-layout/src/layouts/zoned/builder.rs Outdated
Move the legacy Stat accumulator out of the zoned builder and into file stats, where it is still needed for footer statistics. Also derive the layout registry from VortexSession when loading layout flatbuffers.

Signed-off-by: "Nicholas Gates" <nick@nickgates.com>

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Comment thread vortex-file/src/tests.rs Outdated
Install LayoutSession in the unknown-layout test fixture now that layout deserialization resolves the registry through VortexSession.

Signed-off-by: "Nicholas Gates" <nick@nickgates.com>

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Comment thread vortex-layout/src/layouts/zoned/builder.rs Outdated
Comment thread vortex-layout/src/layouts/zoned/schema.rs
Comment thread vortex-layout/src/layouts/zoned/schema.rs
Comment thread vortex-layout/src/layouts/zoned/builder.rs
gatesn added 3 commits June 19, 2026 09:39
Remove the aggregate descriptor helper and cached descriptor field from the zoned stats builder. Use AggregateFnRef Display directly for stats-table field names, and use const NonZeroUsize construction for bounded stat byte defaults.

Signed-off-by: "Nicholas Gates" <nick@nickgates.com>

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Remove the separate in-memory AggregateSpec wrapper. Keep the prost aggregate spec as the raw metadata form, deserialize it into AggregateFnRef during layout build, and store aggregate functions on the runtime layout.

Signed-off-by: "Nicholas Gates" <nick@nickgates.com>

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Replace encoding-id string matching with layout.is::<Zoned>() and layout.is::<LegacyStats>() in the zoned segment-ordering test.

Signed-off-by: "Nicholas Gates" <nick@nickgates.com>

Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn gatesn enabled auto-merge (squash) June 19, 2026 13:55
@gatesn gatesn merged commit 4187c1a into develop Jun 19, 2026
73 checks passed
@gatesn gatesn deleted the ngates/stats-7707/aggregate-zoned-stats branch June 19, 2026 14:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/break A breaking API change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants