bench: gate CodSpeed-unstable canonicalization microbenchmarks from simulation

[connortsui20]

Summary

A small, recurring set of CodSpeed Simulation microbenchmarks report false-positive regressions in nearly every PR, regardless of what the PR changes. This PR gates the provably-unfixable offenders out of CodSpeed simulation (keeping them runnable via local cargo bench), and documents the rest with a keep/remove decision for each.

Note

Draft for discussion. The analysis below justifies every benchmark I touched and explains the ones I deliberately left alone.

How I found them

I read CodSpeed's bot comment on 9 recent PRs spanning unrelated areas (scalar ops, SIMD take, GPU/CUDA, session refactors, stats rules, random-access splitting, etc.): #8345, #8454, #8470, #8478, #8483, #8489, #8496, #8500, #8505, #8511.

A benchmark that shows a large change in a PR that doesn't touch its code is, by definition, noise. The tell is when the same unchanged code reports very different numbers depending only on which commit it was built against. Every one of these comparisons also carried CodSpeed's own ⚠️ "Different runtime environments detected" warning — and this PR itself reproduced it (see Self-validation below).

The flaky set (each row = identical code, different measured value)

Benchmark	File	Observed range (unchanged code)	PRs	Verdict
decompress_rd[*]	encodings/alp/benches/alp_compress.rs	f64,(100k) 842 ↔ 1025 µs; (10k) 108 ↔ 139 µs	7/9	Gate
chunked_varbinview_* (×4)	vortex-array/benches/chunk_array_builder.rs	(1000,10) 161 ↔ 198; (100,100) 223 ↔ 308 µs	6/9	Gate
chunked_bool_canonical_into	vortex-array/benches/chunk_array_builder.rs	16 ↔ 35 µs (~2×)	4/9	Gate
take_10k_*, patched_take_10k_* (the 4 flaky ones)	encodings/fastlanes/benches/bitpacking_take.rs	bimodal 195 ↔ 255 µs	4/9 + this PR	Gate
baseline_eq/lt[*]	encodings/fastlanes/benches/bitpack_compare.rs	243 ↔ 185, 303 ↔ 245 µs	this PR	Keep (this PR)
varbinview_large	vortex-array/benches/listview_rebuild.rs	112 ↔ 131 µs	3/9	Keep (this PR)
bitwise_not_vortex_buffer_mut[128]	vortex-buffer/benches/vortex_bitbuffer.rs	186 ↔ 244 ns	2/9	Keep (this PR)
sum_i32_nullable_all_valid, chunked_dict_*, encode_varbin*, null_count_run_end	various	±34–95%	2/9, only vs base 679e2c5	Keep (lower confidence)

Root cause

CodSpeed Simulation estimates CPU cycles from an instruction trace (Cachegrind-style). That trace is deterministic for a fixed binary in a fixed environment, but it includes instructions executed inside glibc — and memcpy/memmove/memset are resolved at runtime via ifunc to a SIMD variant chosen by the runner's CPU/glibc. When the develop baseline and the PR run on different runner images (which CodSpeed explicitly flags here as "different runtime environments"), any benchmark whose hot path is heap allocation + byte copying (or a tight SIMD loop sensitive to code layout) changes instruction count even though the Vortex code is byte-identical. See CodSpeed's own write-up, "Why glibc is faster on some GitHub Actions Runners".

This predicts exactly which benchmarks flake, and the data confirms it:

• decompress_rd flakes; compress_rd never does. Decode materializes a fresh canonical output buffer (alloc + tight SIMD/memcpy); encode is compute-bound (sampling, dictionary build). Same file, same data — only the copy-bound one is noisy.
• chunked_varbinview_* flakes — canonicalizing chunked strings is concatenating variable-length bytes into one buffer (memcpy-dominated).
• chunked_bool_canonical_into flakes hardest in relative terms — it's also just too small (~16–35 µs).
• take_10k_* is bimodal — 10k gather + canonicalize: output alloc/memcpy plus the SIMD bit-unpack's code-layout sensitivity.

The data movement is the thing being measured, so these cannot be made stable under Simulation by tweaking inputs. docs/developer-guide/benchmarking.md already prescribes the remedy: "Use #[cfg(not(codspeed))] for benchmarks that are incompatible with CodSpeed."

Changes (every gated benchmark justified)

All gated benches stay fully available via local cargo bench — only CodSpeed CI stops tracking them.

encodings/alp/benches/alp_compress.rs

• decompress_rd → gated. The Fix build after move #1 offender: moved in 7/9 sampled PRs, bidirectionally, 842↔1025 µs for identical code. Decodes to a canonical array → output-buffer alloc + memcpy-dominated. compress_rd (encode, compute-bound, never flagged) is kept.

vortex-array/benches/chunk_array_builder.rs

• chunked_varbinview_{canonical_into,into_canonical,opt_canonical_into,opt_into_canonical} → gated. All four flake across 6/9 PRs; all are memcpy-bound string canonicalization.
• chunked_bool_canonical_into → gated. Worst relative swings (~2×, 16↔35 µs) and below the Simulation noise floor.
• Kept: chunked_opt_bool_*, chunked_constant_* — compute-bound, never flagged.

encodings/fastlanes/benches/bitpacking_take.rs

• take_10k_random, take_10k_contiguous, patched_take_10k_random, patched_take_10k_contiguous_patches → gated. Bimodal ~195↔255 µs (±23%); fired in 4/9 sampled PRs and again on this PR. The other take variants in the file have not shown this instability and are kept (they remain useful for the sparse/bitpacked-take threshold tuning the in-file comment describes).

Self-validation (this PR's own CodSpeed run)

The first push proved the thesis on itself: CodSpeed reported "will not alter performance", correctly marked the gated benches as skipped, and the only flagged moves were pre-existing flaky benches in files this PR never touched — take_10k_* (±20–23%), baseline_eq/lt (±23–31%), varbinview_large (±17%). The take_10k gating in this PR was added in response to that run; the others are documented below.

Deliberately kept (flaky ≠ delete)

• bitpack_compare.rs baseline_eq/lt[*] and listview_rebuild.rs varbinview_large — same cross-environment root cause, surfaced on this PR. Left for a focused follow-up to keep this PR's scope contained; varbinview_large was only seen against a couple of bases.
• bitwise_not_vortex_buffer_mut[128] — sub-µs noise, but its INPUT_SIZE is shared by ~25 valid vortex-vs-arrow comparison benches; not worth restructuring for a ±30 ns wobble.
• CUDA walltime benches (e.g. cuda/bitpacked_u8/unpack/3bw[100M]) — these use the WallTime instrument on non-macro GPU runners (you can't simulate a GPU); a separate, intentional trade-off and out of scope.

Verification

Run locally with the pinned nightly + cargo-codspeed, valgrind present, for all three suites:

• ✅ cargo codspeed build (Simulation) succeeds for vortex-alp, vortex-array (--features _test-harness), and vortex-fastlanes — no warnings, no orphaned imports/dead code.
• ✅ cargo codspeed run (Measurement mode: Simulation) executes the suites to completion — gated benches absent at runtime; the kept benches (compress_rd, chunked_opt_bool_*, chunked_constant_*, the other take variants) still measure.
• ✅ Binary symbol check confirms the gated benches are compiled out under --cfg codspeed and present without it.
• ✅ Local cargo bench build path, cargo +nightly fmt --check, and cargo clippy --no-deps on all changed bench targets are clean. (Full-workspace clippy hits an unrelated pre-existing lint in vortex-buffer under this environment's toolchain; not touched here.)

Follow-ups for maintainers

1. Archive the gated benches on CodSpeed so they drop off the dashboard instead of showing as "skipped".
2. The broader class (canonicalization/builder/decode-to-canonical/copy-bound) shares this root cause. The durable fix is infrastructure-level: ensure the develop baseline and PR runs use the same pinned runner image (so "different runtime environments" stops firing), or move these to CodSpeed walltime/macro runners. If that lands, several "kept" benches above could return to Simulation.

[codspeed-hq]

Merging this PR will improve performance by 13.57%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 3 improved benchmarks
✅ 1547 untouched benchmarks
⏩ 31 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	Simulation	varbinview_large	130.8 µs	112.1 µs	+16.65%
⚡	Simulation	bitwise_not_vortex_buffer_mut[128]	244.4 ns	215.3 ns	+13.55%
⚡	Simulation	bitwise_not_vortex_buffer_mut[1024]	304.7 ns	275.6 ns	+10.58%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing claude/sharp-planck-i4ifv8 (17ae8e0) with develop (5df0f94)}

Footnotes

1. 31 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩