Stratified sampling and batched raster points in the native core (ABI v10) - #6
Merged
Merged
Conversation
… v10) Two Python hot paths move into the Rust core: - fc_stratified_sample_mask fuses the per-category sampling loop behind categorical density overlays (sqrt-scaled hash thresholds plus the lowest-hash min_per_category floor) into one native pass, replacing O(n * categories) NumPy rescans. Small non-negative integer categories skip np.unique and serve directly as group codes. Bit-identical masks, parity-tested on both sides of the ABI; ~20x faster at 5M rows / 12 categories (168 ms -> 8.5 ms). - OP_POINTS batches scatter marks in the PNG display list as one struct-of-arrays command, replacing the per-point struct.pack loop and per-point CSS color re-parse for categorical palettes. Pixel-identical to per-mark OP_POINT (parity-tested); display-list build for a 100k-point categorical scatter drops ~186 ms -> ~1 ms. Also reformats examples/demo.ipynb, which was failing the ruff format gate on main and blocking make check-full.
Alek99
pushed a commit
that referenced
this pull request
Aug 2, 2026
#6 — static PNG export (the feature gap the export-size benchmark surfaced): Figure.to_png() renders the standalone HTML in headless Chromium and screenshots it, so the raster matches the live WebGL chart. Dependency-free — shells out to the Chromium binary (env var / PATH / Playwright-cache discovery), no kaleido-class native package; HTML export still needs nothing. Fluid "100%" sizes fall back to an explicit export size. Missing browser raises a fix-naming error. Verified: html_to_png mechanism produces a 640x400 non-blank PNG from a hand-built spec (no numpy); scripts/png_export_smoke.py gates it in CI (stdlib only) and test_figure.py covers the full Figure.to_png path (pointed at Playwright's Chromium so it runs, not skips). #5 — benchmarks/bench_install.py: best-of-N fresh-interpreter cold import + distribution on-disk size vs the competitor set (methodology §1 rows). stdlib only; unavailable libs reported, never dropped. Size is labeled a lower bound (excludes transitive deps — fresh-venv total stays the runbook gold standard). Wired into the CI benchmark job; plotly-resampler added to the competitor install set. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HS5WXAwy3xr5YUH3RH2wzK
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Moves two Python hot paths into the Rust core, per the offload review of the remaining compute in the payload/LOD/export paths.
Stratified sampling (
fc_stratified_sample_mask, ABI v9 → v10)lod.stratified_sample_keep_mask— the category-aware mask behind categorical density overlays — previously ran a per-category NumPy loop whoseinverse == grouprescans made it O(n · categories), on every categorical density build/zoom. It is now one fused native pass: per-category counts,sqrt-scaled hash thresholds, and the lowest-hashmin_per_categoryfloor (parallelized likesample_mask, with a bounded fallback for degenerate groupings). Small non-negative integer categories — the channel-codes hot path — skipnp.uniqueentirely and serve directly as group codes.Batched scatter marks in the PNG display list (
OP_POINTS)Native PNG export built its display list with a per-point Python loop (
struct.packper mark, plus a per-pointcss_checkre-parse for categorical palettes). Scatter marks now ship as one struct-of-arrays command — NumPy-packed coordinate/radius/fill columns plus a shared symbol/stroke header — and the Rust side loops the existing SDF point painter. Palette entries resolve once into a lookup table.OP_POINT(Rust parity test across symbols and strokes). Non-finite marks are skipped defensively (NaN would poison the framebuffer via NaN-vs-clip comparisons); truncated buffers are rejected like every other opcode.Bar/rect emit loops stay in Python — they scale with categories, not points.
Housekeeping
ABI_VERSIONbumped to 10 in both authoritative locations (src/lib.rs,_native.py).scripts/abi_smoke.pycovers the new entry point (70 checks).examples/demo.ipynbreformatted — it was failing theruff formatgate on main and blockedmake check-full.Verification
make check-fullpasses locally (13 checks: floor, public API, claims, CI workflows, ruff lint+format, ty, pytest 883, js bundle, cargo test 59, clippy-D warnings, release build, ABI smoke). Browser smokes and wheel/sdist verification are left to CI per the production-readiness doc.