Skip to content

Stratified sampling and batched raster points in the native core (ABI v10) - #6

Merged
Alek99 merged 1 commit into
mainfrom
native-offload-sampling-raster
Jul 9, 2026
Merged

Stratified sampling and batched raster points in the native core (ABI v10)#6
Alek99 merged 1 commit into
mainfrom
native-offload-sampling-raster

Conversation

@Alek99

@Alek99 Alek99 commented Jul 9, 2026

Copy link
Copy Markdown
Member

Moves two Python hot paths into the Rust core, per the offload review of the remaining compute in the payload/LOD/export paths.

Stratified sampling (fc_stratified_sample_mask, ABI v9 → v10)

lod.stratified_sample_keep_mask — the category-aware mask behind categorical density overlays — previously ran a per-category NumPy loop whose inverse == group rescans made it O(n · categories), on every categorical density build/zoom. It is now one fused native pass: per-category counts, sqrt-scaled hash thresholds, and the lowest-hash min_per_category floor (parallelized like sample_mask, with a bounded fallback for degenerate groupings). Small non-negative integer categories — the channel-codes hot path — skip np.unique entirely and serve directly as group codes.

  • Bit-identical masks: parity-tested against the NumPy reference on both sides of the ABI (Rust test vs a ported oracle; Python test vs the old loop). Exactness holds because SplitMix64 is a bijection, so distinct ids cannot tie; duplicate-id ties break deterministically toward the lower row index.
  • Measured: 168 ms → 8.5 ms (~20×) at 5M rows / 12 categories (Apple Silicon dev box).

Batched scatter marks in the PNG display list (OP_POINTS)

Native PNG export built its display list with a per-point Python loop (struct.pack per mark, plus a per-point css_check re-parse for categorical palettes). Scatter marks now ship as one struct-of-arrays command — NumPy-packed coordinate/radius/fill columns plus a shared symbol/stroke header — and the Rust side loops the existing SDF point painter. Palette entries resolve once into a lookup table.

  • Pixel-identical to per-mark OP_POINT (Rust parity test across symbols and strokes). Non-finite marks are skipped defensively (NaN would poison the framebuffer via NaN-vs-clip comparisons); truncated buffers are rejected like every other opcode.
  • Measured: display-list build for a 100k-point categorical scatter drops ~186 ms → ~1 ms; command buffer shrinks ~40% (2.6 MB → 1.6 MB).

Bar/rect emit loops stay in Python — they scale with categories, not points.

Housekeeping

  • ABI_VERSION bumped to 10 in both authoritative locations (src/lib.rs, _native.py).
  • scripts/abi_smoke.py covers the new entry point (70 checks).
  • examples/demo.ipynb reformatted — it was failing the ruff format gate on main and blocked make check-full.
  • CHANGELOG entries under Unreleased.

Verification

make check-full passes locally (13 checks: floor, public API, claims, CI workflows, ruff lint+format, ty, pytest 883, js bundle, cargo test 59, clippy -D warnings, release build, ABI smoke). Browser smokes and wheel/sdist verification are left to CI per the production-readiness doc.

… v10)

Two Python hot paths move into the Rust core:

- fc_stratified_sample_mask fuses the per-category sampling loop behind
  categorical density overlays (sqrt-scaled hash thresholds plus the
  lowest-hash min_per_category floor) into one native pass, replacing
  O(n * categories) NumPy rescans. Small non-negative integer categories
  skip np.unique and serve directly as group codes. Bit-identical masks,
  parity-tested on both sides of the ABI; ~20x faster at 5M rows / 12
  categories (168 ms -> 8.5 ms).

- OP_POINTS batches scatter marks in the PNG display list as one
  struct-of-arrays command, replacing the per-point struct.pack loop and
  per-point CSS color re-parse for categorical palettes. Pixel-identical
  to per-mark OP_POINT (parity-tested); display-list build for a
  100k-point categorical scatter drops ~186 ms -> ~1 ms.

Also reformats examples/demo.ipynb, which was failing the ruff format
gate on main and blocking make check-full.
@Alek99
Alek99 merged commit 0a60d76 into main Jul 9, 2026
4 of 7 checks passed
Alek99 pushed a commit that referenced this pull request Aug 2, 2026
#6 — static PNG export (the feature gap the export-size benchmark
surfaced): Figure.to_png() renders the standalone HTML in headless
Chromium and screenshots it, so the raster matches the live WebGL
chart. Dependency-free — shells out to the Chromium binary (env var /
PATH / Playwright-cache discovery), no kaleido-class native package;
HTML export still needs nothing. Fluid "100%" sizes fall back to an
explicit export size. Missing browser raises a fix-naming error.
Verified: html_to_png mechanism produces a 640x400 non-blank PNG from a
hand-built spec (no numpy); scripts/png_export_smoke.py gates it in CI
(stdlib only) and test_figure.py covers the full Figure.to_png path
(pointed at Playwright's Chromium so it runs, not skips).

#5 — benchmarks/bench_install.py: best-of-N fresh-interpreter cold
import + distribution on-disk size vs the competitor set (methodology
§1 rows). stdlib only; unavailable libs reported, never dropped. Size
is labeled a lower bound (excludes transitive deps — fresh-venv total
stays the runbook gold standard). Wired into the CI benchmark job;
plotly-resampler added to the competitor install set.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HS5WXAwy3xr5YUH3RH2wzK
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant