Ship streaming append once, split, per tick (protocol v5)

[Alek99]

Closes #162.

What

Streaming append previously rebuilt the whole packed payload (paying the b"".join copy that split mode was invented to avoid) and transmitted it twice per tick on the widget host: once as the spec/buffers trait re-sync (two separate comm messages, no hold_sync) and again as a custom append message — three comm messages per tick carrying the payload twice.

Now:

• append_data emits the split layout — the same per-column borrowed memoryviews as first paint, no join copy — and stamps the spec with append: {seq, affected} (monotonic, kernel-side counter on Figure).
• FigureWidget.append delivers the refresh as one hold_sync trait update that doubles as notebook-reopen state. No custom send. The client applies an append when spec.append.seq advances, and wantsViewChange reads the live spec so the transport flag survives appends.
• Hosts may surface the two trait writes non-atomically, so the client listens to both change events, defers a torn pair (detected by payloadCoherent — every column must fit its buffer) without consuming the seq, and keys applied state on (seq, buffers identity) so the completing write repairs even a same-shape tear.
• The /_xy socket push keeps its msg envelope, now carrying split buffers; _applyAppend reads the layout from spec.buffer_layout (payloadBuffers moved to 00_header.ts and shared), and a legacy packed reopen blob still applies by declaration.
• PROTOCOL_VERSION/PROTOCOL bump to 5: a cached pre-v5 bundle would otherwise wait forever for a custom append message that no longer exists — the handshake now fails loudly at first paint instead.

Measured

Per append tick, old (main @ b0f8780) vs this branch, same venv/native core, median of 100 ticks (50 for the dashboard), data-level wire accounting that mirrors each host's message sequence exactly:

Case	Wire KiB/tick	Comm messages	Kernel ms/tick
direct scatter, 100k pts, 1 trace	1566 → 783 (−50%)	3 → 1	0.22 → 0.18 (−18%)
decimated line, 2M pts, 1 trace	30 → 15 (−50%)	3 → 1	0.48 → 0.48
10×50k dashboard, 1 trace appended	7823 → 3912 (−50%)	3 → 1	0.55 → 0.50

Wire bytes exactly halve on every tier (the payload crosses once instead of twice); the kernel-side win on the direct case is the eliminated packed join copy. The dashboard row also shows what this PR deliberately does not fix: unaffected traces still re-encode and re-ship (~3.9 MB where ~0.4 MB is live) — that is #163's per-column content hashing, kept out of scope.

Spec

wire-protocol.md §4 (append contract, per-host delivery, torn-update handling), §5 (packed/split usage), §7 (v5 note); rust-engine.md §5 and reflex-integration.md streaming bullet updated to match. CHANGELOG entry under Unreleased → Changed.

Verification

• Full suite: 2192 passed, 8 skipped. ruff check / ruff format --check / pre-commit / ty check (25 pre-existing diagnostics, none new) / abi_smoke (121 checks).
• Browser probe (built ESM bundle against a fake anywidget model, real kernel payloads): append applies exactly once from split buffers (n_points 10→12, GPU vertex count 12); same-seq echoes are no-ops; torn spec-first updates defer without consuming the seq and apply when buffers land; buffers-first updates converge on the spec write; a new buffers identity at the same seq re-applies (same-shape tear repair); both listeners detach on destroy. All green.
• node js/build.mjs regenerated both committed bundles.

🤖 Generated with Claude Code

[codspeed-hq]

Merging this PR will not alter performance

✅ 97 untouched benchmarks
⏩ 1 skipped benchmark¹

---

_{Comparing fix/append-single-send (6fb41fd) with main (b0f8780)}

Footnotes

1. 1 benchmark was skipped, so the baseline result was used instead. If it was deleted from the codebase, click here and archive it to remove it from the performance reports. ↩

[greptile-apps]

Greptile Summary

This PR updates streaming append delivery to protocol v5. The main changes are:

• Sends split per-column buffers once per append tick.
• Uses synchronized widget traits instead of a duplicate custom message.
• Reconciles non-atomic spec and buffer change events.
• Supports split appends and legacy packed reopen state.
• Updates tests, generated bundles, and protocol documentation.

Confidence Score: 5/5

This looks safe to merge.

The updated listener handles both spec-first and buffer-first trait events. Coherence failures defer application without consuming the append sequence. No blocking issues were found in the changed code.

No additional files need attention.

T-Rex Logs

What T-Rex did

• Browser probe exited 0 with every observable check true: n_points=12, gpuVertices=12, one append application, duplicate ignored, listener count 1→0, and no post-destroy application.
• Python transport reported protocol 5, split-buffer count 2, append sequence 1, 10→12 points, and custom-send count 0.
• Screenshot pixel validation confirmed both 900×600 captures are nonblank, richly rendered images; the focused repository widget transport test also passed.
• Artifacts from the run (videos, images, logs, and source assets) were uploaded and linked to this proof for reviewer inspection.

_{Ran code and verified through T-Rex}

Important Files Changed

Filename	Overview
js/src/60_entries.ts	Adds spec-and-buffer event reconciliation, coherence checks, append identity tracking, and listener cleanup.
js/src/00_header.ts	Adds shared packed/split payload handling and structural coherence checks.
js/src/54_kernel.ts	Updates append application to handle split buffers and legacy packed payloads.
python/xy/interaction.py	Builds split append payloads and adds the append sequence and affected traces to the spec.
python/xy/widget.py	Uses one synchronized trait update as the widget append transport.

_{Reviews (2): Last reviewed commit: "Tolerate non-atomic spec/buffers trait w..." | Re-trigger Greptile}