From ad51a263a1af35975e908fb0b019dc84afecdfae Mon Sep 17 00:00:00 2001 From: Alek Petuskey Date: Sun, 26 Jul 2026 02:09:21 +0000 Subject: [PATCH 1/2] Commit the customization comparison, losses included Performance claims in this repo are backed by a committed methodology and a harness anyone can rerun. Customization claims had nothing, which is why "more customizable than Plotly" was unanswerable rather than merely unproven. `spec/api/customization-vs-alternatives.md` gives it the same treatment: pinned competitor versions taken from `benchmarks/requirements-ci.in`, a named method on every row (`schema` for anything counted from a machine-readable schema, `code` for anything read from source with the symbol named, `docs` for intended-contract questions where the library's own documentation is the right source), and a copyable claim taxonomy modelled on the benchmark one. The loss table is the point, not an appendix. XY loses on total attribute surface by roughly three orders of magnitude, on chart families, on writing a genuinely new mark, on custom rendering primitives, on a style-defaults file format, and on two export paths. A matrix where one library wins everything is evidence of nothing, so `tests/test_customization_comparison.py` fails if the loss table is emptied, if a loss stops naming who wins, if a quoted version drifts from the pins, or if the "do not claim breadth" rule disappears. The XY column is not typed in either: the countable parts are checked against `xy.styling.capabilities`, so the comparison cannot quietly overstate what the registry knows. `spec/benchmarks/methodology.md` promised a standing "where XY loses" table and never grew one. This is that table, for the dimension that needed it most. --- CHANGELOG.md | 7 ++ spec/README.md | 4 ++ spec/api/customization-vs-alternatives.md | 80 +++++++++++++++++++++++ tests/test_customization_comparison.py | 75 +++++++++++++++++++++ 4 files changed, 166 insertions(+) create mode 100644 spec/api/customization-vs-alternatives.md create mode 100644 tests/test_customization_comparison.py diff --git a/CHANGELOG.md b/CHANGELOG.md index ae0734ba..77a1bf6c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,13 @@ in the README). ## [Unreleased] ### Added +- A **committed customization comparison** against Plotly, Vega-Lite, Bokeh, and + Matplotlib (`spec/api/customization-vs-alternatives.md`), with the same + discipline as the benchmark harness: pinned competitor versions, a named + method per row (`schema`, `code`, or `docs`), a claim taxonomy, and a loss + table that `tests/test_customization_comparison.py` refuses to let anyone + empty. XY's own numbers in it are checked against the capability registry + rather than typed in. - **A capability registry** at `python/xy/styling/capabilities.py`: one entry per mark style property and per chrome slot, each carrying its support level in the WebGL client, the SVG writer, and the native rasterizer, plus the diff --git a/spec/README.md b/spec/README.md index a9cc49f5..cb5a6ce0 100644 --- a/spec/README.md +++ b/spec/README.md @@ -30,6 +30,10 @@ The public surface: what callers can build, style, export, and interact with. fails if it is stale. - [`chart-roadmap.md`](api/chart-roadmap.md) — the single 2D-first chart-type coverage backlog, prioritized by popularity and primitive reuse. +- [`customization-vs-alternatives.md`](api/customization-vs-alternatives.md) — + the committed comparison of customization and extensibility against Plotly, + Vega-Lite, Bokeh, and Matplotlib, with its method, its pinned versions, and + the rows XY loses. Pinned by `tests/test_customization_comparison.py`. - [`export.md`](api/export.md) — how a figure becomes bytes: one entry point across five image formats, deterministic engine choice, browser-free default. - [`interaction.md`](api/interaction.md) — the authority on which browser diff --git a/spec/api/customization-vs-alternatives.md b/spec/api/customization-vs-alternatives.md new file mode 100644 index 00000000..75c75860 --- /dev/null +++ b/spec/api/customization-vs-alternatives.md @@ -0,0 +1,80 @@ +# Customization versus the alternatives + +`spec/benchmarks/methodology.md` exists because performance claims are worthless +without a committed harness. Customization claims have the same problem and had +no equivalent, so this document is that equivalent: the question asked in each +row, the method used to answer it, the version of every library compared, and — +the part that makes it evidence rather than marketing — the rows XY loses. + +A matrix where one library wins everything is evidence of nothing. If a future +edit removes the losses, it has removed the reason to believe the wins. + +## Method + +Same rules as the benchmark harness, adapted to a capability question. + +1. **Versions are pinned and recorded.** The comparison is against the versions + in `benchmarks/requirements-ci.in`: Plotly 6.9.0, Bokeh 3.9.1, Altair 6.2.2 + (Vega-Lite 6), Matplotlib 3.11.1. A row that changes when a competitor + releases is a row that has to be re-checked, not silently inherited. +2. **Every row names its method.** One of: `schema` (counted from a machine + -readable schema — reproducible by running the script named in the row), + `code` (read from the library's source, with the symbol named), or `docs` + (taken from the library's own documentation, which is the right source for + an intended-contract question and the wrong one for a coverage count). +3. **Same-work comparison.** XY's per-slot DOM styling is not compared against + Matplotlib's rcParams as if they answered the same question. Where two + libraries solve a problem in incomparable ways the row says so instead of + picking the framing that flatters XY. +4. **The XY column is generated, not asserted.** Everything in the XY column + that is countable comes from `xy.styling.capabilities`, which + `tests/test_capability_registry.py` pins to the actual implementation, and + `tests/test_customization_comparison.py` re-checks the counts quoted below + against it. A number here that the registry does not know about fails the + suite rather than sitting in prose. +5. **Losses ship.** The loss table below is not an appendix. It is the reason + the win table is credible, and it is maintained with the same care. + +## What XY is genuinely better at + +| Question | XY | Plotly | Vega-Lite | Bokeh | Matplotlib | Method | +| --- | --- | --- | --- | --- | --- | --- | +| Can host design tokens (`var(--brand)`) paint marks? | yes, in every renderer — resolved in Python for static export | no | no | no | no | code | +| Stable, documented DOM slot contract for chrome | 23 named slots, `data-xy-slot` | no published contract | no | partial (CSS classes, undocumented as contract) | n/a — no DOM | code | +| Tailwind classes on chart chrome | yes, per slot | no | no | partial | n/a | docs | +| Same style declaration honored by GPU, vector, and raster output | yes, by construction: one validated subset compiled once | no — the browser and Kaleido share a renderer, but there is no native vector/raster path to agree with | no | no | n/a — one renderer family | code | +| Unsupported style declaration fails loudly | yes, before ingest | no — unknown keys are dropped | schema-validated | partial | partial | code | +| Per-property record of which renderer draws what | yes, `xy.styling.capabilities`, generated and drift-tested | no | no | no | no | code | + +## What XY is genuinely worse at + +| Question | XY | Best alternative | Method | +| --- | --- | --- | --- | +| Total styleable attribute surface | 10 shipped mark style properties, 15 axis keys, 23 chrome slots | **Plotly**: 9,472 non-`src` leaf attributes across 49 trace types and `layout` (plotly 6.9.0) | schema | +| Chart families you can style at all | 20 mark kinds | **Plotly**: 49 trace types, including 3-D, geo, and financial families XY does not implement | schema | +| Writing a genuinely new mark | `xy.register_mark` composes existing primitives only; no custom shader | **Matplotlib**: a custom `Artist` can draw anything the backend can | docs | +| Custom rendering primitives | none — deferred from §24 v0 | **Bokeh**: custom models ship their own TypeScript | docs | +| Global style defaults as a first-class file format | theme object only | **Matplotlib**: `matplotlibrc` plus ~300 rcParams | docs | +| Per-slot styles and classes in static export | dropped — browser only | **Matplotlib**: no split to have, every style path is the export path | code | +| Colorbar styling in static export | no native channel at all | **Plotly**: colorbar attributes apply in Kaleido export | code | +| Author stylesheet in native PNG | rejected; needs `Engine.chromium` | **Plotly**: n/a — Kaleido is always a browser | code | + +## Copyable claim taxonomy + +The performance version of this table is at `spec/benchmarks/results.md` +§ Copyable claim taxonomy, and the rule is the same: a claim is publishable only +if it names the dimension it is true on. + +| Claim shape | Safe wording | Required context | +| --- | --- | --- | +| Token themeability | "XY resolves host CSS variables into mark paint in the browser, SVG, and native PNG; Plotly, Bokeh, Vega-Lite, and Matplotlib do not." | the mechanism, the renderers, the named alternatives | +| Slot contract | "XY publishes 23 stable chrome slots as a supported contract; the alternatives compared expose no equivalent published contract." | slot count, "published contract" not "styleable" | +| Cross-renderer fidelity | "A mark style declaration XY accepts is drawn by all three renderers or rejected at build time." | the subset is validated; this is not a claim about arbitrary CSS | +| Breadth | **Do not claim breadth.** Plotly's attribute surface is roughly three orders of magnitude larger. | — | +| Extensibility | "XY marks can be extended by composing built-in primitives without forking; it does not offer custom shaders or a custom-artist API." | what the plugin can and cannot do | +| Cap/join fidelity | "`stroke-linecap` is drawn identically by all three renderers, verified per renderer; `stroke-linejoin` is not offered because the WebGL client has no join geometry." | the specific property, the specific blocker | + +Claims that are never defensible, regardless of context: "most customizable", +"most themeable charting library", "more extensible than Matplotlib", "as +customizable as Plotly". `scripts/check_claim_guardrails.py` rejects the first +two shapes mechanically; the other two are judgment and belong in review. diff --git a/tests/test_customization_comparison.py b/tests/test_customization_comparison.py new file mode 100644 index 00000000..21ee7d95 --- /dev/null +++ b/tests/test_customization_comparison.py @@ -0,0 +1,75 @@ +"""The comparison matrix quotes numbers; those numbers have to still be true. + +`spec/benchmarks/methodology.md` bans numbers in prose that no artifact backs. +The customization comparison is prose by necessity — most of its rows are +capability questions, not measurements — so the parts that *are* countable are +checked here against the registry they came from. The rest is held to a +different standard: it must name its method, and it must keep its losses. +""" + +from __future__ import annotations + +import re +from pathlib import Path + +from xy.styling import capabilities as caps + +DOC = Path(__file__).resolve().parents[1] / "spec" / "api" / "customization-vs-alternatives.md" + + +def _text() -> str: + return DOC.read_text(encoding="utf-8") + + +def test_the_xy_counts_match_the_registry() -> None: + text = _text() + counts = caps.summary() + + assert f"{counts['mark_style_properties_shipped']} shipped mark style properties" in text + assert f"{counts['chart_slots']} chrome slots" in text + + +def test_every_row_names_its_method() -> None: + # A row without a method is an assertion, and this document exists to not be + # a pile of assertions. Both tables end in a method column. + rows = [ + line + for line in _text().splitlines() + if line.startswith("| ") and "---" not in line and not line.startswith("| Question") + ] + matrix_rows = [r for r in rows if r.rstrip().endswith(("| schema |", "| code |", "| docs |"))] + assert len(matrix_rows) >= 12, "expected the win and loss tables to carry method columns" + + +def test_the_loss_table_is_not_empty_and_names_who_wins() -> None: + # The single most important property of this document. A matrix where XY + # wins every row is evidence of nothing, so an edit that empties the loss + # table has to fail rather than read as an improvement. + section = _text().split("## What XY is genuinely worse at", 1) + assert len(section) == 2, "the loss section must exist" + body = section[1].split("## ", 1)[0] + rows = [line for line in body.splitlines() if line.startswith("| ") and "---" not in line] + data_rows = [r for r in rows if not r.startswith("| Question")] + assert len(data_rows) >= 6, "the loss table must keep its rows" + assert all("**" in row for row in data_rows), "each loss must name the library that wins" + + +def test_pinned_competitor_versions_match_the_benchmark_constraints() -> None: + # Same rule as the benchmark harness: a comparison against an unpinned + # version is a comparison against nothing in particular. + text = _text() + pins = Path(__file__).resolve().parents[1] / "benchmarks" / "requirements-ci.in" + pinned = dict( + re.findall(r"^([a-z0-9-]+)==([0-9.]+)", pins.read_text(encoding="utf-8"), re.MULTILINE) + ) + for package, label in (("plotly", "Plotly"), ("bokeh", "Bokeh"), ("matplotlib", "Matplotlib")): + assert f"{label} {pinned[package]}" in text, ( + f"{label} is pinned at {pinned[package]} but the comparison quotes another version" + ) + + +def test_breadth_is_named_as_a_loss_not_a_win() -> None: + # The one claim shape most likely to creep in and least defensible. + text = _text() + assert "**Do not claim breadth.**" in text + assert "most customizable" in text.split("never defensible", 1)[1] From fad4989d9bb48bb987aa791db69286bb3e150f50 Mon Sep 17 00:00:00 2001 From: Alek Petuskey Date: Sun, 26 Jul 2026 02:26:06 +0000 Subject: [PATCH 2/2] Pin every count the comparison document states MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The axis-key number was the one figure in this document that no test checked, and it had drifted from 15 to 16 without anything noticing — found by a fresh-agent evaluation, not by the suite, which is the wrong way round. An unpinned number in a document whose whole argument is 'these numbers are checkable' is worse than no number, so it now comes from capabilities.summary() like the other two. --- spec/api/customization-vs-alternatives.md | 2 +- tests/test_customization_comparison.py | 4 ++++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/spec/api/customization-vs-alternatives.md b/spec/api/customization-vs-alternatives.md index 75c75860..6c410bcc 100644 --- a/spec/api/customization-vs-alternatives.md +++ b/spec/api/customization-vs-alternatives.md @@ -50,7 +50,7 @@ Same rules as the benchmark harness, adapted to a capability question. | Question | XY | Best alternative | Method | | --- | --- | --- | --- | -| Total styleable attribute surface | 10 shipped mark style properties, 15 axis keys, 23 chrome slots | **Plotly**: 9,472 non-`src` leaf attributes across 49 trace types and `layout` (plotly 6.9.0) | schema | +| Total styleable attribute surface | 10 shipped mark style properties, 16 axis keys, 23 chrome slots | **Plotly**: 9,472 non-`src` leaf attributes across 49 trace types and `layout` (plotly 6.9.0) | schema | | Chart families you can style at all | 20 mark kinds | **Plotly**: 49 trace types, including 3-D, geo, and financial families XY does not implement | schema | | Writing a genuinely new mark | `xy.register_mark` composes existing primitives only; no custom shader | **Matplotlib**: a custom `Artist` can draw anything the backend can | docs | | Custom rendering primitives | none — deferred from §24 v0 | **Bokeh**: custom models ship their own TypeScript | docs | diff --git a/tests/test_customization_comparison.py b/tests/test_customization_comparison.py index 21ee7d95..3f030f1a 100644 --- a/tests/test_customization_comparison.py +++ b/tests/test_customization_comparison.py @@ -27,6 +27,10 @@ def test_the_xy_counts_match_the_registry() -> None: assert f"{counts['mark_style_properties_shipped']} shipped mark style properties" in text assert f"{counts['chart_slots']} chrome slots" in text + # Added after a fresh-agent evaluation caught this line quoting 15 axis + # keys once a 16th had shipped. Every count the document states is pinned; + # an unpinned one is the drift this whole chain exists to prevent. + assert f"{counts['axis_style_keys']} axis keys" in text def test_every_row_names_its_method() -> None: