5/7 Commit the customization comparison, losses included

CHANGELOG.md

@@ -9,6 +9,13 @@ in the README).
 ## [Unreleased]
 
 ### Added
+- A **committed customization comparison** against Plotly, Vega-Lite, Bokeh, and
+  Matplotlib (`spec/api/customization-vs-alternatives.md`), with the same
+  discipline as the benchmark harness: pinned competitor versions, a named
+  method per row (`schema`, `code`, or `docs`), a claim taxonomy, and a loss
+  table that `tests/test_customization_comparison.py` refuses to let anyone
+  empty. XY's own numbers in it are checked against the capability registry
+  rather than typed in.
 - **A capability registry** at `python/xy/styling/capabilities.py`: one entry per
   mark style property and per chrome slot, each carrying its support level in
   the WebGL client, the SVG writer, and the native rasterizer, plus the

spec/README.md

@@ -30,6 +30,10 @@ The public surface: what callers can build, style, export, and interact with.
   fails if it is stale.
 - [`chart-roadmap.md`](api/chart-roadmap.md) — the single 2D-first chart-type
   coverage backlog, prioritized by popularity and primitive reuse.
+- [`customization-vs-alternatives.md`](api/customization-vs-alternatives.md) —
+  the committed comparison of customization and extensibility against Plotly,
+  Vega-Lite, Bokeh, and Matplotlib, with its method, its pinned versions, and
+  the rows XY loses. Pinned by `tests/test_customization_comparison.py`.
 - [`export.md`](api/export.md) — how a figure becomes bytes: one entry point
   across five image formats, deterministic engine choice, browser-free default.
 - [`interaction.md`](api/interaction.md) — the authority on which browser

spec/api/customization-vs-alternatives.md

@@ -0,0 +1,80 @@
+# Customization versus the alternatives
+
+`spec/benchmarks/methodology.md` exists because performance claims are worthless
+without a committed harness. Customization claims have the same problem and had
+no equivalent, so this document is that equivalent: the question asked in each
+row, the method used to answer it, the version of every library compared, and —
+the part that makes it evidence rather than marketing — the rows XY loses.
+
+A matrix where one library wins everything is evidence of nothing. If a future
+edit removes the losses, it has removed the reason to believe the wins.
+
+## Method
+
+Same rules as the benchmark harness, adapted to a capability question.
+
+1. **Versions are pinned and recorded.** The comparison is against the versions
+   in `benchmarks/requirements-ci.in`: Plotly 6.9.0, Bokeh 3.9.1, Altair 6.2.2
+   (Vega-Lite 6), Matplotlib 3.11.1. A row that changes when a competitor
+   releases is a row that has to be re-checked, not silently inherited.
+2. **Every row names its method.** One of: `schema` (counted from a machine
+   -readable schema — reproducible by running the script named in the row),
+   `code` (read from the library's source, with the symbol named), or `docs`
+   (taken from the library's own documentation, which is the right source for
+   an intended-contract question and the wrong one for a coverage count).
+3. **Same-work comparison.** XY's per-slot DOM styling is not compared against
+   Matplotlib's rcParams as if they answered the same question. Where two
+   libraries solve a problem in incomparable ways the row says so instead of
+   picking the framing that flatters XY.
+4. **The XY column is generated, not asserted.** Everything in the XY column
+   that is countable comes from `xy.styling.capabilities`, which
+   `tests/test_capability_registry.py` pins to the actual implementation, and
+   `tests/test_customization_comparison.py` re-checks the counts quoted below
+   against it. A number here that the registry does not know about fails the
+   suite rather than sitting in prose.
+5. **Losses ship.** The loss table below is not an appendix. It is the reason
+   the win table is credible, and it is maintained with the same care.
+
+## What XY is genuinely better at
+
+| Question | XY | Plotly | Vega-Lite | Bokeh | Matplotlib | Method |
+| --- | --- | --- | --- | --- | --- | --- |
+| Can host design tokens (`var(--brand)`) paint marks? | yes, in every renderer — resolved in Python for static export | no | no | no | no | code |
+| Stable, documented DOM slot contract for chrome | 23 named slots, `data-xy-slot` | no published contract | no | partial (CSS classes, undocumented as contract) | n/a — no DOM | code |
+| Tailwind classes on chart chrome | yes, per slot | no | no | partial | n/a | docs |
+| Same style declaration honored by GPU, vector, and raster output | yes, by construction: one validated subset compiled once | no — the browser and Kaleido share a renderer, but there is no native vector/raster path to agree with | no | no | n/a — one renderer family | code |
+| Unsupported style declaration fails loudly | yes, before ingest | no — unknown keys are dropped | schema-validated | partial | partial | code |
+| Per-property record of which renderer draws what | yes, `xy.styling.capabilities`, generated and drift-tested | no | no | no | no | code |
+
+## What XY is genuinely worse at
+
+| Question | XY | Best alternative | Method |
+| --- | --- | --- | --- |
+| Total styleable attribute surface | 10 shipped mark style properties, 16 axis keys, 23 chrome slots | **Plotly**: 9,472 non-`src` leaf attributes across 49 trace types and `layout` (plotly 6.9.0) | schema |
+| Chart families you can style at all | 20 mark kinds | **Plotly**: 49 trace types, including 3-D, geo, and financial families XY does not implement | schema |
+| Writing a genuinely new mark | `xy.register_mark` composes existing primitives only; no custom shader | **Matplotlib**: a custom `Artist` can draw anything the backend can | docs |
+| Custom rendering primitives | none — deferred from §24 v0 | **Bokeh**: custom models ship their own TypeScript | docs |
+| Global style defaults as a first-class file format | theme object only | **Matplotlib**: `matplotlibrc` plus ~300 rcParams | docs |
+| Per-slot styles and classes in static export | dropped — browser only | **Matplotlib**: no split to have, every style path is the export path | code |
+| Colorbar styling in static export | no native channel at all | **Plotly**: colorbar attributes apply in Kaleido export | code |
+| Author stylesheet in native PNG | rejected; needs `Engine.chromium` | **Plotly**: n/a — Kaleido is always a browser | code |
+
+## Copyable claim taxonomy
+
+The performance version of this table is at `spec/benchmarks/results.md`
+§ Copyable claim taxonomy, and the rule is the same: a claim is publishable only
+if it names the dimension it is true on.
+
+| Claim shape | Safe wording | Required context |
+| --- | --- | --- |
+| Token themeability | "XY resolves host CSS variables into mark paint in the browser, SVG, and native PNG; Plotly, Bokeh, Vega-Lite, and Matplotlib do not." | the mechanism, the renderers, the named alternatives |
+| Slot contract | "XY publishes 23 stable chrome slots as a supported contract; the alternatives compared expose no equivalent published contract." | slot count, "published contract" not "styleable" |
+| Cross-renderer fidelity | "A mark style declaration XY accepts is drawn by all three renderers or rejected at build time." | the subset is validated; this is not a claim about arbitrary CSS |
+| Breadth | **Do not claim breadth.** Plotly's attribute surface is roughly three orders of magnitude larger. | — |
+| Extensibility | "XY marks can be extended by composing built-in primitives without forking; it does not offer custom shaders or a custom-artist API." | what the plugin can and cannot do |
+| Cap/join fidelity | "`stroke-linecap` is drawn identically by all three renderers, verified per renderer; `stroke-linejoin` is not offered because the WebGL client has no join geometry." | the specific property, the specific blocker |
+
+Claims that are never defensible, regardless of context: "most customizable",
+"most themeable charting library", "more extensible than Matplotlib", "as
+customizable as Plotly". `scripts/check_claim_guardrails.py` rejects the first
+two shapes mechanically; the other two are judgment and belong in review.

tests/test_customization_comparison.py

@@ -0,0 +1,79 @@
+"""The comparison matrix quotes numbers; those numbers have to still be true.
+
+`spec/benchmarks/methodology.md` bans numbers in prose that no artifact backs.
+The customization comparison is prose by necessity — most of its rows are
+capability questions, not measurements — so the parts that *are* countable are
+checked here against the registry they came from. The rest is held to a
+different standard: it must name its method, and it must keep its losses.
+"""
+
+from __future__ import annotations
+
+import re
+from pathlib import Path
+
+from xy.styling import capabilities as caps
+
+DOC = Path(__file__).resolve().parents[1] / "spec" / "api" / "customization-vs-alternatives.md"
+
+
+def _text() -> str:
+    return DOC.read_text(encoding="utf-8")
+
+
+def test_the_xy_counts_match_the_registry() -> None:
+    text = _text()
+    counts = caps.summary()
+
+    assert f"{counts['mark_style_properties_shipped']} shipped mark style properties" in text
+    assert f"{counts['chart_slots']} chrome slots" in text
+    # Added after a fresh-agent evaluation caught this line quoting 15 axis
+    # keys once a 16th had shipped. Every count the document states is pinned;
+    # an unpinned one is the drift this whole chain exists to prevent.
+    assert f"{counts['axis_style_keys']} axis keys" in text
+
+
+def test_every_row_names_its_method() -> None:
+    # A row without a method is an assertion, and this document exists to not be
+    # a pile of assertions. Both tables end in a method column.
+    rows = [
+        line
+        for line in _text().splitlines()
+        if line.startswith("| ") and "---" not in line and not line.startswith("| Question")
+    ]
+    matrix_rows = [r for r in rows if r.rstrip().endswith(("| schema |", "| code |", "| docs |"))]
+    assert len(matrix_rows) >= 12, "expected the win and loss tables to carry method columns"
+
+
+def test_the_loss_table_is_not_empty_and_names_who_wins() -> None:
+    # The single most important property of this document. A matrix where XY
+    # wins every row is evidence of nothing, so an edit that empties the loss
+    # table has to fail rather than read as an improvement.
+    section = _text().split("## What XY is genuinely worse at", 1)
+    assert len(section) == 2, "the loss section must exist"
+    body = section[1].split("## ", 1)[0]
+    rows = [line for line in body.splitlines() if line.startswith("| ") and "---" not in line]
+    data_rows = [r for r in rows if not r.startswith("| Question")]
+    assert len(data_rows) >= 6, "the loss table must keep its rows"
+    assert all("**" in row for row in data_rows), "each loss must name the library that wins"
+
+
+def test_pinned_competitor_versions_match_the_benchmark_constraints() -> None:
+    # Same rule as the benchmark harness: a comparison against an unpinned
+    # version is a comparison against nothing in particular.
+    text = _text()
+    pins = Path(__file__).resolve().parents[1] / "benchmarks" / "requirements-ci.in"
+    pinned = dict(
+        re.findall(r"^([a-z0-9-]+)==([0-9.]+)", pins.read_text(encoding="utf-8"), re.MULTILINE)
+    )
+    for package, label in (("plotly", "Plotly"), ("bokeh", "Bokeh"), ("matplotlib", "Matplotlib")):
+        assert f"{label} {pinned[package]}" in text, (
+            f"{label} is pinned at {pinned[package]} but the comparison quotes another version"
+        )
+
+
+def test_breadth_is_named_as_a_loss_not_a_win() -> None:
+    # The one claim shape most likely to creep in and least defensible.
+    text = _text()
+    assert "**Do not claim breadth.**" in text
+    assert "most customizable" in text.split("never defensible", 1)[1]