diff --git a/CHANGELOG.md b/CHANGELOG.md index ae0734ba..77a1bf6c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,13 @@ in the README). ## [Unreleased] ### Added +- A **committed customization comparison** against Plotly, Vega-Lite, Bokeh, and + Matplotlib (`spec/api/customization-vs-alternatives.md`), with the same + discipline as the benchmark harness: pinned competitor versions, a named + method per row (`schema`, `code`, or `docs`), a claim taxonomy, and a loss + table that `tests/test_customization_comparison.py` refuses to let anyone + empty. XY's own numbers in it are checked against the capability registry + rather than typed in. - **A capability registry** at `python/xy/styling/capabilities.py`: one entry per mark style property and per chrome slot, each carrying its support level in the WebGL client, the SVG writer, and the native rasterizer, plus the diff --git a/spec/README.md b/spec/README.md index a9cc49f5..cb5a6ce0 100644 --- a/spec/README.md +++ b/spec/README.md @@ -30,6 +30,10 @@ The public surface: what callers can build, style, export, and interact with. fails if it is stale. - [`chart-roadmap.md`](api/chart-roadmap.md) — the single 2D-first chart-type coverage backlog, prioritized by popularity and primitive reuse. +- [`customization-vs-alternatives.md`](api/customization-vs-alternatives.md) — + the committed comparison of customization and extensibility against Plotly, + Vega-Lite, Bokeh, and Matplotlib, with its method, its pinned versions, and + the rows XY loses. Pinned by `tests/test_customization_comparison.py`. - [`export.md`](api/export.md) — how a figure becomes bytes: one entry point across five image formats, deterministic engine choice, browser-free default. - [`interaction.md`](api/interaction.md) — the authority on which browser diff --git a/spec/api/customization-vs-alternatives.md b/spec/api/customization-vs-alternatives.md new file mode 100644 index 00000000..6c410bcc --- /dev/null +++ b/spec/api/customization-vs-alternatives.md @@ -0,0 +1,80 @@ +# Customization versus the alternatives + +`spec/benchmarks/methodology.md` exists because performance claims are worthless +without a committed harness. Customization claims have the same problem and had +no equivalent, so this document is that equivalent: the question asked in each +row, the method used to answer it, the version of every library compared, and — +the part that makes it evidence rather than marketing — the rows XY loses. + +A matrix where one library wins everything is evidence of nothing. If a future +edit removes the losses, it has removed the reason to believe the wins. + +## Method + +Same rules as the benchmark harness, adapted to a capability question. + +1. **Versions are pinned and recorded.** The comparison is against the versions + in `benchmarks/requirements-ci.in`: Plotly 6.9.0, Bokeh 3.9.1, Altair 6.2.2 + (Vega-Lite 6), Matplotlib 3.11.1. A row that changes when a competitor + releases is a row that has to be re-checked, not silently inherited. +2. **Every row names its method.** One of: `schema` (counted from a machine + -readable schema — reproducible by running the script named in the row), + `code` (read from the library's source, with the symbol named), or `docs` + (taken from the library's own documentation, which is the right source for + an intended-contract question and the wrong one for a coverage count). +3. **Same-work comparison.** XY's per-slot DOM styling is not compared against + Matplotlib's rcParams as if they answered the same question. Where two + libraries solve a problem in incomparable ways the row says so instead of + picking the framing that flatters XY. +4. **The XY column is generated, not asserted.** Everything in the XY column + that is countable comes from `xy.styling.capabilities`, which + `tests/test_capability_registry.py` pins to the actual implementation, and + `tests/test_customization_comparison.py` re-checks the counts quoted below + against it. A number here that the registry does not know about fails the + suite rather than sitting in prose. +5. **Losses ship.** The loss table below is not an appendix. It is the reason + the win table is credible, and it is maintained with the same care. + +## What XY is genuinely better at + +| Question | XY | Plotly | Vega-Lite | Bokeh | Matplotlib | Method | +| --- | --- | --- | --- | --- | --- | --- | +| Can host design tokens (`var(--brand)`) paint marks? | yes, in every renderer — resolved in Python for static export | no | no | no | no | code | +| Stable, documented DOM slot contract for chrome | 23 named slots, `data-xy-slot` | no published contract | no | partial (CSS classes, undocumented as contract) | n/a — no DOM | code | +| Tailwind classes on chart chrome | yes, per slot | no | no | partial | n/a | docs | +| Same style declaration honored by GPU, vector, and raster output | yes, by construction: one validated subset compiled once | no — the browser and Kaleido share a renderer, but there is no native vector/raster path to agree with | no | no | n/a — one renderer family | code | +| Unsupported style declaration fails loudly | yes, before ingest | no — unknown keys are dropped | schema-validated | partial | partial | code | +| Per-property record of which renderer draws what | yes, `xy.styling.capabilities`, generated and drift-tested | no | no | no | no | code | + +## What XY is genuinely worse at + +| Question | XY | Best alternative | Method | +| --- | --- | --- | --- | +| Total styleable attribute surface | 10 shipped mark style properties, 16 axis keys, 23 chrome slots | **Plotly**: 9,472 non-`src` leaf attributes across 49 trace types and `layout` (plotly 6.9.0) | schema | +| Chart families you can style at all | 20 mark kinds | **Plotly**: 49 trace types, including 3-D, geo, and financial families XY does not implement | schema | +| Writing a genuinely new mark | `xy.register_mark` composes existing primitives only; no custom shader | **Matplotlib**: a custom `Artist` can draw anything the backend can | docs | +| Custom rendering primitives | none — deferred from §24 v0 | **Bokeh**: custom models ship their own TypeScript | docs | +| Global style defaults as a first-class file format | theme object only | **Matplotlib**: `matplotlibrc` plus ~300 rcParams | docs | +| Per-slot styles and classes in static export | dropped — browser only | **Matplotlib**: no split to have, every style path is the export path | code | +| Colorbar styling in static export | no native channel at all | **Plotly**: colorbar attributes apply in Kaleido export | code | +| Author stylesheet in native PNG | rejected; needs `Engine.chromium` | **Plotly**: n/a — Kaleido is always a browser | code | + +## Copyable claim taxonomy + +The performance version of this table is at `spec/benchmarks/results.md` +§ Copyable claim taxonomy, and the rule is the same: a claim is publishable only +if it names the dimension it is true on. + +| Claim shape | Safe wording | Required context | +| --- | --- | --- | +| Token themeability | "XY resolves host CSS variables into mark paint in the browser, SVG, and native PNG; Plotly, Bokeh, Vega-Lite, and Matplotlib do not." | the mechanism, the renderers, the named alternatives | +| Slot contract | "XY publishes 23 stable chrome slots as a supported contract; the alternatives compared expose no equivalent published contract." | slot count, "published contract" not "styleable" | +| Cross-renderer fidelity | "A mark style declaration XY accepts is drawn by all three renderers or rejected at build time." | the subset is validated; this is not a claim about arbitrary CSS | +| Breadth | **Do not claim breadth.** Plotly's attribute surface is roughly three orders of magnitude larger. | — | +| Extensibility | "XY marks can be extended by composing built-in primitives without forking; it does not offer custom shaders or a custom-artist API." | what the plugin can and cannot do | +| Cap/join fidelity | "`stroke-linecap` is drawn identically by all three renderers, verified per renderer; `stroke-linejoin` is not offered because the WebGL client has no join geometry." | the specific property, the specific blocker | + +Claims that are never defensible, regardless of context: "most customizable", +"most themeable charting library", "more extensible than Matplotlib", "as +customizable as Plotly". `scripts/check_claim_guardrails.py` rejects the first +two shapes mechanically; the other two are judgment and belong in review. diff --git a/tests/test_customization_comparison.py b/tests/test_customization_comparison.py new file mode 100644 index 00000000..3f030f1a --- /dev/null +++ b/tests/test_customization_comparison.py @@ -0,0 +1,79 @@ +"""The comparison matrix quotes numbers; those numbers have to still be true. + +`spec/benchmarks/methodology.md` bans numbers in prose that no artifact backs. +The customization comparison is prose by necessity — most of its rows are +capability questions, not measurements — so the parts that *are* countable are +checked here against the registry they came from. The rest is held to a +different standard: it must name its method, and it must keep its losses. +""" + +from __future__ import annotations + +import re +from pathlib import Path + +from xy.styling import capabilities as caps + +DOC = Path(__file__).resolve().parents[1] / "spec" / "api" / "customization-vs-alternatives.md" + + +def _text() -> str: + return DOC.read_text(encoding="utf-8") + + +def test_the_xy_counts_match_the_registry() -> None: + text = _text() + counts = caps.summary() + + assert f"{counts['mark_style_properties_shipped']} shipped mark style properties" in text + assert f"{counts['chart_slots']} chrome slots" in text + # Added after a fresh-agent evaluation caught this line quoting 15 axis + # keys once a 16th had shipped. Every count the document states is pinned; + # an unpinned one is the drift this whole chain exists to prevent. + assert f"{counts['axis_style_keys']} axis keys" in text + + +def test_every_row_names_its_method() -> None: + # A row without a method is an assertion, and this document exists to not be + # a pile of assertions. Both tables end in a method column. + rows = [ + line + for line in _text().splitlines() + if line.startswith("| ") and "---" not in line and not line.startswith("| Question") + ] + matrix_rows = [r for r in rows if r.rstrip().endswith(("| schema |", "| code |", "| docs |"))] + assert len(matrix_rows) >= 12, "expected the win and loss tables to carry method columns" + + +def test_the_loss_table_is_not_empty_and_names_who_wins() -> None: + # The single most important property of this document. A matrix where XY + # wins every row is evidence of nothing, so an edit that empties the loss + # table has to fail rather than read as an improvement. + section = _text().split("## What XY is genuinely worse at", 1) + assert len(section) == 2, "the loss section must exist" + body = section[1].split("## ", 1)[0] + rows = [line for line in body.splitlines() if line.startswith("| ") and "---" not in line] + data_rows = [r for r in rows if not r.startswith("| Question")] + assert len(data_rows) >= 6, "the loss table must keep its rows" + assert all("**" in row for row in data_rows), "each loss must name the library that wins" + + +def test_pinned_competitor_versions_match_the_benchmark_constraints() -> None: + # Same rule as the benchmark harness: a comparison against an unpinned + # version is a comparison against nothing in particular. + text = _text() + pins = Path(__file__).resolve().parents[1] / "benchmarks" / "requirements-ci.in" + pinned = dict( + re.findall(r"^([a-z0-9-]+)==([0-9.]+)", pins.read_text(encoding="utf-8"), re.MULTILINE) + ) + for package, label in (("plotly", "Plotly"), ("bokeh", "Bokeh"), ("matplotlib", "Matplotlib")): + assert f"{label} {pinned[package]}" in text, ( + f"{label} is pinned at {pinned[package]} but the comparison quotes another version" + ) + + +def test_breadth_is_named_as_a_loss_not_a_win() -> None: + # The one claim shape most likely to creep in and least defensible. + text = _text() + assert "**Do not claim breadth.**" in text + assert "most customizable" in text.split("never defensible", 1)[1]