5/7 Commit the customization comparison, losses included - #306
Conversation
Performance claims in this repo are backed by a committed methodology and a harness anyone can rerun. Customization claims had nothing, which is why "more customizable than Plotly" was unanswerable rather than merely unproven. `spec/api/customization-vs-alternatives.md` gives it the same treatment: pinned competitor versions taken from `benchmarks/requirements-ci.in`, a named method on every row (`schema` for anything counted from a machine-readable schema, `code` for anything read from source with the symbol named, `docs` for intended-contract questions where the library's own documentation is the right source), and a copyable claim taxonomy modelled on the benchmark one. The loss table is the point, not an appendix. XY loses on total attribute surface by roughly three orders of magnitude, on chart families, on writing a genuinely new mark, on custom rendering primitives, on a style-defaults file format, and on two export paths. A matrix where one library wins everything is evidence of nothing, so `tests/test_customization_comparison.py` fails if the loss table is emptied, if a loss stops naming who wins, if a quoted version drifts from the pins, or if the "do not claim breadth" rule disappears. The XY column is not typed in either: the countable parts are checked against `xy.styling.capabilities`, so the comparison cannot quietly overstate what the registry knows. `spec/benchmarks/methodology.md` promised a standing "where XY loses" table and never grew one. This is that table, for the dimension that needed it most.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
…ity-registry' into claude/customization-capability-matrix-vckjxg-5-comparison-matrix
The axis-key number was the one figure in this document that no test checked, and it had drifted from 15 to 16 without anything noticing — found by a fresh-agent evaluation, not by the suite, which is the wrong way round. An unpinned number in a document whose whole argument is 'these numbers are checkable' is worse than no number, so it now comes from capabilities.summary() like the other two.
Merging this PR will not alter performance
Comparing 
Footnotes
|
1 participant
Stacked on #305.
Performance claims here are backed by a committed methodology and a rerunnable harness. Customization claims had nothing, which is why "more customizable than Plotly" was unanswerable rather than merely unproven.
spec/api/customization-vs-alternatives.mdgives it the same treatment.Method
benchmarks/requirements-ci.in: Plotly 6.9.0, Bokeh 3.9.1, Altair 6.2.2 (Vega-Lite 6), Matplotlib 3.11.1.schema(counted from a machine-readable schema, reproducible),code(read from source, symbol named), ordocs(the library's own documentation, which is right for intended-contract questions and wrong for coverage counts).xy.styling.capabilitiesbytests/test_customization_comparison.py.The loss table is the point
xy loses on total attribute surface by roughly three orders of magnitude, on chart families, on writing a genuinely new mark, on custom rendering primitives, on a style-defaults file format, and on two export paths.
A matrix where one library wins everything is evidence of nothing, so the tests fail if the loss table is emptied, if a loss stops naming who wins, if a quoted version drifts from the pins, or if the "do not claim breadth" rule disappears.
Note
spec/benchmarks/methodology.mdpromises a standing "where XY loses" table — "Nothing buys credibility like published losses." — and no such table was ever written. This is that table, for the dimension that needed it most. The benchmark one is still outstanding.uv run pytest2352 passed.Generated by Claude Code