ENG-10167 Expand xy.pyplot shim: plot-family depth, scales, export chrome, and pinned Matplotlib 3.11 evidence

.github/workflows/ci.yml

@@ -14,6 +14,29 @@ concurrency:
   cancel-in-progress: true
 
 jobs:
+  matplotlib_reference:
+    name: Matplotlib 3.11 reference compatibility
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.2.2
+      - uses: astral-sh/setup-uv@d4b2f3b6ecc6e67c4457f6d3e41ec42d3d0fcb86 # v5.4.2
+      - name: Install xy and released reference wheel
+        run: |
+          uv venv .venv
+          uv pip install -p .venv/bin/python -e ".[dev]"
+          uv pip install -p .venv/bin/python "matplotlib==3.11.0"
+      - name: Verify released reference and reviewed snapshot
+        run: |
+          .venv/bin/python -c "import matplotlib; assert matplotlib.__version__ == '3.11.0'"
+          .venv/bin/python scripts/sync_matplotlib_compat.py --check
+      - name: Run optional-interoperability and dual-engine corpus tests
+        env:
+          MPLBACKEND: Agg
+        run: |
+          .venv/bin/pytest -q tests/pyplot/test_launch_compat.py
+          .venv/bin/pytest -q tests/pyplot/test_reference_corpus.py
+          .venv/bin/pytest -q tests/pyplot/test_reference_semantics.py
+
   test:
     name: Test (Rust + Python + JS)
     runs-on: ubuntu-latest

docs/chart-roadmap.md

@@ -21,8 +21,10 @@ entry point, not the boundary of the product.
 
 The current implemented surface is **line**, **scatter**, **area**,
 **histogram**, **bar/column**, **heatmap**, **error bars/bands**,
-**box/violin/ECDF**, **hexbin/contour**, **step/stairs/stem**, and
-**faceted small multiples**. Scatter already covers direct
+**box/violin/ECDF**, **hexbin/contour**, **step/stairs/stem**, **pie/donut**,
+**scientific vector fields**, **irregular triangular meshes**, and **faceted
+small multiples**. The last three families are exposed through the
+Matplotlib-flavoured shim over shared xy primitives. Scatter already covers direct
 points, color/size channels, GPU picking, selection, and Tier-2 density
 aggregation. Line and area cover direct and M4-decimated time series. Histogram
 and bar/column share the instanced rectangle renderer; heatmap ships a compact
@@ -99,7 +101,7 @@ not fall out of sight.
 | 3 | Bar / column | vertical bar, horizontal bar, grouped, stacked, normalized stacked, diverging bar | Implemented core | `fc.bar(...)` / `fc.column(...)` ship categorical/numeric vertical and horizontal bars, grouped bars, stacked bars, and normalized stacked bars (`mode="normalized"`) through the shared rectangle renderer. Follow-up: labels. |
 | 4 | Area | filled line, stacked area, streamgraph, ridgeline-lite area bands | Implemented core | `fc.area(...)` ships a filled area with scalar/array baseline and optional line overlay. Follow-ups: stacked area helpers and streamgraph offsets. |
 | 5 | Histogram | count, probability, density, cumulative histogram | Implemented core | Python-side binning plus the shared rectangle renderer; `cumulative=True` (count CDF and, with `density=True`, empirical CDF) is implemented. Follow-up: viewport-aware re-binning for huge streamed distributions. |
-| 6 | Pie / donut | pie, donut, nested donut, variable-radius pie | Planned compatibility | Extremely common in dashboards even though performance differentiation is low. |
+| 6 | Pie / donut | pie, donut, nested donut, variable-radius pie | Implemented in `xy.pyplot` | Native pie/donut tessellation with Matplotlib-style containers and labels; richer nested/variable-radius composition remains future depth. |
 | 7 | Heatmap / image / matrix | heatmap, image, annotated matrix, correlation matrix, cohort heatmap | Implemented core | `fc.heatmap(...)` renders matrix cells through a compact grid texture with continuous colormaps and categorical/numeric axes. Native static export borrows canonical f64 spans and normalizes only sampled pixels in Rust, verified through 4.29B cells without a derived grid or RGBA expansion. Follow-ups: annotation and tiled huge-image browser transport. |
 | 8 | Box plot | box, grouped box, notched box, outlier points | Implemented core | Tukey quartiles, whiskers, median, deterministic outliers, numeric or categorical groups. |
 | 9 | Candlestick / OHLC | candlestick, OHLC bars, volume overlay, range selector | Prototyped (PR closed unmerged) | `fc.candlestick(...)`/`fc.ohlc(...)` + `fc.candlestick_chart(...)` on the closed `codex/finance-charting-surface` exploration branch: OHLC decimation, shared-y f32 frame, time axes, hover, and a volume pane. Critical finance surface; inherits LOD and time-axis work from core primitives. |
@@ -125,8 +127,8 @@ not fall out of sight.
 | 29 | Parallel coordinate/category | parallel coordinates, parallel categories, alluvial-lite | Planned later | Present in Plotly/ECharts; useful for high-dimensional EDA. |
 | 30 | Sankey / alluvial | Sankey, alluvial, dependency wheel | Planned later | Important flow chart, but requires layout and interaction work. |
 | 31 | Network/tree/org | network graph, force graph, tree, dendrogram, org chart, arc diagram | Planned later | Valuable but layout-heavy; should follow core 2D marks. |
-| 32 | Scientific vector fields | quiver, barbs, streamplot, wind rose | Planned later | Science/engineering breadth; needs arrows, vector fields, and polar variants. |
-| 33 | Irregular grid science | pcolormesh, tricontour, tripcolor, triangular mesh | Planned later | Matplotlib/science compatibility; separate data model from regular heatmaps. |
+| 32 | Scientific vector fields | quiver, barbs, streamplot, wind rose | Implemented in `xy.pyplot` | Quiver, barbs, and bounded streamlines feed shared instanced segments; wind rose remains tied to future polar axes. |
+| 33 | Irregular grid science | pcolormesh, tricontour, tripcolor, triangular mesh | Implemented in `xy.pyplot` | Curvilinear quads and explicit/native triangulations route through indexed meshes and marching-triangle kernels. |
 | 34 | Specialist coordinate systems | ternary, Smith chart, carpet plot, polar scatter/line/bar | Planned later | Plotly/science compatibility; axis systems are the main work. |
 | 35 | Finance advanced | VWAP, moving averages, Bollinger bands, RSI, MACD, depth chart, order book heatmap, market profile, Renko, Heikin-Ashi, Kagi, point-and-figure | Prototyped (PR closed unmerged) | The closed `codex/finance-charting-surface` exploration branch has a `FinanceChart`/`FinanceLayer` system with volume bars, SMA, VWAP, Bollinger bands, RSI, and MACD as overlay/pane layers plus drawings. Remaining: depth/order-book, market profile, Renko/Heikin-Ashi/Kagi/P&F. |
 | 36 | Maps and geo | choropleth, tile choropleth, point map, bubble map, density map, route map, filled-area map | Deferred 2D domain | 2D, but requires projection/tile/geography stack; do after core chart breadth. |
@@ -167,7 +169,7 @@ depth: strip/swarm/boxen/rug distributions, regression diagnostics, richer
 | Rank | Chart | Why it matters | Caveat |
 |---:|---|---|---|
 | 13 | Composed / mixed charts | Overlay line, scatter, bars, bands, candlesticks, and secondary axes cleanly. | API and spec work comes before many chart families. |
-| 14 | Pie / donut | Very popular in basic chart libraries and user expectations. | Low xy differentiation; implement for completeness, not performance. |
+| 14 | Pie / donut | Very popular in basic chart libraries and user expectations. | Implemented through `xy.pyplot`; future work is composition and styling depth. |
 | 15 | Candlestick / OHLC | Important for finance users and appears in Plotly/Highcharts stock tooling. | **Prototyped (PR closed unmerged):** candlestick/OHLC marks with date axes, gaps, and hover format on the closed finance exploration branch. Remaining polish: range selectors. |
 | 16 | Finance overlays | Volume bars, VWAP, moving averages, Bollinger bands, depth/order-book heatmap, market profile, Renko, Heikin-Ashi, Kagi, point-and-figure. | **Prototyped (PR closed unmerged):** volume pane, SMA, VWAP, Bollinger, RSI, MACD as `FinanceLayer`s reusing composed charts + time axes. Remaining: depth/order-book, market profile, Renko/Heikin-Ashi/Kagi/P&F. |
 | 17 | Waterfall | Common in business reporting and Plotly/Highcharts. | Mostly categorical bars plus running baseline. |
@@ -192,8 +194,8 @@ depth: strip/swarm/boxen/rug distributions, regression diagnostics, richer
 |---:|---|---|---|
 | 27 | Radar / polar / radial bar | Common in Chart.js/Highcharts and dashboards. | Needs polar axes and interaction semantics. |
 | 28 | Ternary / Smith / carpet | Plotly/scientific compatibility. | New coordinate systems, not new mark primitives. |
-| 29 | Quiver / barbs / streamplot / wind rose | Scientific and engineering vector fields. | Needs arrows, vector sampling, and polar support. |
-| 30 | Pcolormesh / tricontour / tripcolor | Matplotlib-style irregular grid science. | Separate mesh data model from regular heatmaps. |
+| 29 | Quiver / barbs / streamplot / wind rose | Scientific and engineering vector fields. | Quiver, barbs, and streamplot are implemented through `xy.pyplot`; wind rose awaits polar support. |
+| 30 | Pcolormesh / tricontour / tripcolor | Matplotlib-style irregular grid science. | Implemented through `xy.pyplot` with native quad/triangle geometry. |
 | 31 | Waffle / mosaic / Mekko / variwide | Business/category compatibility. | Mostly rectangle layout algorithms. |
 | 32 | Packed bubble / Venn / Euler | Compatibility and presentation charts. | Layout algorithms and label placement dominate. |
 | 33 | Pictorial bar / item chart / image markers / text marks | ECharts/Highcharts compatibility polish. | Symbol systems and asset handling. |

docs/matplotlib-compat-changelog.md

@@ -0,0 +1,115 @@
+# Matplotlib compatibility changelog
+
+This changelog records changes to the upstream compatibility target and to the
+meaning of xy's compatibility levels. It complements the project changelog,
+which covers user-visible releases across the whole package.
+
+## Matplotlib 3.11 development snapshot — 2026-07-13
+
+- Pinned upstream revision `bde111fb4e`
+  (`v3.11.0-348-gbde111fb4e`, 2026-07-10).
+- Captured the supported 66-method `Axes` Plotting inventory as a generated,
+  reviewed snapshot instead of a hard-coded assertion.
+- Added `grouped_bar`, `pie_label`, and `violin` from the 3.11 development
+  surface.
+- Added a dedicated CI environment for optional Matplotlib-object interop and
+  isolated dual-engine execution of the full compatibility corpus.
+- Published approximation levels and a generated method compatibility matrix.
+
+### Post-review corrections — 2026-07-13
+
+- Downgraded "Unstructured triangles" from *exact geometry* to *equivalent
+  semantics*: the family has no cross-engine reference comparison yet.
+- Converted the formerly silent option discards into loud rejections: pie
+  shadow/frame/rotatelabels/hatch, quiver/barbs/quiverkey head geometry,
+  units, increments and styling, contour origin/linestyles/corner_mask,
+  table placement, tricontour extend, non-linear norm objects everywhere,
+  spy aspect, and pie_label rotate.
+- Implemented (rather than rejected) where the marks could honor the value:
+  contour `extent`; plain `Normalize` reduced to vmin/vmax for pcolormesh and
+  the tri* family; stem/eventplot/triplot dashed linestyles via data-space
+  dash segmentation (scales with zoom — not screen-space patterns);
+  `bar_label(fontsize=)` and pie/pie_label/table textprops
+  fontsize/ha/va; streamplot `start_points`, `integration_direction`,
+  array widths/colors, `num_arrows`.
+- `streamplot` now always uses the shim's bounded fixed-step integrator;
+  results no longer differ between environments with and without Matplotlib
+  installed, and paths approximate Matplotlib's adaptive integrator.
+- `hist(histtype="stepfilled")` renders a filled step polygon instead of
+  silently degrading to the unfilled step outline.
+- Documented the accepted visual approximations explicitly (barbs glyph,
+  imshow smoothing collapse and truecolor passthrough, annotate arrowprops,
+  errorbar limit carets, data-space dashes) and the HTML-only scope of
+  chrome rcParams.
+
+### PDSH gap features — 2026-07-13 (Matplotlib 3.11.0 reference)
+
+Driven by the Python Data Science Handbook ch. 4 benchmark (import-swap over
+14 notebooks): pass rate 121/171 → 154/171 runnable cells (90%), 147/154
+(95%) excluding the out-of-scope 3-D notebook, with zero savefig errors on
+passing cells.
+
+- Tick machinery: xy-owned `NullLocator`/`FixedLocator`/`MultipleLocator`/
+  `MaxNLocator`/`LinearLocator`/`LogLocator` and `NullFormatter`/
+  `FixedFormatter`/`FuncFormatter`/`FormatStrFormatter`/`StrMethodFormatter`/
+  `ScalarFormatter`, wired through `set_major_locator`/`set_major_formatter`
+  and resolved at build time so ticks track live data limits. `set_xticks`
+  and explicit labels displace stored tickers (last call wins). Minor
+  locators/formatters are retained but minor ticks still do not render.
+- Styles: `plt.style.context(...)` (snapshot/restore incl. theme tokens) and
+  the stock sheets fivethirtyeight, ggplot, bmh, dark_background, grayscale,
+  seaborn-v0_8-white(grid), reduced to the supported rcParams subset; new
+  `grid.color` rcParam wired into the axes chrome; `cycler()` (color only).
+- Colormaps: RdGy and jet engine tables (11 anchors sampled from Matplotlib
+  3.11) across Python SVG/PNG and the JS client; `LinearSegmentedColormap.
+  from_list` / `ListedColormap` as Python-side callables; `cm.get_cmap`.
+- Mappables: pyplot wrappers register the current image (`gci`/`sci`);
+  `plt.clim` retargets it and any colorbar derived from it; `set_clim` on
+  scatter/poly collections; scatter vmin/vmax now flow into a real
+  `color_domain` on the engine's color channel (previously they crashed the
+  render). `colorbar()` returns its handle from pyplot, falls back to the
+  current image, renders `ticks=`/`extend=` in PNG and SVG, and rejects
+  unknown kwargs (previously swallowed silently).
+- Layout: `plt.GridSpec` with slice spans and wspace/hspace/ratios resolved
+  to explicit figure rectangles; `add_subplot(spec, sharex=, sharey=,
+  xticklabels=[])`; `subplot(r, c, i)` mixes into figures that already hold
+  free-form axes; `subplots(subplot_kw=)`.
+- Axes surface: `get_figure`, `get_lines`, `get_shared_x/y_axes`,
+  `get_x/yticklabels` (recolorable handles), `set_facecolor` (+ the
+  `plt.axes(facecolor=)` route), `set_axisbelow(True)`, spine iteration with
+  deferred both-or-loud hiding, `tick_bottom`/`tick_left`, `fig.canvas`
+  facade; pandas `Period` coordinates convert to timestamps.
+- Fixed en route: `grid(linestyle='solid')` injected an invalid `None` style
+  value and crashed every subsequent export of that axes; `plt.subplot()` and
+  `plt.axes()` silently dropped their keyword arguments; `projection=` other
+  than rectilinear now fails with a clear NotImplementedError.
+- Known remaining boundaries measured by the benchmark: pandas' dynamic
+  timeseries plotting (its private ordinal-axis locators), legend geometry
+  options (`borderpad`, `labelspacing` — still loud), `Legend` handles for
+  second legends, markers on axhline, and 3-D axes.
+
+### Second review pass — 2026-07-13
+
+- Silent divergences converted to correct behavior: scatter drops rows masked
+  in x/y/s (not just c); `fill_between(interpolate=True)` draws single-point
+  `where` regions; `imsave` colormaps original values instead of a
+  pre-quantized uint8 copy; `set_cmap` validates names and feeds
+  imshow/scatter defaults; boxplot `sym` is honored (empty string suppresses
+  fliers) and flierprops colors reach the drawn dots; usermedians no longer
+  shift notch CIs; hexbin `mincnt` filtering and `C` aggregation use the same
+  bin membership.
+- Silent discards converted to loud rejections: bxp component linestyles,
+  secondary-axis `set_ticks` extras, axes/figure-fraction transforms on data
+  artists, `savefig(format='html', metadata=)`; singular transforms fail at
+  `set_transform` time with ValueError.
+- Export: SVG/HTML honor `savefig(facecolor=)` (background rect / styled
+  container); single-chart SVG includes the suptitle; composed-SVG suptitle
+  `y` maps as a figure fraction; non-Latin-1 PNG metadata keys raise
+  ValueError.
+- Scales: logit masks values at/outside (0, 1) instead of emitting ±inf;
+  scale-generated ticks refresh as data arrives and are dropped when the
+  scale returns to linear; explicit `set_*ticks` under a nonlinear scale
+  label the original data values.
+
+Future entries must identify the Matplotlib release/revision, inventory
+additions or removals, and any compatibility-level changes.