docs: add roadmap page outlining future plans#4149
Conversation
Adds a Roadmap page to the documentation describing the goals for the
next major cycle of work ("v4"), the intended changes per theme, and
the three-stream release model (additive minors, deprecations, one
minimal removals-only major).
Assisted-by: ClaudeCode:claude-fable-5
|
cc @ilan-gold @maxrjones, opening this as a draft for visibility. lmk if there's anything more or less we should say here |
This doesn't lead me where I thought it would, somehow linking to an old page |
| Zarr-Python around the stack, so that each level is something you can depend | ||
| on, conform to, or replace, without buying every other level: | ||
|
|
||
| - **A focused package per level** — `zarr-metadata`, `zarr-store`, |
There was a problem hiding this comment.
Would maybe note the zarrs example here
| complementary rather than competitive: Zarr-Python aims to be the best | ||
| pure-Python Zarr implementation *and* the best wrapper around the | ||
| compiled-language implementations, so that users who need native throughput can |
There was a problem hiding this comment.
I'm really conflicted on this, because I think baking fragmentation into the ecosystem is not great, but the flipside is that languages may have different features/advantages. It would be great to have a concrete reason (or list of reasons) as to why we would design around this level of extensibility
There was a problem hiding this comment.
good points, I'd say we want the backend to be an extension point for a few reasons:
- we always want a traditional python impl. IMO it's our job as the python implementation, and it keeps the ecosystem healthy to have multiple implementations
- we can't guarantee uniform, complete feature support across all implementations. As long as we allow people to hack on stuff in python, there will be things the python impl does that e.g. a rust impl doesn't support, and that's OK. For now, a compiled backend is effectively an external dep, and we simply can't promise what external deps will do in perpetuity.
- IMO a full backend is just an refined version of our current model, which requires extenders manage buffer prototypes, data type implementations, codec implementations, and store implementations. Consolidating all of this pluggability on a single surface should make extending Zarr Python easier even for other python backends (like a cupy backend, or a pysparse backend)
|
|
||
| ### Codecs | ||
|
|
||
| The current codec API wraps every codec in an unnecessary async layer (a |
| - **Engine pluggability at the chunk-decoding level** — alternative engines | ||
| (zarrs, TensorStore) can take over IO without re-implementing hierarchy | ||
| traversal, indexing, or metadata handling. |
There was a problem hiding this comment.
this seems a bit misaligned with other statements that imply pluggabillity at each level
| Each theme below is backed by a detailed technical proposal; the summaries | ||
| here describe the intended end state. |
There was a problem hiding this comment.
as we discussed at the dev meeting, if the idea is to reference zarr-python-planning for detailed technical proposals, we should move it to the zarr-developers github organization
| Make Zarr-Python's IO surfaces device-agnostic rather than adding GPU support | ||
| as a bolted-on feature: stores and codecs grow APIs for writing into a | ||
| caller-provided buffer (`read_into`, `decode_into`), and the `Array` facade | ||
| returns array-like objects in the user's chosen Array API namespace. GPU | ||
| support falls out once the assumption of CPU destinations is removed, and CPU | ||
| paths get faster too, because pre-allocated output buffers eliminate per-chunk | ||
| allocation. |
There was a problem hiding this comment.
@TomAugspurger it'd be great to get your thoughts on this component in particular.
| ### Decision points | ||
|
|
||
| Flipping the default of bare `array[...]` from eager to lazy is the single | ||
| highest-migration-cost item in the plan, so it is handled as an explicit | ||
| decision, not bundled into the additive work. The opt-in `array.lazy[...]` | ||
| accessor ships first, with no default change. Whether the default ever flips | ||
| hinges on whether Array API conformance at the bare-`__getitem__` surface turns | ||
| out to be a hard requirement; if it does, the flip happens as a long-window | ||
| deprecation with an explicit eager escape hatch and downstream coordination — | ||
| never as a reason to adopt a major version. |
There was a problem hiding this comment.
To some extent, I think it would be better to separate this from the roadmap. Include the commitment to delivering lazy indexing, leave the how to the detailed documents and reference zarr-python-planning.
Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com>
Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com>
Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4149 +/- ##
=======================================
Coverage 93.84% 93.84%
=======================================
Files 91 91
Lines 12549 12549
=======================================
Hits 11777 11777
Misses 772 772 🚀 New features to boost your workflow:
|
Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com>
The mkdocs-redirects plugin was generating a redirect stub for roadmap.md pointing at the old v3.0.8 docs, which clobbered the new roadmap page added in this PR. The developers/roadmap.html redirect is kept so old links to the historical v3 design roadmap still resolve. Assisted-by: ClaudeCode:claude-fable-5
3 participants
Adds a Roadmap page to the documentation describing the goals for the
next major cycle of work ("v4"), the intended changes per theme, and
the three-stream release model (additive minors, deprecations, one
minimal removals-only major), with an option for a
legacymodule.Assisted-by: ClaudeCode:claude-fable-5
Author attestation