Use a release checklist and narrative announcement for breaking changes

[maxrjones]

I'd like to propose some improvements to Zarr-Python's release process based on a retro of the release of Zarr Python 3.0.9.

What happened

• June 30 - @d-v-b shared on the Zarr Zulip that he planned to release 3.0.9 and asked people to check for any issues within 24 hours of his announcement.
• July 1 - Zarr-Python 3.0.9 was released after no issues were raised
• July 1/2 - we received multiple concerns about the impact of Create read only copy if needed when opening a store path #3156 on Xarray:
  • How to roundtrip to a ZipStore? #3194
  • New release seems to break open_dataset from xarray #3189
  • #Zarr > release 3.0.9 @ 💬
  • Some DMs
• July 2 - Vincent Sarago let me know that titiler.xarray's tests were failing with the latest Zarr release. I opened a quick PR to fix the regression cause by Support async FSMap objects in zarr.open #2774. Part of that fix was broken off into Add missing import for AsyncFileSystemWrapper for _make_async in _fsspec.py #3195.
• July 2 - @TomNicholas raised Peformance regression: 3.0.9 issues twice as many GET requests when used with xarray #3199 about a potential performance regression based on Xarray's test suite. We eventually determined that this related to a bug in Xarray's test suite rather than any regression in Zarr Python
• July 3 - Zarr-Python 3.0.10 was released with fixes for the regression caused by Support async FSMap objects in zarr.open #2774

What we can do better in the future

• Explicitly test against downstream libraries prior to releases
  • Short term: Add instructions on checking the upstream CI workflows of downstream dependencies to a release-checklist
  • Long term: Consider adding CI to zarr-python's repository that runs downstream-libraries' test suites
• Label bug fixes that involve breakages (e.g., Create read only copy if needed when opening a store path #3156) in the changelog
• Put out a narrative announcement in addition to the more concise release notes any time there is a breaking change. I found myself repeating a lot last week something along the lines of "I'm sorry this is inconvenient and know that the with_read_only() compromise is not great. It was a necessary short-term fix to protect against data deletion" and think it would have been better to have a short narrative at the top of https://zarr.readthedocs.io/en/stable/release-notes.html#id2 to point to.

[d-v-b]

thanks for this writeup max!

Big +1 to more narrative communication. I think this can go a long way towards explaining why stuff is happening, especially when there's a historical context for it.

I'd love to see some automated downstream testing as part of our CI. Does anyone have ideas for implementing this, and / or suggestions for packages to consider testing against (xarray is evidently a good test target)?

[dcherian]

This runs Xarray's test suite against icechunk PR brances: https://github.com/earth-mover/icechunk/blob/9bf74ea813692c70897bda07c6cf2724ef825113/.github/workflows/python-check.yaml#L147-L175

[maxrjones]

The release checklist was added in #3210 and is being used for various releases.