How to roundtrip to a ZipStore?

[TomNicholas]

In pydata/xarray#10457 and pydata/xarray#10485 we found a few xarray tests which roundtrip to a ZipStore. Previously the store could just be a writable one for both reading and writing, but now that's broken: #3189.

Unlike the other stores supplied in zarr-python, ZipStore does not implement .with_read_only, and IIRC someone said it cannot ever implement that method. But nevertheless I need to be able to write to a ZipStore, and then read back that same data, even if it's not the same python store instance. What's the recommended way to do that?

cc @d-v-b @maxrjones

[maxrjones]

You need to explicitly use a store with the read_only attribute as True when calling xr.open_zarr() because xarray's Zarr backend uses mode="r" which is incompatible with a writeable store under Zarr's current permissions model. This was a necessary breaking change to fix the bug leading to potential data deletion reported in #3147. While breaking, the change still follows Zarr's effver versioning policy - you only need to add one line of code to explicitly supply a read-only store. Nevertheless, I am sorry for the inconvenience.

For example:

# /// script
# dependencies = [
#   "xarray @ git+https://github.com/pydata/xarray@main",
#   "zarr @ git+https://github.com/zarr-developers/zarr-python@main",
#   "numpy==2.3.1",
#   "pooch",
#   "netcdf4",
# ]
# ///

import zarr
import numpy as np
import xarray as xr
# Store the array in a ZIP file
store = zarr.storage.ZipStore("example-3.zip", mode='w')
ds = xr.tutorial.load_dataset("rasm")
ds.to_zarr(store, mode="w", consolidated=False)

store = zarr.storage.ZipStore("example-3.zip", mode='r')
ds = xr.open_zarr(store, zarr_format=3, consolidated=False)
print(ds)

[TomNicholas]

You need to explicitly use a store with the read_only attribute as True

I know - but I'm asking if I already have a writable ZipStore, how do I get a read_only ZipStore that points to the same data? I can't use .with_read_only, but surely there must be a way to read the data I just wrote, so how do I do it? Is it possible to derive the information I need just from the first ZipStore instance?

[TomNicholas]

From a DM with @d-v-b :

from what I know, if all you want to do is read from it, you only need to know where it is on the file system

This is what I would have thought - can I not just use that path to create a new readonly store? I could explicitly close the first store before opening the second one using the same path? And if that's possible then why can't that just be the implementation of ZipStore.with_read_only?

[maxrjones]

This is what I would have thought - can I not just use that path to create a new readonly store?

You can use that path to create a new readonly store. That is what I showed in the code example in #3194 (comment). ZipStore also has a public path attribute which may be convenient to use in your tests.

[TomNicholas]

Okay thank you. So if that's possible, why don't we just implement .with_read_only on ZipStore using that approach?

[TomNicholas]

To be clear, I understand that the breaking change is necessary, and greatly appreciate your efforts here. But AFAICT this is a way to make the change a little less breaking?

[maxrjones]

Okay thank you. So if that's possible, why don't we just implement .with_read_only on ZipStore using that approach?

This seems sound to me, but I'm not very familiar with the ZipStore implementation so I'd defer to others as to whether there's a downside to this suggestion.

[d-v-b]

lets see what breaks if we implement with_read_only the naive way, by just creating a new filehandle with mode=r

[TomNicholas]

I can try that now

[TomNicholas]

Alright I tried it and I'm already confused. Is the file handle what's stored under ._zf? That property is only set when the store is opened, not when it's instantiated, but does .with_read_only count as opening the store?

[d-v-b]

you have found one of the wrinkles of the store API, which is that they can be initialized in a partial (i.e., useless for doing store stuff) state until _open is called. See

zarr-python/src/zarr/storage/_zip.py

Lines 93 to 104 in baabf08

	def _sync_open(self) -> None:
	if self._is_open:
	raise ValueError("store is already open")
	self._lock = threading.RLock()
	self._zf = zipfile.ZipFile(
	self.path,
	mode=self._zmode,
	compression=self.compression,
	allowZip64=self.allowZip64,
	)

[d-v-b]

so I think you just need to call _sync_open() after creating the store.