Zarr version
v2.14.2
Numcodecs version
v0.11.0
Python Version
v3.8.10
Operating System
Linux (Google Colab)
Installation
Google Colab (!pip install --upgrade zarr xarray fsspec aiohttp)
Description
When accessing a sparse zarr Array over HTTP missing chunks are handled inconsistently with and without simplecache. When simplecache is chained into the zarr.storage.FSStore missing files (HTTP 404) raise a KeyError, instead of being filled with the fill_value.
This behaviour extends to popular access methods, such as xarry.open_zarr().
Unfortunately, the apparent change in behaviour coincides with a revised structure on my side. Therefore, I am unsure on when the bug (first) occurred or if this configuration ever worked.
Steps to reproduce
A query along the first axis without simplecache works:
z = zarr.convenience.open_consolidated('https://geotiles.nl/cube.zarr')
z['S2']['R051']['B04'][:, -1, -1]With simplecache, it fails:
z = zarr.convenience.open_consolidated('simplecache::https://geotiles.nl/cube.zarr')
z['S2']['R051']['B04'][:, -1, -1]as fsspec fails to identify the missing key as empty chunk:
---------------------------------------------------------------------------
FileNotFoundError Traceback (most recent call last)
[/usr/local/lib/python3.8/dist-packages/fsspec/mapping.py](https://localhost:8080/#) in getitems(self, keys, on_error)
92 try:
---> 93 out = self.fs.cat(keys2, on_error=oe)
94 if isinstance(out, bytes):
40 frames
FileNotFoundError: https://geotiles.nl/cube.zarr/S2/R051/B04/343.10.11
The above exception was the direct cause of the following exception:
KeyError Traceback (most recent call last)
[/usr/local/lib/python3.8/dist-packages/fsspec/mapping.py](https://localhost:8080/#) in getitems(self, keys, on_error)
95 out = {keys2[0]: out}
96 except self.missing_exceptions as e:
---> 97 raise KeyError from e
98 out = {
99 k: (KeyError() if isinstance(v, self.missing_exceptions) else v)
KeyError:Additional output
Explicitly mentioning FileNotFoundError in the list of missing_exception of zarr.storage.FSStore() has no effect.
z = zarr.convenience.open_consolidated(zarr.storage.FSStore('simplecache::https://geotiles.nl/cube.zarr', normalize_keys=False, mode='r', missing_exceptions=(FileNotFoundError,)))
z['S2']['R051']['B04'][:, -1, -1]---------------------------------------------------------------------------
FileNotFoundError Traceback (most recent call last)
[/usr/local/lib/python3.8/dist-packages/fsspec/mapping.py](https://localhost:8080/#) in getitems(self, keys, on_error)
92 try:
---> 93 out = self.fs.cat(keys2, on_error=oe)
94 if isinstance(out, bytes):
17 frames
FileNotFoundError: https://geotiles.nl/cube.zarr/S2/R051/B04/343.10.11
The above exception was the direct cause of the following exception:
KeyError Traceback (most recent call last)
[/usr/local/lib/python3.8/dist-packages/fsspec/mapping.py](https://localhost:8080/#) in getitems(self, keys, on_error)
95 out = {keys2[0]: out}
96 except self.missing_exceptions as e:
---> 97 raise KeyError from e
98 out = {
99 k: (KeyError() if isinstance(v, self.missing_exceptions) else v)
KeyError:
Zarr version
v2.14.2
Numcodecs version
v0.11.0
Python Version
v3.8.10
Operating System
Linux (Google Colab)
Installation
Google Colab (!pip install --upgrade zarr xarray fsspec aiohttp)
Description
When accessing a sparse zarr Array over HTTP missing chunks are handled inconsistently with and without
simplecache. Whensimplecacheis chained into thezarr.storage.FSStoremissing files (HTTP 404) raise aKeyError, instead of being filled with thefill_value.This behaviour extends to popular access methods, such as
xarry.open_zarr().Unfortunately, the apparent change in behaviour coincides with a revised structure on my side. Therefore, I am unsure on when the bug (first) occurred or if this configuration ever worked.
Steps to reproduce
A query along the first axis without
simplecacheworks:With
simplecache, it fails:as
fsspecfails to identify the missing key as empty chunk:--------------------------------------------------------------------------- FileNotFoundError Traceback (most recent call last) [/usr/local/lib/python3.8/dist-packages/fsspec/mapping.py](https://localhost:8080/#) in getitems(self, keys, on_error) 92 try: ---> 93 out = self.fs.cat(keys2, on_error=oe) 94 if isinstance(out, bytes): 40 frames FileNotFoundError: https://geotiles.nl/cube.zarr/S2/R051/B04/343.10.11 The above exception was the direct cause of the following exception: KeyError Traceback (most recent call last) [/usr/local/lib/python3.8/dist-packages/fsspec/mapping.py](https://localhost:8080/#) in getitems(self, keys, on_error) 95 out = {keys2[0]: out} 96 except self.missing_exceptions as e: ---> 97 raise KeyError from e 98 out = { 99 k: (KeyError() if isinstance(v, self.missing_exceptions) else v) KeyError:Additional output
Explicitly mentioning
FileNotFoundErrorin the list ofmissing_exceptionofzarr.storage.FSStore()has no effect.--------------------------------------------------------------------------- FileNotFoundError Traceback (most recent call last) [/usr/local/lib/python3.8/dist-packages/fsspec/mapping.py](https://localhost:8080/#) in getitems(self, keys, on_error) 92 try: ---> 93 out = self.fs.cat(keys2, on_error=oe) 94 if isinstance(out, bytes): 17 frames FileNotFoundError: https://geotiles.nl/cube.zarr/S2/R051/B04/343.10.11 The above exception was the direct cause of the following exception: KeyError Traceback (most recent call last) [/usr/local/lib/python3.8/dist-packages/fsspec/mapping.py](https://localhost:8080/#) in getitems(self, keys, on_error) 95 out = {keys2[0]: out} 96 except self.missing_exceptions as e: ---> 97 raise KeyError from e 98 out = { 99 k: (KeyError() if isinstance(v, self.missing_exceptions) else v) KeyError: