should zarr-python ensure an upper bound on key range/

[dcherian]

Zarr version

v3

Numcodecs version

?

Python Version

?

Operating System

?

Installation

?

Description

At the moment hypothesis generates this kind of nonsensical byte range

path = PosixPath('/private/var/folders/sc/hsdthq2x7mnbfvwnk7rp4xmr0000gn/T/pytest-of-deepak/pytest-957/test_zarr_hierarchy_local_0/0')
prototype = BufferPrototype(buffer=<class 'zarr.core.buffer.cpu.Buffer'>, nd_buffer=<class 'zarr.core.buffer.cpu.NDBuffer'>)
byte_range = (None, 105072079929312)

which is fine for MemoryStore but not for LocalStore:

>               return prototype.buffer.from_bytes(f.read(end - f.tell()))
E               MemoryError

I also don't understand how this works if the byte_range is larger than the file.

Steps to reproduce

...

Additional output

No response

[jhamman]

In the most general case, without polling the underlying store/object, we aren't in a position to know the full size of the object. In practice though, we only use range-reads when we know the start/end and there is an implicit assumption that this always will fit into memory. Therefore, I think you could safely limit your upper limit to something much smaller.

[dstansby]

Is this good to close then?