[v3] support for ragged arrays

[jhamman]

Zarr-Python 2 supported ragged arrays. This functionality has not made it into Zarr-Python 3 yet (see also #2617).

An example demonstrating this functionality using Zarr-Python 2:

z = zarr.empty(4, dtype=object, object_codec=numcodecs.VLenArray(int))
z
<zarr.core.Array (4,) object>
z.filters
[VLenArray(dtype='<i8')]
z[0] = np.array([1, 3, 5])
z[1] = np.array([4])
z[2] = np.array([7, 9, 14])
z[:]
array([array([1, 3, 5]), array([4]), array([ 7,  9, 14]),
       array([], dtype=int64)], dtype=object

This issue tracks the development of ragged arrays support in Zarr-Python 3.

[CSSFrancis]

Just hopping into this dicusssion, but this does limit the ability for Hyperspy to support zarr 3.0.0. Our usecase is for ragged arrays which should be supported, and don't have the same security issues that directly json encoding a python object.

We could just unwrap the ragged arrays and store them alongside an second array with information of how to recreate the ragged array. Is that the best way to handle this or is there a better way to encode varible length objects.

[d-v-b]

I think ragged arrays are definitely in-scope for 3.x, we just haven't had time to implement it.

[CSSFrancis]

@d-v-b Thanks for the response! There is the VLenBytesCodec which seems like it could handle most of the encoding as long as the underlying array is 1 dimensional? The underlying source says that this might be changed in the future and is not explicitly supporting in v3. Is that still correct?

[sehoffmann]

Any updates on this?

[rabernat]

In rabernat#1 we are developing an experimental prototype allowing any Arrow datatype to be stored in Zarr. This would enable ragged arrays using arrow list types.

[JamiePringle]

Just a follow on comment that for my application (https://github.com/JamiePringle/EZfate) in which we store the connectivity patterns between different parts of the coastal ocean, the ragged array type is very important. I look forward to seeing it come to v3! Jamie

[d-v-b]

We are still missing a specification of a data type. IMO it should probably be a generic vlen data type, which will subsume the existing variable-length bytes and variable-length string data types.

[dcherian]

Jamie!

Would it make more sense to store a sparse array instead? Not that Zarr has a good sparse story yet, but there are existing storage patterns for sparse array that could be adapted to Zarr

[JamiePringle]

Deepak --alas, no, it would be awkward. This is not a simple sparse connectivity matrix. I have a variable number of paths from location i to location j. I calculate things like the time to get from i to j, and the mean temperature experienced, etc. And all of these values are different for each individual path from i to j. So I can't do a simple E[i,j]=(one number)

Also, for my sins, I need to convert these data structures to R, so biologists can use them easily. The standard R data structure, the DataFrame, can handle ragged arrays.

hope all is well!