array.nchunks is ambiguous

[d-v-b]

should array.nchunks report the number of shards or the number of chunks?

[dstansby]

At the moment it reports the number of chunks in the shards:

>>> import zarr
>>> arr = zarr.create_array(store={}, shape=(8,), shards=(4,), chunks=(2,), dtype='uint8')
>>> arr.nchunks
4

which seems to make sense to me? What's the situation/reasoning for it to report the number of shards instead?

[d-v-b]

the docstring says "The number of chunks in the stored representation of this array." For v3 data with sharding, the number of stored objects is not the same as the number of chunks.

[d-v-b]

nchunks_initialized has the same problem -- these properties come from a time before there was a difference between the number of chunks and the number of files stored on disk. that means we kind of need to pick one of the two meanings for these properties. for nchunks_initialized I think it's clear that the intent is to report the number of objects on disk supporting the array. I'm extending that logic to nchunks.

[LDeakin]

A bit of a curveball, but what if .chunks / .nchunks instead consistently mapped to the on-disk chunk shape, irrespective of the presence of the sharding codec? This is how it was in the zarr-python 3.0.0 betas.

At the moment, reading/writing "chunk-by-chunk" with a sharded array is a bit of a footgun. And workarounds like this are necessary:

• https://github.com/zarrs/zarr_benchmarks/blob/9c07a5ccf8994c76a9a02fcf9cb60a0a42b4860c/src/zarr_benchmarks/zarrs_python_benchmark_read.py#L54
• https://github.com/zarrs/zarr_benchmarks/blob/9c07a5ccf8994c76a9a02fcf9cb60a0a42b4860c/src/zarr_benchmarks/zarrs_dask_python_benchmark_roundtrip.py#L32

Introduce a new .subchunks concept instead? Perhaps a good time to change this is when you start supporting non-regular chunk grids, because .chunks will break anyway.

[d-v-b]

@LDeakin have a look at #3299, I go all-in on distinguishing shards (on-disk chunk shape) from chunks (unit of concurrent reading). If I understand you correctly, this would remove these footguns

[d-v-b]

A bit of a curveball, but what if .chunks / .nchunks instead consistently mapped to the on-disk chunk shape, irrespective of the presence of the sharding codec?

can you explain what you mean by "on-disk chunk shape" here?

[LDeakin]

The chunk shape defined by the chunk grid.

[d-v-b]

there are two complications with the idea of making the top-level Array attributes which refer to "chunks" to express the in-metadata chunks / chunk_grid.chunk_shape attributes:

• When reading data, people expect that the chunks attribute conveys the unit of parallelism for that operation. With sharding, this is no longer true.
• We are already distinguishing between shards and chunks in our array creation API. chunks specifies the unit of read parallelism, and shards, if specified, turns on sharding and allows separate control over the unit of write parallelism. So if you specify chunks alone, it controls the chunks / chunk_grid.chunk_shape attributes, but if you specify chunks and shards together, the chunks controls the chunk_shape attribute of the sharding codec config, and shards controls the chunks / chunk_grid.chunk_shape attributes. IMO we should try to be consistent with this terminology, which argues for normalizing "shards" as the in-memory name for the chunks / chunk_grid.chunk_shape attributes.