0.2.0: switch to nested layout

[joshmoore]

This defines the dimension separator (sometimes called
"key separator") for chunks in the NGFF file to be "/"
rather than ".". The benefit of storing hierarchically
is that for very large volumes the number of files in
a directory is limited to shape / chunk for a given
dimension.

close: #29

Review:

• Read: http://api.csswg.org/bikeshed/?url=https://raw.githubusercontent.com/joshmoore/ngff/0_2_0/0.2/index.bs
• Diff: https://services.w3.org/htmldiff?doc1=https%3A%2F%2Fngff.openmicroscopy.org%2Flatest%2F&doc2=http%3A%2F%2Fapi.csswg.org%2Fbikeshed%2F%3Furl%3Dhttps%3A%2F%2Fraw.githubusercontent.com%2Fjoshmoore%2Fngff%2F0_2_0%2F0.2%2Findex.bs

(Note: the review URLs use my branch name and therefore will be updated when I push)

Primary change:

[sbesson]

The W3 services diff looks good and makes sense to me thanks @joshmoore.

From the upstream Zarr perspective, this matches the work that has been engaged to use / as the default rather than .. Our latest understanding is that this will becomes the default in Zarr v3 - see zarr-developers/zarr-specs#78 for the original discussion. So in additional to solving concrete use cases, this also brings us closer to the upcoming specification release.

On the implementation side, various efforts have started and will keep happening to prepare tools to update data writing and reading to handle the new separator- see glencoesoftware/bioformats2raw#80 for instance.

My main question for other implementations was the following: what is the most direct way to detect that an image has been stored according to version 0.2 (or later) of this specification and that / should be used as a separator? My expectation would be the version key in the multiscales specification is the most relevant key/value pair here, is that correct?. If so, would it make sense to pulling the subset of the changes in #39 into 0.2. that expands this section of the specification to include a clear "version": "0.2" statement?

[joshmoore]

Practically, I think an initialization of the zarr store will happen to get access to the metadata and then if it's a version that requires special action (e.g. version == 0.1, therefore hard-code . as the separator) a new store will be created with a log statement to the user.

I thought about migrating @constantinpape's changes to this PR, but I wasn't completely sure on their status (i.e. if more are coming). Happy to add those on top of the 0.2/index.bs at any point in the future.

[constantinpape]

I thought about migrating @constantinpape's changes to this PR, but I wasn't completely sure on their status (i.e. if more are coming). Happy to add those on top of the 0.2/index.bs at any point in the future.

I think these changes are done, except for a decision about MUST vs SHOULD for the "axes" field.

I am happy to rebase #39 onto the changes here, or change #39 so that we merge it into this PR instead of master.

[joshmoore]

Going to start merging all the related nested-chunk-storage PRs. In implementing zarr-developers/zarr-python#716, I think I've convinced myself that we will be able to drop this requirement soon, i.e. this specification should be agnostic to the underlying storage, but I'd propose getting this in. Future versions can describe the nested layout as a SHOULD.

[joshmoore]

@constantinpape : happy to take your wording update on top of 0.2/* as you modify 0.3/*

[constantinpape]

@constantinpape : happy to take your wording update on top of 0.2/* as you modify 0.3/*

Sorry, I was busy with other stuff last week...
I will refactor and split the PRs now, so that we can get the wording update into 0.2.