RF "files.py" validation/metadata-loading to support BIDS

[yarikoptic]

#1011 is adding basic metadata extraction for BIDS datasets. Introduced in #1011 is more of a "hack" than proper addition of support for BIDS datasets. It is 'ad-hoc' in part due to the clear separation of "asset types" in https://github.com/dandi/dandi-cli/blob/HEAD/dandi/files.py e.g. to NWBAsset (with custom metadata extraction and validation) vs VideoAsset (nothing special ATM) to GenericAsset (really nothing special ;) ). With introduction of support of BIDS datasets it gets tricky:

• we need to upload pretty much every file (not just .nwb) if it is found to be a BIDS dataset
  • we decide if it is a BIDS dataset if there is dataset_description.json with BIDSVersion in it
• we might have "super-BIDS datasets" like https://dandiarchive.org/dandiset/000026/draft/files?location= where we have following hierarchy within a dandiset

derivatives/<some subdatasets some of which are BIDS>/
rawdata/ - BIDS dataset

so a dandiset can contain multiple BIDS (sb)datasets

• There is multiple "files" from which metadata could be loaded from. Below I outline 3 possible ways, but most likely we would offload both to 1 - file format specific (nwb and its .overwrite.json) + 2 - BIDS specific (using BIDS library), with BIDS overloading what prior one provided. But here are the details
  • metadata-precedence-1: metadata will/can come from filename in addition to being extracted from the data file. .nwb files are legit within BIDS datasets, so NWBAsset by itself is not describing entirety of the case. And for NWBAsset belonging to BIDS dataset we would want filename based metadata overload what is in the file.
  • metadata-precedence-2 NWB folks are working on introducing overlays support (WiP, not yet finalized). So for sub-1_slice-1.nwb it would likely to come from sub-1_slice-1.overwrite.json (if present).
  • metadata-precedence-3: metadata can come from BIDS sidecar file, e.g for sub-1_slice-1.nwb it could come from sub-1_slice-1.json
• note: it will be up for validator to complain whenever there is incongruence between different sources of metadata

@jwodder -- how files.py and anything else needed should be refactored so we support such multiple sources of metadata: file format based + BIDS

[jwodder]

@yarikoptic I'm thinking something like this:

• Add a DatasetDescriptionAsset(LocalFileAsset) class for representing dataset_description.json files
  • Any BIDS dataset-wide operations like metadata generation and validation are performed as methods of this class
• Add a BIDSFileAsset(LocalFileAsset) class for representing any other file underneath a directory with a dataset_description.json; instances will have an attribute storing the corresponding DatasetDescriptionAsset
• Add a NWBBidsAsset(NWBAsset, BIDSFileAsset) class for representing NWBs in BIDS datasets, with any necessary custom behavior
• Modify find_dandi_files() to check whether each directory contains a dataset_description.json and, if it does, produce the appropriate asset classes for the files within
• Give dandi_file a dataset_description: Optional[DatasetDescriptionAsset] = None argument for use when the file is known to reside within a BIDS dataset; if this argument is None, it is assumed to not be in one

Note that this assumes that, if a Dandiset contains one or more BIDS datasets that aren't at the root, then we want any non-NWB, non-Zarr files outside of the BIDS datasets to not be uploaded as assets by default.

[yarikoptic]

Sounds good for a start! I dislike a little implicit DatasetDescriptionAsset implying having a BIDS dataset. I guess we can start with that since I don't have a better idea ATM. Let's just make it more explicit BIDSDatasetDescriptionAsset and bids_dataset_description.

My main concern is the combinatoric NWBBidsAsset(NWBAsset, BIDSFileAsset) since we might then need to also do it for any other asset type such as VideoAsset and get VideoBidsAsset. I guess we can start doing this until we run into some real explosion would require instead establishing some more flexible construct. But what would you say about having CompositeBIDSFileAsset(BIDSFileAsset) which would be just parametrized at instantiation with an instance of underlying NWBAsset (or VideoAsset) which could be asked for its metadata/validation. This way it would just delegate to some "underlying" data type consistently across all such combinations without requiring multiple inheritance. WDYT?

NB I would prefer consistent capitalization of BIDS so to have NWBBIDSAsset even though parsing between two acronyms becomes impossible.

[jwodder]

@yarikoptic I'd prefer multiple inheritance to delegating to an attribute. For one thing, given an instance of an NWB in a BIDS dataset, I would expect isinstance(nwb, NWBAsset) and isinstance(nwb, BIDSFileAsset) to both be true.

Let's just make it more explicit ... bids_dataset_description.

Are you suggesting changing the name of the file that denotes a BIDS dataset? I don't think that's up to us.

[yarikoptic]

Let's just make it more explicit ... bids_dataset_description.

Are you suggesting changing the name of the file that denotes a BIDS dataset? I don't think that's up to us.

no, I am just suggesting adding BIDS to our class/option names as e.g. in dataset_description: Optional[DatasetDescriptionAsset] = None argument you suggested.

@yarikoptic I'd prefer multiple inheritance to delegating to an attribute. For one thing, given an instance of an NWB in a BIDS dataset, I would expect isinstance(nwb, NWBAsset) and isinstance(nwb, BIDSFileAsset) to both be true.

hm... ok... but then let's have still have the logic of behavior centralized, may be it should be a mixin NWBBidsAsset(NWBAsset, BIDSFileAsset, CompositeBIDSFileAssetMixIn) where it would be the CompositeBIDSFileAssetMixIn centralizing that logic of preferring BIDSFileAsset extracted metadata and doing the validation for consistency?

[yarikoptic]

@jwodder will work on RFing code per above, @TheChymera please point him to the interfaces for BIDS validation and metadata extraction he should use.

[jwodder]

@yarikoptic @TheChymera Question: What happens if a directory contains a dataset_description.json file and one of its (possibly nested) subdirectories also contains a dataset_description.json file? Does the inner file start a new BIDS dataset, or is it treated as just another file of the outer dataset?

[yarikoptic]

treat it as a "start of a new BIDS dataset". It will be for validator to decide if that is a permitted location to have such nested BIDS dataset (some, like rawdata/ or any directory under derivatives/ folder are ok to be nested BIDS datasets, and in turn have other nested BIDS datasets)

[TheChymera]

@jwodder from the point of view of the pilot DANDI support we have just merged, both would be detected as candidates, and the validation would be run for both.
However, running the validator on the parent directory will also index the child directory and will flag any duplicate “top level” files (such as README) as invalid.

This is a repercussion of top level matches being necessarily unique, i.e. unlike entity files there should only ever be one file matching one pattern...

Ultimately this is a shortcoming of the validator path selection and not of our DANDI wrapper.
I'll work on fixing it in the validator code.

[TheChymera]

Fix incoming: bids-standard/bids-specification#1145

[jwodder]

@TheChymera So, if I have a dataset_description.json file and a list of all files in a BIDS dataset:

• How do I validate the dataset/assets?
• How does the DANDI metadata for the assets differ from non-BIDS assets?

[TheChymera]

@jwodder if the dataset_description.json file is inside the BIDS directory, you would just just point the validator to the BIDS directory, e.g.:

chymera@decohost ~/src/bids-examples $ dandi validate-bids asl003/
2022-07-19 16:53:10,234 [ WARNING] A newer version (0.45.1) of dandi/dandi-cli is available. You are using 0.44.1+45.g2ed8929.dirty
2022-07-19 16:53:10,384 [    INFO] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2022-07-19 16:53:10,384 [    INFO] NumExpr defaulting to 8 threads.
2022-07-19 16:53:11,608 [ WARNING] BIDSVersion 1.5.0 is less than the minimal working 1.7.0+369. Falling back to 1.7.0+369. To force the usage of earlier versions specify them explicitly when calling the validator.
All filenames are BIDS-valid and no mandatory files are missing.
2022-07-19 16:53:12,824 [    INFO] Logs saved in /home/chymera/.cache/dandi-cli/log/20220719205309Z-9230.log

If you do not have the directory at hand and just a list of paths which do not exist on your system this will only be possible after we have merge dummy_path support:

• Upstream PR including this improvement: [ENH] Validator inheritance support bids-standard/bids-specification#1120
• DANDI fast-track PR which includes this feature before we start using upstream code as a library: [ENH] BIDS Metadata reporting #1061

How does the DANDI metadata for the assets differ from non-BIDS assets?

Currently it differs in that the keys are not fully homogenized with the DANDI nomenclature. I did a bit of standardizing here, but there are a few more keys coming in a new PR soon.

[jwodder]

@TheChymera

• I meant, how would I do the validation using the Python API, not the CLI?
• How should I generate the metadata for BIDS assets?

[TheChymera]

@jwodder

I meant, how would I do the validation using the Python API, not the CLI?

currently that can't be done on :master because we need the actual paths, however, as soon as the dummy_paths parameter is in, you could do it as I have in this PR (it's the same PR which implements dummy path support ahead of upstream):

dandi-cli/dandi/bids_utils.py

Lines 116 to 131 in 1ae0a3b

	with DandiAPIClient.for_dandi_instance("dandi") as client:
	dandiset = client.get_dandiset(dandi_id)
	path_list = []
	for asset in dandiset.get_assets():
	i = f"dummy/{asset.path}"
	if "sub-MITU01h3" in i and "sub-MITU01" in i:
	lgr.warning("Fixing subject field inconsistencies:")
	lgr.warning(" - Pre-repair: %s", i)
	i = i.replace("sub-MITU01h3", "sub-MITU01")
	lgr.warning(" + Post-repair: %s", i)
	# ome.zarr support pending:
	# https://github.com/dandi/dandi-cli/pull/1050
	if "ome.zarr" not in i:
	path_list.append(i)
	result = validate_bids(path_list, dummy_paths=True)

How should I generate the metadata for BIDS assets?

The validator returns a list of dictionaries for all matches found, where the keys are BIDS-standard (but not DANDI-standard) metadata fields, that is the standardization I previously mentioned:

dandi-cli/dandi/bids_utils.py

Line 134 in 1ae0a3b

match_listing = result["match_listing"]