Interface new implementation (API server) in dandi-cli top-level interfaces

[yarikoptic]

A decision was made in #283 to aim to get master in a shape to support interaction with both current girder-based deployed dandiarchive and WiP https://github.com/dandi/dandi-api implementations within the same code base. I think it is doable. #283 will provide "internal" interfaces for an upload/download cycle and then we would need to use/expose them through top level interfaces. For that, I think we should

• to known_instances add a new entry "dandi-api" which would have no girder, we ui, redirector and only API.
  • we would need also at least a dandi-api-local-docker-tests to point to the fixture'd instance
  • we should not occlude other existing fixtures/instances such as local-docker-tests since they reflect the instances of currently deployed dandiarchive setup (albeit now without any API which was provided by publish. for a bit)
    • edit: https://github.com/dandi/dandi-cli/pull/283/files#diff-96e695b73dd490b5f194b670b03c04d549000cb6c3e9b3fef76e8065c68c041aR100 suggests to reuse the same record, which seems to be Ok ATM since we have no longer publish. API to worry about. But I think it would just delay establishing such a dedicated record, whenever we do get a test deployment of new web ui/redirector which would work with new API. So IMHO might be better to be explicit and have separate dedicated records
  • most likely we should add metadata-version "field" to those records, where for old ones it would be 0 (unversioned ad-hoc), and then 1 for the new version of metadata (ideally it should be the API server which tells which version of metadata schema it expects, but that is -- later).

upload

• --dandi-instance option of dandi upload (in DANDI_DEVEL mode) would automagically list those added instance(s)
  • upload code should be RFed (main logic is in process_path so it could support both upload to original girder-based client and new API-based one
    • for new API-based one, metadata extraction should use new metadata (version 1) schema

download

for download, which does not rely on explicit specification of which instance to talk to, but parses from URL/identifier we need to come up with a "schema" on how to reference content from new API server, which would be easy on humans and flexible to support multiple instances/servers. Specification/parsing of such urls should be added to known_urls (https://github.com/dandi/dandi-cli/blob/master/dandi/dandiarchive.py#L109).

So what about just taking what we had/(have) on a recent iteration with publish API and tune it to correspond to our API (/dandisets/{version__dandiset__pk}/versions/{version__version}/assets/paths/) call:

       f"{server_grp}#.*/(?P<asset_type>dandiset)s/{dandiset_id_grp}"
        "/(?P<version>versions/([.0-9]{5,}|draft))"
        "(/assets/path(?location=(?P<location>.*)?)?)?"
        "$": {"server_type": "dandi-api"}

NB initially I thought we might need to add some optional prefix f"(dandi-api::)?" for early "decision making" but because of unique /dandisets/` I think we do not need it

so sample url for download would look like

• https://api.dandiarchive.org/dandisets/000001, https://api.dandiarchive.org/dandisets/000001/versions/draft, https://api.dandiarchive.org/dandisets/000001/versions/draft/path/ -- entire dataset, draft version
• https://api.dandiarchive.org/dandisets/000001/0.0.1 -- entire dataset, 0.0.1 version
• https://api.dandiarchive.org/dandisets/000001/draft/path/sub-XXX -- sub-XXX folder (or a file if points to an asset) of draft version

instead of api.dandiarchive.org, urls could point to localhost:port in the test.

Later, whenever web UI starts using API, we would need to adjust other "schemas". redirector will start redirecting to new web ui, possibly different urls. But for now, IMHO it is sensible to just follow the API URLs. Those will not be really "visible" to users, but would allow us to test new functionality/interaction with API server.

[jwodder]

@yarikoptic

we would need also at least a dandi-api-local-docker-tests to point to the fixture'd instance

The dandi-api Docker Compose setup seems to currently depend on the Girder Docker Compose setup, so they can't be split apart, so creating separate known_instance values doesn't seem like a good idea.

[yarikoptic]

I don't exactly see how known_instances so tightly bound to docker compose setup - it is ok to reuse the same compose setup for multiple instances with some code support

[jwodder]

@yarikoptic The Girder upload code has a number of provisions for handling files that have already been uploaded. Should this be carried over to the new API? I'm not entirely sure how all the parts should be translated.

[yarikoptic]

No. In our reimplantation, iirc, we will rely on gc (not sure if there already) to delete no longer used assets and unfinished uploads.

[yarikoptic]

No. In our reimplantation, iirc, we will rely on gc (not sure if there already) to delete no longer used assets and unfinished uploads.

[jwodder]

@yarikoptic Adding a new dandi-api server type for the new download URL pattern requires adding a dandi-api entry to _dandi_url_parser.map_to and populating it. How exactly should that be done?

[yarikoptic]

That is something I do not have immediate answer for. I have suggested/asked on slack either we might want to establish gui-beta.dandiarchive.org and then provide such mapping. But I am not sure if that is the approach we would be taking, so for now let's just breed a custom api+<URL> url schema to parse, so https://gui.dandiarchive.org/#/dandiset/000007 - would be old girder based, and api+https://gui.dandiarchive.org/#/dandiset/000007 would be the new one. That would allow later to change easily.

[jwodder]

@yarikoptic I'm not entirely clear on what you're proposing here. Could you elaborate?

[yarikoptic]

I have pushed f9fa14b to #330 which adds that and trims lots of elderly no longer needed URL schemas etc. See commit message (and some TODOs in the diff) for more information. Now you should be able to download api+https://gui.dandiarchive.org/#/dandiset/000007/draft or https://gui.dandiarchive.org/#/dandiset/000007/draft (which would be entirely different things ;)).