Access indexes in flight

[amniskin]

Overview

Users should be able to monitor dags in flight. This means accessing Index.dag as a normal dag. This would allow us to implement the following:

Dashboards

We can have dashboards as a node in the dag that allow users to monitor jobs while in flight. This is a typical thing in ML research.

Examples:

1. tensorboard
2. mlflow
3. weights and biases

Alternatives

We could just wait until the dag has finished. This is what we currently do, but sometimes you have to show progress to stakeholders, and this would make that easier.