Project: Exploring a Data Repo

[kadrlica]

@drphilmarshall made a great start at this here; however, I think we need to go a bit farther and document what is in each dataset rather than just listing the data sets.

[drphilmarshall]

@kadrlica is too kind: I'd say I made a _minimal_ start, rather than a great one :-) Definitely agree with his assessment, though - when I ran out of time I was starting to think about writing a `def` that does the following things, given a repo: * Display a big bold markdown section heading with the name of the dataset * Print the README of the dataset directory (or a warning if there is none...) * List the directory contents * Check which pipeline tasks have been run / which datatypes are available in the repo. (This is non-trivial with the gen 2 butler, and will be easier with the gen 3 butler) * Compute some vital statistics (no of visits, sky are covered, no of objects, no of populated columns in the object table, etc etc), probably for each available data type. * Plot the sky coverage (that's what I needed Jim's DC2 notebook for) and so on. The Exploring a Data repo notebook could either demo this function on one of our repos, or _it could just contain the pieces of functionality that we want the function to contain_ I'd probably start this way, and postpone the actual writing of the function to the `DataInventory.ipynb` notebook itself (and then I'd put the function in a `stackclub` library module and import it to the inventory notebook). Either way, the data inventory notebook ( #130 ) should use this function to loop over all available datasets, and we'd have an automatic inventory, that grew as our available datasets multiply.

[drphilmarshall]

@rmorgan10 I just edited my comment above to better reflect my current thinking about what the Exploring A Data Repo and DataInventory notebooks should each do, see what you think. These notes are consistent with the current state of the explorer notebook - I'm currently trying to figure out how to use the butler to check for the existence of various data types.

Some more notes:

• In the DataInventory notebook I started trying to download an external notebook that contains functions to plot tracts and patches on the sky and import it as a module - I got close, but not close enough. I think the next step might be to just copy the relevant code from the remote notebook, and provide correct attribution to its author. All that should then happen in the Exploring notebook instead, while the Data Inventory notebook is kept clean.
• Jeff Carlin showed some useful-looking visualization code in his tutorial stub on Friday, browse the project branches and commit history to find it.
• I think we can get quick code snippets for butler-querying repos from the DM team, either in #dm-newbies or #dm-pipelines - my guess is they have to do this kind of thing all the time.
• Gregory Dubois-Felsmann @gpdf has comments on this project - he's part of the team developing the gen 3 butler, and said he wanted to talk to us! I'll point him at this thread so he can comment.

[drphilmarshall]

Since @rmorgan10 showed the data inventory working pretty well at the Holiday Party, I'm going to close this issue and its sibling(?) #130, and issue an upgrade as a new project. Well done Rob! :-)