Downloading data from BaseSpace

[arisp99]

Proposal

Currently, we use an outdated python script to download data from BaseSpace: https://github.com/bailey-lab/MIPTools/blob/70c9c26cd86af33f5eb75bdd0c4c43edfedc26d4/bin/BaseSpaceRunDownloader_v2.py. BaseSpace has released tools to work with data on the CLI. To prevent code breakage down the line, we propose using BaseSpace's tools.

Working With BaseSpace CLI

Below, I discuss some of my thoughts in reading through the BaseSpace documentation.

Installation

Installation is straightforward, however, we may consider changing the installation location. We may also need to change file permissions using chmod.

# Install
wget "https://launch.basespace.illumina.com/CLI/latest/amd64-linux/bs" -O $HOME/bin/bs

# Change file permission
chmod u+x $HOME/bin/bs

Authentication

Interactively, the user can run the authentication command and then go the URL provided to sign in.

bs auth
#> Please go to this URL to authenticate:  https://basespace.illumina.com/oauth/device?code=6Cesj

The resulting config file will be stored in $HOME/.basespace.

However, there are a couple of additional factors to consider. We need to think about the best way to automate this process. A couple of notes to consider:

1. We are able to specify the API server. It may be useful to let the user customize this depending on where they are located (e.g., US vs UK).
2. The user can store config info in a file and then load it: bs load config. This may make it easier to inject credentials.

Downloading data

Downloading data is simple, but there are many options. What is the best strategy to implement for our purposes?

# Single run
bs download run -i <RunID> -o <output>

# Multiple runs in a project
bs download project -i <ProjectID> -o <output>

# Subset of a project
bs download project -i <ProjectID> -o <output> --extension=fastq.gz

Implementation

We can install the CLI tool into our container. We will need to modify the download app to call either a series of commands or a script rather than the python script. We can provide several options with default values:

Flag	Function
-s, --api-server	the API server
-i, -run-id	run ID
-o, --out-dir	output dir

[arisp99]

@CliffO89: Can you provide some input into authentication? How have you been authenticating before downloading your data?

When we add this to MIPTools, it seems tedious to need to authenticate each time you'd like to download data. In that case, the user may want to store their config file somewhere and load it using bs load config.

[arisp99]

This issue and accompanying PR will be incorporated into future versions of MIPTools.

[JeffAndBailey]

bs is clean CLI for interacting with Illumina cloud

1. we use it for downloading MIP sequencing run
2. it is not free -- so a user must add it not us to repository
3. configuration and authentication is required
4. users may want to access addtional commands
   Therefore, to add nor not to add -- whether it is better to standalone or be integrated into MIPtools I would suggest if it is a simple singular command then we have users run outside of MIPtools. If internally we need things then integrate it.
   The reason it was previously in the pipe was there wasn't a clean interface in large part.

[aydemiro]

What are the specific issues with the current Illumine downloader script? Why would it cause breakage down the line? Depending on what the problems are, we may try to fix that script instead.

[arisp99]

2. it is not free -- so a user must add it not us to the repository

In order to install the CLI, I did not have to pay or log in to any account. The CLI itself, I believe, is free. It can be downloaded simply by using wget, curl, or even brew.

3. configuration and authentication is required

You do need an Illumina account in order to download data, but this is no different from the current download app.

4. users may want to access additional commands

If it is installed within the container, users should still be able to access additional commands by using the singularity exec command.

In my view, the benefits of the proposed app compared to the current app are as follows:

• Use an official tool instead of a web scrapping python script
• The python script uses deprecated tools to download data and, therefore, is likely to break¹
• The official CLI is significantly faster in terms of download speeds.

Footnotes

1. This could be fixed, but I believe it is better to use an official, faster tool. ↩

[aydemiro]

If it is faster we should use CLI. This was the opposite when I was testing some years ago; bs was much slower. The current script is also Illumina's software, btw. It is probably not supported anymore, so it would be up to us to maintain it if needed.

[arisp99]

If it is faster we should use CLI. This was the opposite when I was testing some years ago; bs was much slower.

With the example dataset I downloaded, the CLI was noticeably faster. I could certainly run some more tests to compare download speed as well...

The current script is also Illumina's software, btw. It is probably not supported anymore, so it would be up to us to maintain it if needed.

Ahh, I did not know this.