MissingOutputException with directory and GS remote provider

[eric-czech]

I am trying to run a workflow rule that creates a directory in GS, but Snakemake continually fails to recognize that the directory exists. The error message recommends using the directory() flag, which I am.

This appears to be related to #396.

Snakemake version

5.22.1

Describe the bug

Output directories are flagged as missing when using GS remote provider.

Logs

The most salient part:

Uploading to remote: rs-ukb/logs/bgen_to_zarr.XY.txt
Finished upload.
ImproperOutputException in line 57 of /workdir/Snakefile:
Outputs of incorrect type (directories when expecting files or vice versa). Output directories must be flagged with directory(). for rule bgen_to_zarr:
rs-ukb/prep-data/gt-imputation/ukb_chrXY.zarr
  File "/opt/conda/envs/snakemake/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 544, in handle_job_success
  File "/opt/conda/envs/snakemake/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 225, in handle_job_success

Full log: error_log.txt

Minimal example

Here is the offending rule, and I apologize that this isn't fully reproducible but it's difficult to share some of the details:

def bgen_samples_path(wc):
    n_samples = bgen_contigs.loc[wc.bgen_contig]['n_consent_samples']
    return [f"raw-data/gt-imputation/ukb59384_imp_chr{wc.bgen_contig}_v3_s{n_samples}.sample"]

rule bgen_to_zarr:
    input:
        bgen_path="raw-data/gt-imputation/ukb_imp_chr{bgen_contig}_v3.bgen",
        variants_path="raw-data/gt-imputation/ukb_mfi_chr{bgen_contig}_v3.txt",
        samples_path=bgen_samples_path
    output:
        directory("prep-data/gt-imputation/ukb_chr{bgen_contig}.zarr")
    params:
        contig_index=lambda wc: bgen_contigs.loc[str(wc.bgen_contig)]['index']
    conda:
        "envs/gwas.yaml"
    log:
        "logs/bgen_to_zarr.{bgen_contig}.txt"
    shell:
        # This will write to the local {output} path
        "python scripts/convert.py bgen_to_zarr "
        "--input-path-bgen={input.bgen_path} "
        "--input-path-variants={input.variants_path} "
        "--input-path-samples={input.samples_path} "
        "--output-path={output} "
        "--contig-name={wildcards.bgen_contig} "
        "--contig-index={params.contig_index} "
        "--remote=False 2> {log} "

Invocation:

snakemake --use-conda --cores 1 \
--default-remote-provider GS --default-remote-prefix $GS_BUCKET \
$GS_BUCKET/prep-data/gt-imputation/ukb_chrXY.zarr

I also get the same error when running on a cluster, i.e. using:

snakemake --use-conda --kubernetes \
--default-remote-provider GS --default-remote-prefix $GS_BUCKET \
$GS_BUCKET/prep-data/gt-imputation/ukb_chrXY.zarr

Additional context

I am able to work around this by using an individual checkpoint/sentinel file of some kind, but it's unclear to me if directories are even supported for Google Storage. Is that in the docs somewhere? Am I just trying to use some feature that doesn't exist?

[eric-czech]

Is this the answer I'm looking for?

from snakemake.remote.GS import RemoteProvider as GSRemoteProvider
GS = GSRemoteProvider()
GS.allows_directories
> False

[vsoch]

hey @eric-czech just a quick question - if the *.zarr file is a directory, did you try appending a slash to the end? E.g., prep-data/gt-imputation/ukb_chr{bgen_contig}.zarr/.

[eric-czech]

Hm I had not but I just tried a few variations on that using a simpler rule:

rule test:
    input:
        f"tmp/input.txt"
    output:
        # Attempts:
        # "tmp/output.zarr"
        # "tmp/output.zarr/"
        # directory("tmp/output.zarr")
        directory("tmp/output.zarr/")
    conda:
        "envs/spark.yaml"
    shell:
        "mkdir -p {output} && "
        "echo '1' > {output}/f1.txt && "
        "echo '2' > {output}/f2.txt"

I got the same result each time -- here's the log from one of the runs:

Log

> snakemake --use-conda --cores=1 --local-cores=1 --default-remote-provider GS --default-remote-prefix rs-ukb rs-ukb/tmp/output.zarr

Building DAG of jobs...
Using shell: /bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job counts:
count jobs
1 test
1

[Wed Aug 19 16:16:20 2020]
rule test:
input: rs-ukb/tmp/input.txt
output: rs-ukb/tmp/output.zarr
jobid: 0

Skipped removing non-empty directory rs-ukb/tmp/output.zarr
Downloading from remote: rs-ukb/tmp/input.txt
Finished download.
Activating conda environment: /home/eczech/repos/ukb-gwas-pipeline-nealelab/.snakemake/conda/1a7806c2
ImproperOutputException in line 109 of /home/eczech/repos/ukb-gwas-pipeline-nealelab/Snakefile:
Outputs of incorrect type (directories when expecting files or vice versa). Output directories must be flagged with directory(). for rule test:
rs-ukb/tmp/output.zarr
File "/home/eczech/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/executors/init.py", line 544, in handle_job_success
File "/home/eczech/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/executors/init.py", line 225, in handle_job_success
Removing output files of failed job test since they might be corrupted:
rs-ukb/tmp/input.txt, rs-ukb/tmp/output.zarr
Skipped removing non-empty directory rs-ukb/tmp/output.zarr
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
An error occurred
Complete log: /home/eczech/repos/ukb-gwas-pipeline-nealelab/.snakemake/log/2020-08-19T161617.862561.snakemake.log

[eric-czech]

I thought an extension on the directory name could be an issue so I tried with and without the .zarr (and with/without trailing slashes) but still got the same error.

[vsoch]

I think the core issue here (and the reason we have allow_directories set to False) is because we would need a way to determine if a "subfolder" in Google Storage exists. Here is a dummy test for one of these paths that I know to exist:

bucket = client.bucket('snakemake-testing')
<Bucket: snakemake-testing>

bucket.blob("1/").exists()
False

bucket.blob("1").exists()
False

So they key would be finding a way to get support for that. Another alternative is that snakemake could (when it's creating the inventory) keep a set of separate directory names that are deemed to exist based on filenames. And of course the third options to do what you already did, and use a file in the directory as a proxy for the directory existence. @johanneskoester what are your thoughts?

[aryarm]

Hi all, I'll just chime in to say that I've also been having this problem and my vote is for creating a set of separate directory names based on filenames, since that is what it seems like others have done.

[CowanCS1]

I have also hit this issue when attempting to move an existing workflow onto Google's Cloud Life Sciences.

• As expected from the above discussion, any output with the directory() wrapper gets removed causing a MissingOutputException

• If I instead set the output to a created file within the created sub-directory, I still get the same error. A possible cause is that most of the created files for the tool I am using (Star aligner) first creates directories (e.g. path/Log.out/.../... ) and then aggregates results into a file with the same name ( path/Log.out ). The original designation of the path/filename as a directory appears to persist causing it to be removed, exemplified by the fact that the log proceeds to state Skipped removing non-empty directory path/Log.out. I looked through the code to see if this was feasible, but I am not clear on when each function is called.

• If I create a file as a proxy to indicate the directory's existence, the workflow completes without error but the only file that gets transferred to remote (GCS) is the proxy file, not the rest of the outputs in the created sub-directory. From the comments of @vsoch and @eric-czech I would have expected that this would cause the entire sub-directory to be tracked and all outputs saved to remote upon completion. For example, I create the tracking file path/status.txt and as my sole output and then in remote storage I will see bucket/path/status.txt but not other files such as bucket/path/Log.out.

It would be very useful to hear what parts of this are expected behavior. At present the only method I can think of to make this particular rule work on Cloud Life Sciences is to list all of the required outputs explicitly, and attempt to remove the is_directory flag where it incorrectly persists by having the remote shell copy them into a second location after they are created.

[vsoch]

hey @CowanCS1 I just wanted to let you know that I hear you! There is one thing I want to try that might allow for a folder, and I'll try to make some time this weekend. I can't make promises because Google Storage isn't really designed to have "folders" (a folder is just part of the path to a file in an object store) but I at least want to test something out. I'll let you know if it gets anywhere. In the meantime, could you provide a simple Snakefile that reproduces your particular issue? That will be very helpful for me to test.

[CowanCS1]

Hi @vsoch - thanks! I very much appreciate the work you and others have done to make it easier to use Snakemake on GCP.

I wrote a minimal-ish example to both demonstrate the issue and identify a work-around for uploading arbitrary directory contents, aiming to keep the solution somewhat within the existing framework. The work-around I have settled on is to use the gls.py helper file and explicitly start a "save to remote" call for the relevant sub-directories. As mentioned above, a single file in the sub-directory is referenced as the snakemake output.

Here is the example with work-around in place:

Snakemake version: 5.26.1

Command used to execute workflow:

snakemake --google-lifesciences --use-conda --default-remote-prefix bucket --google-lifesciences-region europe-west1 -p

The code is always run with the most recent version of all files committed into a github repository.

rule results:
    input:
        'data/test/1.txt'
    resources:
        machine_type = 'n1-standard-16', # vCPUs: 16, RAM: 60GB
        disk_mb      = 10000             # 10 GB


rule test_save:
    input:
        gtf               = expand("data/ref/{gtf}.gtf", gtf=config['gtf']),      # ignore the specifics, holdover from the original rule I modified
    output:
        first             = 'data/test/1.txt'
    resources:
        machine_type      = 'n1-standard-16', # vCPUs: 16, RAM: 60GB
        disk_mb           = 10000             # 10 GB
    log:
        'log/test_save.log',
    shell:
        """
        /bin/bash
        echo "Creating files" >&{log} 
        mkdir -p bucket/data/test/ && for i in $(seq 1 5); do echo $i > bucket/data/test/$i.txt; done && ls -l bucket_workflow/data/test >>{log} 2>&1
        echo "Downloading gls.py" >>{log} 2>&1
        wget -O /gls.py https://gist.githubusercontent.com/vsoch/f5a6a6d1894be1e67aa4156c5b40c8e9/raw/a4e9ddbeba20996ca62745fcd4d9ecd7bfa3b311/gls.py >>{log} 2>&1
        echo "Setting attributes of gls.py" >>{log} 2>&1
        chmod +x /gls.py >>{log} 2>&1
        echo "Activating snakemake virtual environment" >>{log} 2>&1
        echo "Done" >>{log} 2>&1
        set +eu
        source activate snakemake >>{log} 2>&1
        set -eu
        echo "Executing gls.py to save bucket/data/test" >>{log} 2>&1
        python /gls.py save bucket bucket/data/test data/test >>{log} 2>&1
        """

And the resulting log file:

Creating files
total 20
-rw-r--r-- 1 root root 2 Oct  9 13:24 1.txt
-rw-r--r-- 1 root root 2 Oct  9 13:24 2.txt
-rw-r--r-- 1 root root 2 Oct  9 13:24 3.txt
-rw-r--r-- 1 root root 2 Oct  9 13:24 4.txt
-rw-r--r-- 1 root root 2 Oct  9 13:24 5.txt
Downloading gls.py
--2020-10-09 13:24:51--  https://gist.githubusercontent.com/vsoch/f5a6a6d1894be1e67aa4156c5b40c8e9/raw/a4e9ddbeba20996ca62745fcd4d9ecd7bfa3b311/gls.py
Resolving gist.githubusercontent.com (gist.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...
Connecting to gist.githubusercontent.com (gist.githubusercontent.com)|151.101.0.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4071 (4.0K) [text/plain]
Saving to: ‘/gls.py’

     0K ...                                                   100% 49.9M=0s

2020-10-09 13:24:51 (49.9 MB/s) - ‘/gls.py’ saved [4071/4071]

Setting attributes of gls.py
Activating snakemake virtual environment
Executing gls.py to save data/test

The following files will be uploaded: data/test/3.txt
data/test/4.txt
data/test/5.txt
data/test/2.txt
data/test/1.txt
data/test/3.txt -> bucket/data/test/3.txt
Uploading data/test/3.txt to bucket/data/test/3.txt
data/test/4.txt -> bucket/data/test/4.txt
Uploading data/test/4.txt to bucket/data/test/4.txt
data/test/5.txt -> bucket/data/test/5.txt
Uploading data/test/5.txt to bucket/data/test/5.txt
data/test/2.txt -> bucket/data/test/2.txt
Uploading data/test/2.txt to bucket/data/test/2.txt
data/test/1.txt -> bucket/data/test/1.txt
Uploading data/test/1.txt to bucket/data/test/1.txt
Done

• Changing the output in the above code to data/test reproduces the findings of @eric-czech
• Without python /gls.py save bucket bucket/data/test data/test the workflow will complete successfully, but only the file bucket/data/test/1.txt will be present in GCS not the other 4 files. Possibly the expected behavior, I am unclear based on the discussion above, but one would desire that there would be a way to track and save created files within certain directories without explicitly listing each output file within the directory explicitly.
• I haven't completed tests with the hypothesized "create file as directory, remove, rename" issue, as it is circumvented by this work-around.

Errata:

• This, and all other workflows I have tested refuse to overwrite the Snakemake log files at bucket/bucket/google-lifesciences-logs/. They will write them the first time, or replace them if they are deleted, but will not overwrite existing files. Expected behavior is for the log files to be overwritten on subsequent runs. Log files for individual rules work as expected.
• I am aware the entire command would normally be inside a /bin/bash -c "" but it was better legible this way and didn't appear to break anything downstream.
• The set +eu command was to avoid this issue: [conda 4.6] activate fails due to unbound PS1 in bash strict mode conda/conda#8186

[aryarm]

@CowanCS1 I can confirm the bug with the log files not getting overwritten. We've experienced this ourselves, too.
But I didn't have the time to come up with a minimally reproducible example, so I never submitted a proper issue for it.
Also, I wasn't sure it it was a bug or just intended behavior.

[vsoch]

hey @CowanCS1! So it looks like you are copying the code that the worker runs for the step (downloading gls.py) can you provide a basic example of generating some number of files with a directory prefix for the output that fails? We need to work on one thing at a time, and there are many moving pieces in the Snakefile above.

[CowanCS1]

Hi @vsoch ! Absolutely.

Snakemake version: 5.26.1

Rule:

# Return all files created in the sub-directory /data/test/
rule results:
    input:
        directory('data/test/')

# Run a command which creates an arbitrary set of files in /data/test/
rule minimal_test:
    resources:
        machine_type      = 'n1-standard-4', 
        disk_mb           = 10000           
    output:
        directory('data/test/')
    shell:
        '''
        mkdir -p {output}
        for i in $(seq 1 5); do echo $i > {output}/$i.txt; done
        '''

Output when executed with snakemake --google-lifesciences --use-conda --default-remote-prefix bucket --google-lifesciences-region europe-west1 -p:

ImproperOutputException in line 7 of /workdir/Snakefile:
      Outputs of incorrect type (directories when expecting files or vice versa). Output directories must be flagged with directory(). for rule minimal_test:

Purpose of rule: Intended to mimic any rule whose output is a collection of files stored within a new sub-directory.

Expected outcome: For all 5 files created within the bucket/data/test subdirectory within the GCP VM instance to be saved to bucket/data/test/ on GCS. When this rule is run locally with snakemake --cores 8 there are no errors, and the files are present.

Hopefully this is what you were requesting - there is a degree of uncertainty on my part, but I'm happy to help further as needed.