Merge branch 'dev' into nf-core-template-merge-2.8

Author: maxulysse

.editorconfig

@@ -22,3 +22,11 @@ indent_size = unset
 
 [/assets/email*]
 indent_size = unset
+
+# C++ compiles code
+[/bin/cutsite_trimming]
+end_of_line = unset
+insert_final_newline = unset
+trim_trailing_whitespace = unset
+indent_style = unset
+indent_size = unset

.github/workflows/awsfulltest.yml

@@ -14,10 +14,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Launch workflow via tower
-        uses: seqeralabs/action-tower-launch@v1
-        # TODO nf-core: You can customise AWS full pipeline tests as required
-        # Add full size test data (but still relatively small datasets for few samples)
-        # on the `test_full.config` test runs with only one set of parameters
+        uses: nf-core/tower-action@v3
         with:
           workspace_id: ${{ secrets.TOWER_WORKSPACE_ID }}
           access_token: ${{ secrets.TOWER_ACCESS_TOKEN }}
@@ -27,7 +24,7 @@ jobs:
             {
               "outdir": "s3://${{ secrets.AWS_S3_BUCKET }}/hic/results-${{ github.sha }}"
             }
-          profiles: test_full,aws_tower
+          profiles: test_full,public_aws_ecr
       - uses: actions/upload-artifact@v3
         with:
           name: Tower debug log file

.github/workflows/awstest.yml

@@ -22,7 +22,7 @@ jobs:
             {
               "outdir": "s3://${{ secrets.AWS_S3_BUCKET }}/hic/results-test-${{ github.sha }}"
             }
-          profiles: test,aws_tower
+          profiles: test,public_aws_ecr
       - uses: actions/upload-artifact@v3
         with:
           name: Tower debug log file

.github/workflows/ci.yml

@@ -31,13 +31,10 @@ jobs:
         uses: actions/checkout@v3
 
       - name: Install Nextflow
-        uses: nf-core/setup-nextflow@v1
+        uses: nf-core/setup-nextflow@v1.3.0
         with:
           version: "${{ matrix.NXF_VER }}"
 
       - name: Run pipeline with test data
-        # TODO nf-core: You can customise CI pipeline run tests as required
-        # For example: adding multiple test runs with different parameters
-        # Remember that you can parallelise this by using strategy.matrix
         run: |
           nextflow run ${GITHUB_WORKSPACE} -profile test,docker --outdir ./results

CHANGELOG.md

@@ -3,14 +3,160 @@
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## v2.1.0dev - [date]
+## v2.1.0dev
 
-Initial release of nf-core/hic, created with the [nf-core](https://nf-co.re/) template.
+### `Added`
+
+- Added public_aws_ecr profile for using containers stored on ECR.
+
+### `Fixed`
+
+## v2.0.0 - 2023-01-12
 
 ### `Added`
 
+- DSL2 version of nf-core-hic pipeline
+- Add full test dataset (#80)
+- Replace local modules by the cooler nf-core module
+
+### `Fixed`
+
+- Fix error in the Arima preset (#127)
+
+## v1.3.1 - 2021-09-25
+
+### `Fixed`
+
+- Fix bug in conda environment for cooltools (#109)
+
+## v1.3.0 - 2021-05-22
+
+- Change the `/tmp/` folder to `./tmp/` folder so that all tmp files are now in the work directory (#24)
+- Add `--hicpro_maps` options to generate the raw and normalized HiC-Pro maps. The default is now to use cooler
+- Add chromosome compartments calling with cooltools (#53)
+- Add HiCExplorer distance decay quality control (#54)
+- Add HiCExplorer TADs calling (#55)
+- Add insulation score TADs calling (#55)
+- Generate cooler/txt contact maps
+- Normalize Hi-C data with cooler instead of iced
+- New `--digestion` parameter to automatically set the restriction_site and ligation_site motifs
+- New `--keep_multi` and `keep_dup` options. Default: false
+- Template update for nf-core/tools
+- Minor fix to summary log messages in pipeline header
+
 ### `Fixed`
 
-### `Dependencies`
+- Fix bug in stats report which were not all correcly exported in the results folder
+- Fix recurrent bug in input file extension (#86)
+- Fix bug in `--bin_size` parameter (#85)
+- `--min_mapq` is ignored if `--keep_multi` is used
 
 ### `Deprecated`
+
+- `--rm_dup` and `--rm_multi` are replaced by `--keep_dups` and `--keep_multi`
+
+## v1.2.2 - 2020-09-02
+
+### `Added`
+
+- Template update for nf-core/tools v1.10.2
+- Add the `--fastq_chunks_size` to specify the number of reads per chunks if split_fastq is true
+
+### `Fixed`
+
+- Bug in `--split_fastq` option not recognized
+
+## v1.2.1 - 2020-07-06
+
+### `Fixed`
+
+- Fix issue with `--fasta` option and `.fa` extension (#66)
+
+## v1.2.0 - 2020-06-18
+
+### `Added`
+
+- Bump v1.2.0
+- Merge template nf-core 1.9
+- Move some options to camel_case
+- Update python scripts for python3
+- Update conda environment file
+  - python base `2.7.15` > `3.7.6`
+  - pip `19.1` > `20.0.1`
+  - scipy `1.2.1` > `1.4.1`
+  - numpy `1.16.3` > `1.18.1`
+  - bx-python `0.8.2` > `0.8.8`
+  - pysam `0.15.2` > `0.15.4`
+  - cooler `0.8.5` > `0.8.6`
+  - multiqc `1.7` > `1.8`
+  - iced `0.5.1` > `0.5.6`
+  - _*New*_ pymdown-extensions `7.1`
+  - _*New*_ hicexplorer `3.4.3`
+  - _*New*_ bioconductor-hitc `1.32.0`
+  - _*New*_ r-optparse `1.6.6`
+  - _*New*_ ucsc-bedgraphtobigwig `377`
+  - _*New*_ cython `0.29.19`
+  - _*New*_ cooltools `0.3.2`
+  - _*New*_ fanc `0.8.30`
+  - _*Removed*_ r-markdown
+
+### `Fixed`
+
+- Fix error in doc for Arima kit usage
+- Sort output of `get_valid_interaction` process as the input files of `remove_duplicates`
+  are expected to be sorted (sort -m)
+
+### `Deprecated`
+
+- Command line options converted to `camel_case`:
+  - `--skipMaps` > `--skip_maps`
+  - `--skipIce` > `--skip_ice`
+  - `--skipCool` > `--skip_cool`
+  - `--skipMultiQC` > `--skip_multiqc`
+  - `--saveReference` > `--save_reference`
+  - `--saveAlignedIntermediates` > `--save_aligned_intermediates`
+  - `--saveInteractionBAM` > `--save_interaction_bam`
+
+## v1.1.1 - 2020-04-02
+
+### `Fixed`
+
+- Fix bug in tag. Remove '['
+
+## v1.1.0 - 2019-10-15
+
+### `Added`
+
+- Update hicpro2higlass with `-p` parameter
+- Support 'N' base motif in restriction/ligation sites
+- Support multiple restriction enzymes/ligattion sites (comma separated) ([#31](https://github.com/nf-core/hic/issues/31))
+- Add --saveInteractionBAM option
+- Add DOI ([#29](https://github.com/nf-core/hic/issues/29))
+- Update manual ([#28](https://github.com/nf-core/hic/issues/28))
+
+### `Fixed`
+
+- Fix bug for reads extension `_1`/`_2` ([#30](https://github.com/nf-core/hic/issues/30))
+
+## v1.0 - [2019-05-06]
+
+Initial release of nf-core/hic, created with the [nf-core](http://nf-co.re/) template.
+
+### `Added`
+
+First version of nf-core Hi-C pipeline which is a Nextflow implementation of
+the [HiC-Pro pipeline](https://github.com/nservant/HiC-Pro/).
+Note that all HiC-Pro functionalities are not yet all implemented.
+The current version supports most protocols including Hi-C, in situ Hi-C,
+DNase Hi-C, Micro-C, capture-C or HiChip data.
+
+In summary, this version allows :
+
+- Automatic detection and generation of annotation files based on igenomes
+  if not provided.
+- Two-steps alignment of raw sequencing reads
+- Reads filtering and detection of valid interaction products
+- Generation of raw contact matrices for a set of resolutions
+- Normalization of the contact maps using the ICE algorithm
+- Generation of cooler file for visualization on [higlass](https://higlass.io/)
+- Quality report based on HiC-Pro MultiQC module

CITATIONS.md

@@ -1,5 +1,9 @@
 # nf-core/hic: Citations
 
+## [HiC-Pro](https://genomebiology.biomedcentral.com/articles/10.1186/s13059-015-0831-x)
+
+> Servant N, Varoquaux N, Lajoie BR, Viara E, Chen C, Vert JP, Dekker J, Heard E, Barillot E. Genome Biology 2015, 16:259 doi: [10.1186/s13059-015-0831-x](https://dx.doi.org/10.1186/s13059-015-0831-x)
+
 ## [nf-core](https://pubmed.ncbi.nlm.nih.gov/32055031/)
 
 > Ewels PA, Peltzer A, Fillinger S, Patel H, Alneberg J, Wilm A, Garcia MU, Di Tommaso P, Nahnsen S. The nf-core framework for community-curated bioinformatics pipelines. Nat Biotechnol. 2020 Mar;38(3):276-278. doi: 10.1038/s41587-020-0439-x. PubMed PMID: 32055031.

README.md

@@ -1,6 +1,6 @@
 # ![nf-core/hic](docs/images/nf-core-hic_logo_light.png#gh-light-mode-only) ![nf-core/hic](docs/images/nf-core-hic_logo_dark.png#gh-dark-mode-only)
 
-[![AWS CI](https://img.shields.io/badge/CI%20tests-full%20size-FF9900?labelColor=000000&logo=Amazon%20AWS)](https://nf-co.re/hic/results)[![Cite with Zenodo](http://img.shields.io/badge/DOI-10.5281/zenodo.XXXXXXX-1073c8?labelColor=000000)](https://doi.org/10.5281/zenodo.XXXXXXX)
+[![AWS CI](https://img.shields.io/badge/CI%20tests-full%20size-FF9900?labelColor=000000&logo=Amazon%20AWS)](https://nf-co.re/hic/results)[![Cite with Zenodo](http://img.shields.io/badge/DOI-10.5281/zenodo.2669512-1073c8?labelColor=000000)](https://doi.org/10.5281/zenodo.2669512)
 
 [![Nextflow](https://img.shields.io/badge/nextflow%20DSL2-%E2%89%A522.10.1-23aa62.svg)](https://www.nextflow.io/)
 [![run with conda](http://img.shields.io/badge/run%20with-conda-3EB049?labelColor=000000&logo=anaconda)](https://docs.conda.io/en/latest/)
@@ -12,20 +12,29 @@
 
 ## Introduction
 
-**nf-core/hic** is a bioinformatics pipeline that ...
+**nf-core/hic** is a bioinformatics best-practice analysis pipeline for Analysis of Chromosome Conformation Capture data (Hi-C).
 
-<!-- TODO nf-core:
-   Complete this sentence with a 2-3 sentence summary of what types of data the pipeline ingests, a brief overview of the
-   major pipeline sections and the types of output it produces. You're giving an overview to someone new
-   to nf-core here, in 15-20 seconds. For an example, see https://github.com/nf-core/rnaseq/blob/master/README.md#introduction
--->
+The pipeline is built using [Nextflow](https://www.nextflow.io), a workflow tool to run tasks across multiple compute infrastructures in a very portable manner. It uses Docker/Singularity containers making installation trivial and results highly reproducible. The [Nextflow DSL2](https://www.nextflow.io/docs/latest/dsl2.html) implementation of this pipeline uses one container per process which makes it much easier to maintain and update software dependencies. Where possible, these processes have been submitted to and installed from [nf-core/modules](https://github.com/nf-core/modules) in order to make them available to all nf-core pipelines, and to everyone within the Nextflow community!
 
-<!-- TODO nf-core: Include a figure that guides the user through the major workflow steps. Many nf-core
-     workflows use the "tube map" design for that. See https://nf-co.re/docs/contributing/design_guidelines#examples for examples.   -->
-<!-- TODO nf-core: Fill in short bullet-pointed list of the default steps in the pipeline -->
+On release, automated continuous integration tests run the pipeline on a full-sized dataset on the AWS cloud infrastructure. This ensures that the pipeline runs on AWS, has sensible resource allocation defaults set to run on real-world datasets, and permits the persistent storage of results to benchmark between pipeline releases and other analysis sources.The results obtained from the full-sized test can be viewed on the [nf-core website](https://nf-co.re/hic/results).
+
+## Pipeline summary
 
 1. Read QC ([`FastQC`](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/))
-2. Present QC for raw reads ([`MultiQC`](http://multiqc.info/))
+2. Hi-C data processing
+   1. [`HiC-Pro`](https://github.com/nservant/HiC-Pro)
+      1. Mapping using a two steps strategy to rescue reads spanning the ligation
+         sites ([`bowtie2`](http://bowtie-bio.sourceforge.net/bowtie2/index.shtml))
+      2. Detection of valid interaction products
+      3. Duplicates removal
+      4. Generate raw and normalized contact maps ([`iced`](https://github.com/hiclib/iced))
+3. Create genome-wide contact maps at various resolutions ([`cooler`](https://github.com/open2c/cooler))
+4. Contact maps normalization using balancing algorithm ([`cooler`](https://github.com/open2c/cooler))
+5. Export to various contact maps formats ([`HiC-Pro`](https://github.com/nservant/HiC-Pro), [`cooler`](https://github.com/open2c/cooler))
+6. Quality controls ([`HiC-Pro`](https://github.com/nservant/HiC-Pro), [`HiCExplorer`](https://github.com/deeptools/HiCExplorer))
+7. Compartments calling ([`cooltools`](https://cooltools.readthedocs.io/en/latest/))
+8. TADs calling ([`HiCExplorer`](https://github.com/deeptools/HiCExplorer), [`cooltools`](https://cooltools.readthedocs.io/en/latest/))
+9. Quality control report ([`MultiQC`](https://multiqc.info/))
 
 ## Usage
 
@@ -34,30 +43,24 @@
 > to set-up Nextflow. Make sure to [test your setup](https://nf-co.re/docs/usage/introduction#how-to-run-a-pipeline)
 > with `-profile test` before running the workflow on actual data.
 
-<!-- TODO nf-core: Describe the minimum required steps to execute the pipeline, e.g. how to prepare samplesheets.
-     Explain what rows and columns represent. For instance (please edit as appropriate):
 
 First, prepare a samplesheet with your input data that looks as follows:
 
 `samplesheet.csv`:
 
 ```csv
 sample,fastq_1,fastq_2
-CONTROL_REP1,AEG588A1_S1_L002_R1_001.fastq.gz,AEG588A1_S1_L002_R2_001.fastq.gz
+HIC_ES_4,SRR5339783_1.fastq.gz,SRR5339783_2.fastq.gz
 ```
 
-Each row represents a fastq file (single-end) or a pair of fastq files (paired end).
-
--->
-
+Each row represents a pair of fastq files (paired end).
 Now, you can run the pipeline using:
 
-<!-- TODO nf-core: update the following command to include all required parameters for a minimal example -->
-
 ```bash
 nextflow run nf-core/hic \
    -profile <docker/singularity/.../institute> \
    --input samplesheet.csv \
+   --genome GRCh37 \
    --outdir <OUTDIR>
 ```
 
@@ -78,10 +81,6 @@ For more details about the output files and reports, please refer to the
 
 nf-core/hic was originally written by Nicolas Servant.
 
-We thank the following people for their extensive assistance in the development of this pipeline:
-
-<!-- TODO nf-core: If applicable, make list of people who have also contributed -->
-
 ## Contributions and Support
 
 If you would like to contribute to this pipeline, please see the [contributing guidelines](.github/CONTRIBUTING.md).
@@ -90,10 +89,7 @@ For further information or help, don't hesitate to get in touch on the [Slack `#
 
 ## Citations
 
-<!-- TODO nf-core: Add citation for pipeline after first release. Uncomment lines below and update Zenodo doi and badge at the top of this file. -->
-<!-- If you use  nf-core/hic for your analysis, please cite it using the following doi: [10.5281/zenodo.XXXXXX](https://doi.org/10.5281/zenodo.XXXXXX) -->
-
-<!-- TODO nf-core: Add bibliography of tools and data used in your pipeline -->
+If you use nf-core/hic for your analysis, please cite it using the following doi: doi: [10.5281/zenodo.2669512](https://doi.org/10.5281/zenodo.2669512)
 
 An extensive list of references for the tools used by the pipeline can be found in the [`CITATIONS.md`](CITATIONS.md) file.
 

assets/methods_description_template.yml

@@ -3,7 +3,7 @@ description: "Suggested text and references to use when describing pipeline usag
 section_name: "nf-core/hic Methods Description"
 section_href: "https://github.com/nf-core/hic"
 plot_type: "html"
-## TODO nf-core: Update the HTML below to your prefered methods description, e.g. add publication citation for this pipeline
+## nf-core: Update the HTML below to your prefered methods description, e.g. add publication citation for this pipeline
 ## You inject any metadata in the Nextflow '${workflow}' object
 data: |
   <h4>Methods</h4>
@@ -12,6 +12,7 @@ data: |
   <pre><code>${workflow.commandLine}</code></pre>
   <h4>References</h4>
   <ul>
+    <li>Servant, N., Ewels, P. A., Peltzer, A., Garcia, M. U. (2021) nf-core/hic. Zenodo. https://doi.org/10.5281/zenodo.2669512</li>
     <li>Di Tommaso, P., Chatzou, M., Floden, E. W., Barja, P. P., Palumbo, E., & Notredame, C. (2017). Nextflow enables reproducible computational workflows. Nature Biotechnology, 35(4), 316-319. <a href="https://doi.org/10.1038/nbt.3820">https://doi.org/10.1038/nbt.3820</a></li>
     <li>Ewels, P. A., Peltzer, A., Fillinger, S., Patel, H., Alneberg, J., Wilm, A., Garcia, M. U., Di Tommaso, P., & Nahnsen, S. (2020). The nf-core framework for community-curated bioinformatics pipelines. Nature Biotechnology, 38(3), 276-278. <a href="https://doi.org/10.1038/s41587-020-0439-x">https://doi.org/10.1038/s41587-020-0439-x</a></li>
   </ul>

assets/samplesheet.csv

@@ -1,3 +1,2 @@
 sample,fastq_1,fastq_2
-SAMPLE_PAIRED_END,/path/to/fastq/files/AEG588A1_S1_L002_R1_001.fastq.gz,/path/to/fastq/files/AEG588A1_S1_L002_R2_001.fastq.gz
-SAMPLE_SINGLE_END,/path/to/fastq/files/AEG588A4_S4_L003_R1_001.fastq.gz,
+SRR4292758,https://github.com/nf-core/test-datasets/raw/hic/data/SRR4292758_00_R1.fastq.gz,https://github.com/nf-core/test-datasets/raw/hic/data/SRR4292758_00_R2.fastq.gz