Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .github/workflows/deploy.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
name: deploy
on:
push:
branches:
- master
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
with:
python-version: 3.x
- run: pip install mkdocs-material
- run: mkdocs gh-deploy --force
13 changes: 13 additions & 0 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
name: test
on: push
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-java@v3
with:
distribution: 'temurin'
java-version: '17'
- uses: nf-core/setup-nextflow@v1
- run: bash scripts/test.sh
9 changes: 5 additions & 4 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
.git/
.nextflow*
.git
.idea*
work/
.nextflow*
my-results
public
results
my-results
work
11 changes: 0 additions & 11 deletions .travis.yml

This file was deleted.

56 changes: 28 additions & 28 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,46 +2,46 @@

A curated collection of Nextflow implementation patterns

[![Build Status](https://travis-ci.org/nextflow-io/patterns.svg?branch=master)](https://travis-ci.org/nextflow-io/patterns)
![Build Status](https://github.com/nextflow-io/patterns/actions/workflows/test.yml/badge.svg)

## Basic patterns

* [Channel duplication](docs/channel-duplication.adoc)
* [Channel duplication](docs/channel-duplication.md)

### Scatter executions
## Scatter executions

* [Process per file path](docs/process-per-file-path.adoc)
* [Process per file chunk](docs/process-per-file-chunk.adoc)
* [Process per file pairs](docs/process-per-file-pairs.adoc)
* [Process per file range](docs/process-per-file-range.adoc)
* [Process per CSV record](docs/process-per-csv-record.adoc)
* [Process per file output](docs/process-per-file-output.adoc)
* [Process per file path](docs/process-per-file-path.md)
* [Process per file chunk](docs/process-per-file-chunk.md)
* [Process per file pairs](docs/process-per-file-pairs.md)
* [Process per file range](docs/process-per-file-range.md)
* [Process per CSV record](docs/process-per-csv-record.md)
* [Process per file output](docs/process-per-file-output.md)

### Gather results
## Gather results

* [Process all outputs altogether](docs/process-collect.adoc)
* [Process outputs into groups](docs/process-into-groups.adoc)
* [Collect outputs into a file](docs/collect-into-file.adoc)
* [Process all outputs altogether](docs/process-collect.md)
* [Process outputs into groups](docs/process-into-groups.md)
* [Collect outputs into a file](docs/collect-into-file.md)

### Organize outputs
## Organize outputs

* [Store process outputs](docs/publish-process-outputs.adoc)
* [Store outputs matching a glob pattern](docs/publish-matching-glob.adoc)
* [Store outputs renaming files](docs/publish-rename-outputs.adoc)
* [Store process outputs](docs/publish-process-outputs.md)
* [Store outputs matching a glob pattern](docs/publish-matching-glob.md)
* [Store outputs renaming files](docs/publish-rename-outputs.md)

## Other

* [Get process work directory](docs/process-get-workdir.adoc)
* [Ignore failing process](docs/ignore-failing-process.adoc)
* [State dependency](docs/state-dependency.adoc)
* [Get process work directory](docs/process-get-workdir.md)
* [Ignore failing process](docs/ignore-failing-process.md)
* [State dependency](docs/state-dependency.md)

## Advanced patterns

* [Conditional process resources](docs/conditional-resources.adoc)
* [Conditional process executions](docs/conditional-process.adoc)
* [Skip process execution](docs/skip-process-execution.adoc)
* [Feedback loop](docs/feedback-loop.adoc)
* [Optional input](docs/optional-input.adoc)
* [Optional output](docs/optional-output.adoc)
* [Process when empty](docs/process-when-empty.adoc)
* [Task batching](docs/task-batching.adoc)
* [Conditional process resources](docs/conditional-resources.md)
* [Conditional process executions](docs/conditional-process.md)
* [Skip process execution](docs/skip-process-execution.md)
* [Feedback loop](docs/feedback-loop.md)
* [Optional input](docs/optional-input.md)
* [Optional output](docs/optional-output.md)
* [Process when empty](docs/process-when-empty.md)
* [Task batching](docs/task-batching.md)
2 changes: 0 additions & 2 deletions docs/Makefile

This file was deleted.

19 changes: 8 additions & 11 deletions docs/channel-duplication.adoc → docs/channel-duplication.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,14 @@
= Channel duplication

== Problem
## Problem

You need to you use the same channel as input in two or more processes.

== Solution
## Solution

In DSL2, you can just do it! The https://www.nextflow.io/docs/latest/operator.html#into[into] operator is no longer needed.
In DSL2, you can just do it! The [into](https://www.nextflow.io/docs/latest/operator.html#into) operator is no longer needed.

== Code
## Code

[source,nextflow,linenums,options="nowrap"]
----
```groovy
process foo {
input: path x
script:
Expand All @@ -34,12 +31,12 @@ workflow {
foo(input_ch)
bar(input_ch)
}
----
```

== Run it
## Run it

Use the the following command to execute the example:

```
```bash
nextflow run patterns/channel-duplication.nf
```
19 changes: 8 additions & 11 deletions docs/collect-into-file.adoc → docs/collect-into-file.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,15 @@
= Collect outputs into a file

== Problem
## Problem

You need to concatenate into a single file all output files produced by an upstream process.

== Solution
## Solution

Use the https://www.nextflow.io/docs/latest/operator.html#collectfile[collectFile] operator to merge all
Use the [collectFile](https://www.nextflow.io/docs/latest/operator.html#collectfile) operator to merge all
the output files into a single file.

== Code
## Code

[source,nextflow,linenums,options="nowrap"]
----
```groovy
process foo {
input:
path x
Expand All @@ -30,12 +27,12 @@ workflow {
| collectFile \
| view
}
----
```

== Run it
## Run it

Use the the following command to execute the example:

```
```bash
nextflow run patterns/collect-into-file.nf
```
36 changes: 16 additions & 20 deletions docs/conditional-process.adoc → docs/conditional-process.md
Original file line number Diff line number Diff line change
@@ -1,22 +1,19 @@
= Conditional process executions

== Problem
## Problem

One of two different tasks should be executed based on some condition,
and a third task should process the results of the selected task.

== Solution
## Solution

Simply execute either process using `if/else` statements on the condition.
Define a channel, e.g. `omega_ch`, which emits the output of the selected process
in each case. Then, execute the third process with this output channel.

Or, use a ternary expression and a pipe to keep things short and sweet.

== Code
## Code

[source,nextflow,linenums,options="nowrap"]
----
```groovy
params.flag = false

process foo {
Expand Down Expand Up @@ -66,39 +63,38 @@ workflow {
// the short way
(params.flag ? bar : foo) | omega
}
----
```

== Run it
## Run it

Use the the following command to execute the example:

```
```bash
nextflow run patterns/conditional-process.nf
```

The processes `foo` and `omega` are executed. Run the same command
with the `--flag` command line option.

```
```bash
nextflow run patterns/conditional-process.nf --flag
```

This time the processes `bar` and `omega` are executed.

== Alternative solution
## Alternative solution

Create an input channel for each process that is either populated with data or an
https://www.nextflow.io/docs/latest/channel.html#empty[empty] channel.
[empty](https://www.nextflow.io/docs/latest/channel.html#empty) channel.
Each process will execute only if its input channel has data.

Then use the https://www.nextflow.io/docs/latest/operator.html#mix[mix] operator to create
Then use the [mix](https://www.nextflow.io/docs/latest/operator.html#mix) operator to create
a new channel that emits the outputs produced by the two processes, and use it as the input
for the third process.

== Code
## Code

[source,nextflow,linenums,options="nowrap"]
----
```groovy
params.flag = false

process foo {
Expand Down Expand Up @@ -148,10 +144,10 @@ workflow {

foo.out | mix(bar.out) | omega
}
----
```

== Run it
## Run it

```
```bash
nextflow run patterns/conditional-process2.nf
```
17 changes: 7 additions & 10 deletions docs/conditional-resources.adoc → docs/conditional-resources.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,10 @@
= Conditional resources definition

== Problem
## Problem

A task in your workflow needs to use some amount of computing
resources (e.g. memory) that depends on the size or the name of one
or more input files.

== Solution
## Solution

Declare the resource requirements (`memory`, `cpus`, etc.)
in a dynamic manner using a closure.
Expand All @@ -15,10 +13,9 @@ The closure computes the required amount of resources using the file
attributes (e.g. `size`) of the inputs declared in the process
definition.

== Code
## Code

[source,nextflow,linenums,options="nowrap"]
----
```groovy
process foo {
memory { reads.size() < 70.KB ? 1.GB : 5.GB }

Expand All @@ -34,10 +31,10 @@ workflow {
Channel.fromPath("$baseDir/data/reads/*_1.fq.gz", checkIfExists:true) \
| foo
}
----
```

== Run it
## Run it

```
```bash
nextflow run patterns/conditional-resources.nf
```
Loading