ngs-preprocess pipeline

A pipeline for preprocessing short and long sequencing reads

About

ngs-preprocess is built using Nextflow, a workflow tool to run tasks across multiple compute infrastructures in a very portable manner. It uses Docker/Singularity containers making installation trivial and results highly reproducible. It is an easy to use pipeline that uses state-of-the-art software for quality check and pre-processing ngs reads of Illumina, Pacbio and Oxford Nanopore Technologies.

It wraps up the following software:

| Step | tools | | :--- | :---- | | SRA NBCI fetch | Entrez-direct & sra-tools | | Illumina pre-processing | Fastp | | Nanopore pre-processing | Porechop, Porechop ABI, pycoQC, NanoPack | | Pacbio pre-processing | bam2fastx, bax2bam, lima, pacbio ccs |

Further reading

This pipeline has two complementary pipelines (also written in nextflow) for genome assembly and prokaryotic genome annotation that can give the user a complete workflow for bacterial genomics analyses.

Quickstart

1. Install Nextflow:

   bash curl -s https://get.nextflow.io | bash

2. Give it a try:

   bash nextflow run fmalmeida/ngs-preprocess --help

3. Download required tools

* for docker

    ```bash
    # for docker
    docker pull fmalmeida/ngs-preprocess:v2.7

    # run
    nextflow run fmalmeida/ngs-preprocess -profile docker [options]
    ```

* for singularity

    ```bash
    # for singularity
    # remember to properly set NXF_SINGULARITY_LIBRARYDIR
    # read more at https://www.nextflow.io/docs/latest/singularity.html#singularity-docker-hub
    export NXF_SINGULARITY_LIBRARYDIR=MY_SINGULARITY_IMAGES    # your singularity storage dir
    export NXF_SINGULARITY_CACHEDIR=MY_SINGULARITY_CACHE       # your singularity cache dir
    singularity pull \
        --dir $NXF_SINGULARITY_LIBRARYDIR \
        fmalmeida-ngs-preprocess-v2.7.img docker://fmalmeida/ngs-preprocess:v2.7

    # run
    nextflow run fmalmeida/ngs-preprocess -profile singularity [options]
    ```

* for conda

    ```bash
    # for conda
    # it is better to create envs with mamba for faster solving
    wget https://github.com/fmalmeida/ngs-preprocess/raw/master/environment.yml
    conda env create -f environment.yml   # advice: use mamba

    # must be executed from the base environment
    # This tells nextflow to load the available ngs-preprocess environment when required
    nextflow run fmalmeida/ngs-preprocess -profile conda [options]
    ```

1. Start running your analysis

   bash nextflow run fmalmeida/ngs-preprocess -profile <docker/singularity/conda>

:fire: Please read the documentation below on selecting between conda, docker or singularity profiles, since the tools will be made available differently depending on the profile desired.

Documentation

Selecting between profiles

Nextflow profiles are a set of "sensible defaults" for the resource requirements of each of the steps in the workflow, that can be enabled with the command line flag -profile. You can learn more about nextflow profiles at:

• https://nf-co.re/usage/configuration#basic-configuration-profiles
• https://www.nextflow.io/docs/latest/config.html#config-profiles

The pipeline have "standard profiles" set to run the workflows with either conda, docker or singularity using the local executor, which is nextflow's default and basically runs the pipeline processes in the computer where Nextflow is launched. If you need to run the pipeline using another executor such as sge, lsf, slurm, etc. you can take a look at nextflow's manual page to proper configure one in a new custom profile set in your personal copy of ngs-preprocess config file and take advantage that nextflow allows multiple profiles to be used at once, e.g. -profile conda,sge.

By default, if no profile is chosen, the pipeline will try to load tools from the local machine $PATH. Available pre-set profiles for this pipeline are: docker/conda/singularity, you can choose between them as follows:

• conda

  ```bash

  must be executed from the base environment

  This tells nextflow to load the available ngs-preprocess environment when required

  nextflow run fmalmeida/ngs-preprocess -profile conda [options] ```

• docker

  bash nextflow run fmalmeida/ngs-preprocess -profile docker [options]

• singularity

  bash nextflow run fmalmeida/ngs-preprocess -profile singularity [options]

:book: Please use conda as last resource since the packages will not be "frozen and pre-installed", problems may arise.

Usage

For understading pipeline usage and configuration, users must read the complete online documentation »

Using a configuration file

All the parameters showed above can be, and are advised to be, set through the configuration file. When a configuration file is set the pipeline is run by simply executing:

bash nextflow run fmalmeida/ngs-preprocess -c ./configuration-file

Your configuration file is what will tell to the pipeline the type of data you have, and which processes to execute. Therefore, it needs to be correctly set up.

Create a configuration file in your working directory:

bash nextflow run fmalmeida/ngs-preprocess [ --get_config ]

Interactive graphical configuration and execution

Via NF tower launchpad (good for cloud env execution)

Nextflow has an awesome feature called NF tower. It allows that users quickly customise and set-up the execution and configuration of cloud enviroments to execute any nextflow pipeline from nf-core, github (this one included), bitbucket, etc. By having a compliant JSON schema for pipeline configuration it means that the configuration of parameters in NF tower will be easier because the system will render an input form.

Checkout more about this feature at: https://seqera.io/blog/orgs-and-launchpad/

Via nf-core launch (good for local execution)

Users can trigger a graphical and interactive pipeline configuration and execution by using nf-core launch utility. nf-core launch will start an interactive form in your web browser or command line so you can configure the pipeline step by step and start the execution of the pipeline in the end.

```bash

Install nf-core

pip install nf-core

Launch the pipeline

nf-core launch fmalmeida/ngs-preprocess ```

It will result in the following:

Citation

In order to cite this pipeline, please refer to:

Almeida FMd, Campos TAd and Pappas Jr GJ. Scalable and versatile container-based pipelines for de novo genome assembly and bacterial annotation. F1000Research 2023, 12:1205 (https://doi.org/10.12688/f1000research.139488.1)

Additionally, archived versions of the pipeline are also found in Zenodo.

This pipeline uses code and infrastructure developed and maintained by the nf-core community, reused here under the GPLv3.

The nf-core framework for community-curated bioinformatics pipelines.

Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen.

Nat Biotechnol. 2020 Feb 13. doi: 10.1038/s41587-020-0439-x.

In addition, users are encouraged to cite the programs used in this pipeline whenever they are used. Links to resources of tools and data used in this pipeline are as follows:

• Entrez-direct
• sra-tools
• Fastp
• Porechop
• Porechop ABI
• pycoQC
• bax2bam
• bam2fastq
• lima
• pacbio ccs
• NanoPack.