Fetch Public Sequencing Data and Metadata Using iSeq

A Snakemake 8 workflow to fetch (download) and process public sequencing data and metadata from GSA, SRA, ENA, GEO and DDBJ databases using iSeq.

[!NOTE]
This workflow adheres to the module specifications of MrBiomics, an effort to augment research by modularizing (biomedical) data science. For more details, instructions, and modules check out the project's repository.

⭐️ Star and share modules you find valuable 📤 - help others discover them, and guide our future work!

[!IMPORTANT]
If you use this workflow in a publication, please don't forget to give credit to the authors by citing it using this DOI 10.5281/zenodo.15005419.

🖋️ Authors

• Stephan Reichl
• Christoph Bock

💿 Software

This project wouldn't be possible without the following software and their dependencies.

| Software | Reference (DOI) | | :---: | :---: | | iSeq | https://github.com/BioOmics/iSeq | | pandas | https://doi.org/10.5281/zenodo.3509134 | | Picard | https://broadinstitute.github.io/picard/ | | Snakemake | https://doi.org/10.12688/f1000research.29032.2 |

🔬 Methods

This is a template for the Methods section of a scientific publication and is intended to serve as a starting point. Only retain paragraphs relevant to your analysis. References [ref] to the respective publications are curated in the software table above. Versions (ver) have to be read out from the respective conda environment specifications (workflow/envs/*.yaml file) or post-execution in the result directory ({module}/envs/*.yaml). Parameters that have to be adapted depending on the data or workflow configurations are denoted in squared brackets e.g., [X].

Data Acquisition & Processing. Public sequencing data were retrieved from [GSA|SRA|ENA|DDBJ] under the accession(s) [accession_ids] using iSeq (ver) [ref]. The data were downloaded as FASTQ files (and converted to unmapped BAM (uBAM) files using Picard FastqToSam (ver) [ref], preserving sample information and read groups while supporting both single-end and paired-end sequencing data). Metadata for each dataset was collected and merged into a single Comprehensive reference file.

The data acquisition and processing described here were performed using a publicly available Snakemake (ver) [ref] workflow 10.5281/zenodo.15005419.

🚀 Features

The workflow performs the following steps that produce the outlined results:

• Data Acquisition
  • Downloads sequencing data from public repositories GSA, SRA, ENA, and DDBJ using various accession ID types
  • Extracts comprehensive metadata for each dataset
  • Supports parallel downloading for improved performance using threads
• Data Processing
  • Automatic handling of both single-end and paired-end sequencing data
  • Creation of a unified comprehensive metadata file with accession IDs and file paths
  • Optional conversion from FASTQ (as *.fastq.gz) to unmapped BAM(as *.bam) format using Picard's FastqToSam
• Metadata-only mode for quick exploration without downloading sequence files (metadata_only: 1)
• Considerations
  • Dependent on iSeq's supported repositories and accession types
  • Requires internet connectivity and sufficient storage space for downloaded data

The workflow produces the following directory structure:

{result_path}/ └── fetch_ngs/ ├── metadata.csv # merged metadata for all accessions ├── .fastq_to_bam/ # processing marker files │ └── [accession].done └── [accession]/ # one directory per accession ├── [accession].metadata.csv # metadata for this accession └── [sample].[bam/fastq.gz] # sequence files

🛠️ Usage

Here are some tips for the usage of this workflow: - Run your workflow with snakemake --resources parallel_downloads=3 to restrict concurrent download jobs to three (worked well for me), thereby reducing the risk of triggering IP blacklisting from excessive parallel FTP connections. This can also be achieved by using the workflow's profile. In case of usage as module, put the parameter into the parent workflow's profile. - Specify accession IDs in the configuration file as a list to download multiple datasets in one run - Use metadata_only: 1 for a quick preview of available data before committing to full downloads - Choose between FASTQ or BAM output formats based on your downstream analysis needs - For large datasets, consider increasing threads and mem parameters - For super series (e.g., GSE) or projects containing many samples, start by running in metadata_only: 1 mode to extract run accession IDs. Then use these IDs in the config to enable maximum parallelization, avoiding sequential download and conversion. - The merged metadata file can be used as a basis for sample annotation files downstream - BAM output format (output_format: bam) is recommended for direct integration with BAM compatible downstream analysis workflows

⚙️ Configuration

Detailed specifications can be found here ./config/README.md

📖 Examples

Explore detailed examples showcasing module usage in our comprehensive end-to-end MrBiomics Recipes, including data, configuration, annotation and results: - ATAC-seq Analysis Recipe - RNA-seq Analysis Recipe

🔗 Links

• GitHub Repository
• GitHub Page
• Zenodo Repository
• Snakemake Workflow Catalog Entry

📚 Resources

• Recommended compatible MrBiomics Modules for downstream analyses:
  • ATAC-seq Data Processing & Quantification Pipeline for processing, quantification and annotation of chromatin accessibility.
  • Genome Browser Track Visualization for quality control and visual inspection/analysis of genomic regions/genes of interest or top hits.
  • Split, Filter, Normalize and Integrate Sequencing Data after count quantification.
  • Differential Analysis with limma to identify and visualize statistically significantly different features (e.g., genes or genomic regions) between sample groups.
  • Enrichment Analysis for biomedical interpretation of (differential) analysis results using prior knowledge.
  • Unsupervised Analysis to understand and visualize similarities and variations between cells/samples, including dimensionality reduction and cluster analysis. Useful for all tabular data including single-cell and bulk sequencing data.

📑 Publications

The following publications successfully used this module for their analyses. - FirstAuthors et al. (202X) Journal Name - Paper Title. - ...

⭐ Star History