Copyright © 2023-2025 SDSC - Swiss Data Science Center.
Licensed under the MIT License - see the LICENSE file for details.
Funded by SPHN and PHRT.

Context

This repository includes "digital infrastructure" code from a Swiss National Data Stream (NDS): LUCID. The general goal of the NDS initiative is to collect clinical data across five Swiss University Hospitals and share it with researchers. In the case of LUCID, research focuses on low-value care: services that provide little or no benefit to patients. If you're interested, check the official project page.

Digital infrastructure for LUCID project is also available in these repositories.

Code overview

This repository provides an automated pipeline to perform (RDF) data validation with two flows:

• Success: upon successful validation, data is provided in the output folder
• Failure: upon unsuccessful validation, a report is generated in the notification folder

The code was first built around the BioMedIT environment, but to allow re-usability, most software and tools rely on public containers, meaning that there are few requirements to test it on any machine (see Requirements section).

The sections below provide more technical details about the pipeline, its implementation and use.
For any question, feel free to open an issue or contact us directly.

Workflow framework

The pipeline is built using Nextflow, a workflow tool to run tasks across multiple compute infrastructures in a very portable manner. It uses podman containers making installation trivial and results highly reproducible. The Nextflow DSL2 implementation of this pipeline uses one container per process which makes it much easier to maintain and update software dependencies.

Requirements

By default, the workflow assumes that: * Nextflow is installed (>=22.10.1) * Podman is installed for full pipeline reproducibility (tested on version 4.9.4) * Basic UNIX libraries are installed: gzip, cat, unzip, md5sum and fdfind

With the biomedit profile, in addition to the points above, the workflow assumes that: * sett-rs is installed with the command-line interface available (sett-cli, tested on version 5.3.0) * A Nextflow secret SETT_OPENPGP_KEY_PWD is set to provide the secret OpenPGP key used to decrypt data * jq is installed and available (>=1.6)

See usage instructions for more information.

Pipeline summary

1. Check for new zip files or rerun worfklow on current zip files in source directory
2. Decrypt and decompress files to extract datasets
   • With the biomedit profile, metadata is extracted and used to rename the datasets' directory
3. Create batches of data with sizes defined in nextflow.config
4. One by one, each data batch is bundled with external terminologies and validated using SPHN SHACL rules:
   a. For invalid or empty batches: create a file with datasets and their error in a notification folder
   b. For valid batches: continue to steps 5-7
5. Convert valid datasets to nt format
6. Compress all nt files into gzip
7. Move gzip to output directory

mermaid flowchart TD input_dir(input_dir) sett_unpack[sett_unpack] patient_data(patient_data) bundled_batches(bundled_batches) val[validation] report(report.ttl) exit(exitStatus) copy[copy_to_output] compress[compress] output_dir[output_dir] notification[send_notification] input_dir -->|ch_sett_pkg| check_integrity check_integrity --> get_sett_metadata check_integrity --> unpack unpack --> patient_data subgraph biomedit get_sett_metadata --> sett_unpack end subgraph config input_dir output_dir SPHN_SHACL_shapes SPHN_schema terminologies end output_dir --> sett_unpack output_dir --> unpack sett_unpack --> patient_data patient_data --> |batching| bundled_batches SPHN_SHACL_shapes --> val terminologies --> |nt converter| enriched_terms SPHN_schema --> |nt_converter| enriched_terms enriched_terms --> bundled_batches val --> report val --> exit exit --> |if !=0| notification exit --> |else| nt_converter nt_converter --> compress compress --> copy bundled_batches --> |if empty| notification bundled_batches --> val

Quick Start

See usage docs for all of the available options when running the pipeline.

1. Download the pipeline and test it on a minimal dataset with a single command:

bash nextflow run main.nf -profile standard,test

Note that some form of configuration will be needed so that Nextflow knows how to fetch the required software. This is usually done in the form of a config profile (test in the example command above). You can chain multiple config profiles in a comma-separated string.

1. Start running your own analysis!

bash nextflow run main.nf -profile standard,test --input_dir /data/source --output_dir /data/target --notification_dir /data/notification --shapes /data/shapes

Production use

To use ingestion within the BioMedIT system, we advise pointing the Nextflow working directory to a folder on a separate partition with sufficient volume and appropriate permissions. For the pipeline constantly monitoring for incoming data:

nextflow run main.nf -profile biomedit -w /data/work/ For the pipeline to re-run on already landed data:

nextflow run main.nf -profile biomedit -w /data/work/ --rerun=true

Credits

nds-lucid/ingestion was originally written by Stefan Milosavljevic and Cyril Matthey-Doret.

Cite this work by getting citation information from the GitHub menu on the right or the Zenodo DOI record, or like below (APA style citation):

Milosavljevic, S., Matthey-Doret, C., & Riba Grognuz, O. (2025). LUCID BioMedIT Ingestion Pipeline. Zenodo. https://doi.org/10.5281/zenodo.14726408

References

This pipeline uses code developed and maintained by the nf-core community, reused here under the MIT license.

The nf-core framework for community-curated bioinformatics pipelines.

Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen.

Nat Biotechnol. 2020 Feb 13. doi: 10.1038/s41587-020-0439-x.