Data

Tools

Indexes

Applications

Experiments

Open Source Science

drugresponseeval

Pipeline for testing drug response prediction models in a statistically and biologically sound way.

https://github.com/nf-core/drugresponseeval

Science Score: 57.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ✓
    CITATION.cff file
    Found CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ✓
    DOI references
    Found 16 DOI reference(s) in README
  • ○
    Academic publication links
  • ○
    Academic email domains
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (10.8%) to scientific vocabulary

Keywords

bioinformatics cell-lines cross-validation deep-learning drug-response drug-response-prediction drugs fair-principles generalization hyperparameter-tuning machine-learning nextflow nf-core pipeline randomization-tests reproducible-research robustness-assessment training workflow
Last synced: 6 months ago · JSON representation ·

Repository

Pipeline for testing drug response prediction models in a statistically and biologically sound way.

Basic Info
Statistics
  • Stars: 20
  • Watchers: 183
  • Forks: 4
  • Open Issues: 9
  • Releases: 2
Topics
bioinformatics cell-lines cross-validation deep-learning drug-response drug-response-prediction drugs fair-principles generalization hyperparameter-tuning machine-learning nextflow nf-core pipeline randomization-tests reproducible-research robustness-assessment training workflow
Created over 2 years ago · Last pushed 8 months ago
Metadata Files
Readme Changelog Contributing License Code of conduct Citation

README.md

nf-core/drugresponseeval

GitHub Actions CI Status GitHub Actions Linting StatusAWS CICite with Zenodo

nf-test

Nextflow nf-core template version run with conda run with docker run with singularity Launch on Seqera Platform

Get help on SlackFollow on BlueskyFollow on MastodonWatch on YouTube

Introduction

drevalpy_summary

DrEval is a bioinformatics framework that includes a PyPI package (drevalpy) and a Nextflow pipeline (this repo). DrEval ensures that evaluations are statistically sound, biologically meaningful, and reproducible. DrEval simplifies the implementation of drug response prediction models, allowing researchers to focus on advancing their modeling innovations by automating standardized evaluation protocols and preprocessing workflows. With DrEval, hyperparameter tuning is fair and consistent. With its flexible model interface, DrEval supports any model type, ranging from statistical models to complex neural networks. By contributing your model to the DrEval catalog, you can increase your work's exposure, reusability, and transferability.

  1. The response data is loaded
  2. All models are trained and evaluated in a cross-validation setting
  3. For each CV split, the best hyperparameters are determined using a grid search per model
  4. The model is trained on the full training set (train & validation) with the best hyperparameters to predict the test set
  5. If randomization tests are enabled, the model is trained on the full training set with the best hyperparameters to predict the randomized test set
  6. If robustness tests are enabled, the model is trained N times on the full training set with the best hyperparameters
  7. Plots are created summarizing the results

For baseline models, no randomization or robustness tests are performed.

Usage

[!NOTE] If you are new to Nextflow and nf-core, please refer to this page on how to set-up Nextflow. Make sure to test your setup with -profile test before running the workflow on actual data.

Now, you can run the pipeline using:

bash nextflow run nf-core/drugresponseeval \ -profile <docker/singularity/.../institute> \ --models <RandomForest,model2,...> \ --baselines <NaiveMeanEffectsPredictor,baseline2,...> \ --dataset_name <CTRPv2|CTRPv1|CCLE|GDSC1|GDSC2|custom_dataset>

[!WARNING] Please provide pipeline parameters via the CLI or Nextflow -params-file option. Custom config files including those provided by the -c Nextflow option can be used to provide any configuration except for parameters; see docs.

For more details and further functionality, please refer to the usage documentation and the parameter documentation.

Pipeline output

To see the results of an example test run with a full size dataset refer to the results tab on the nf-core website pipeline page. For more details about the output files and reports, please refer to the output documentation.

Credits

nf-core/drugresponseeval was originally written by Judith Bernett (TUM) and Pascal Iversen (FU Berlin).

We thank the following people for their extensive assistance in the development of this pipeline:

Contributions and Support

Contributors to nf-core/drugresponseeval and the drevalpy PyPI package:

If you would like to contribute to this pipeline, please see the contributing guidelines.

For further information or help, don't hesitate to get in touch on the Slack #drugresponseeval channel (you can join with this invite).

Citations

If you use nf-core/drugresponseeval for your analysis, please cite it using the following doi: 10.5281/zenodo.14779984

Our corresponding publication is at doi 10.1101/2025.05.26.655288

Bernett, J., Iversen, P., Picciani, M., Wilhelm, M., Baum, K., & List, M. From Hype to Health Check: Critical Evaluation of Drug Response Prediction Models with DrEval.

bioRxiv, 2025-05.

The underlying data is available at doi: 10.5281/zenodo.12633909.

The underlying python package is drevalpy, availably on PyPI as standalone, for which we also have an extensive ReadTheDocs Documentation.

An extensive list of references for the tools used by the pipeline can be found in the CITATIONS.md file.

You can cite the nf-core publication as follows:

The nf-core framework for community-curated bioinformatics pipelines.

Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen.

Nat Biotechnol. 2020 Feb 13. doi: 10.1038/s41587-020-0439-x.

Owner

  • Name: nf-core
  • Login: nf-core
  • Kind: organization
  • Email: core@nf-co.re

A community effort to collect a curated set of analysis pipelines built using Nextflow.

Citation (CITATIONS.md) 

# nf-core/drugresponseeval: Citations

## [DrugResponseEval](https://github.com/nf-core/drugresponseeval/)

> Bernett, J., Iversen, P., Picciani, M., Wilhelm, M., Baum, K., & List, M. **From Hype to Health Check: Critical Evaluation of Drug Response Prediction Models with DrEval.** > [10.1101/2025.05.26.655288](https://doi.org/10.1101/2025.05.26.655288) > _bioRxiv_, 2025-05.

## [nf-core](https://pubmed.ncbi.nlm.nih.gov/32055031/)

> Ewels PA, Peltzer A, Fillinger S, Patel H, Alneberg J, Wilm A, Garcia MU, Di Tommaso P, Nahnsen S. The nf-core framework for community-curated bioinformatics pipelines. Nat Biotechnol. 2020 Mar;38(3):276-278. doi: 10.1038/s41587-020-0439-x. PubMed PMID: 32055031.

## [Nextflow](https://pubmed.ncbi.nlm.nih.gov/28398311/)

> Di Tommaso P, Chatzou M, Floden EW, Barja PP, Palumbo E, Notredame C. Nextflow enables reproducible computational workflows. Nat Biotechnol. 2017 Apr 11;35(4):316-319. doi: 10.1038/nbt.3820. PubMed PMID: 28398311.

## Pipeline tools

- [DrEvalPy](https://github.com/daisybio/drevalpy): The pipeline mostly automates the individual steps of the DrEvalPy PyPI package.

  > Bernett, J, Iversen, P, Picciani, M, Wilhelm, M, Baum, K, List, M. Will be published soon.

- [CurveCurator](https://www.nature.com/articles/s41467-023-43696-z): For custom curve fitting on custom datasets. We also used it to re-process the response curves of GDSC1, GDSC2, CCLE, and CTRP.

  > Bayer, F.P., Gander, M., Kuster, B., The, M. CurveCurator: a recalibrated F-statistic to assess, classify, and explore significance of dose–response curves. Nature Communications. 2023 Nov;14(7902).

- [DIPK](https://doi.org/10.1093/bib/bbae153): Implemented model in the pipeline.

  > Li P, Jiang Z, Liu T, Liu X, Qiao H, Yao X. Improving drug response prediction via integrating gene relationships with deep learning. Briefings in Bioinformatics. 2024 May;25(3):bbae153.

- [MOLI](https://doi.org/10.1093/bioinformatics/btz318): Implemented model in the pipeline.

  > Sharifi-Noghabi H, Zolotareva O, Collins CC, Ester M. MOLI: multi-omics late integration with deep neural networks for drug response prediction. Bioinformatics. 2019 Jul;35(14):i501-9.

- [SRMF](https://doi.org/10.1186/s12885-017-3500-5): Implemented model in the pipeline.

  > Wang L, Li X, Zhang L, Gao Q. Improved anticancer drug response prediction in cell lines using matrix factorization with similarity regularization. BMC cancer. 2017 Dec;17:1-2.

- [SuperFELT](https://doi.org/10.1186/s12859-021-04146-z): Implemented model in the pipeline.

  > Park S, Soh J, Lee H. Super. FELT: supervised feature extraction learning using triplet loss for drug response prediction with multi-omics data. BMC bioinformatics. 2021 May 25;22(1):269.

## Software packaging/containerisation tools

- [Anaconda](https://anaconda.com)

  > Anaconda Software Distribution. Computer software. Vers. 2-2.4.0. Anaconda, Nov. 2016. Web.

- [Bioconda](https://pubmed.ncbi.nlm.nih.gov/29967506/)

  > Grüning B, Dale R, Sjödin A, Chapman BA, Rowe J, Tomkins-Tinch CH, Valieris R, Köster J; Bioconda Team. Bioconda: sustainable and comprehensive software distribution for the life sciences. Nat Methods. 2018 Jul;15(7):475-476. doi: 10.1038/s41592-018-0046-7. PubMed PMID: 29967506.

- [BioContainers](https://pubmed.ncbi.nlm.nih.gov/28379341/)

  > da Veiga Leprevost F, Grüning B, Aflitos SA, Röst HL, Uszkoreit J, Barsnes H, Vaudel M, Moreno P, Gatto L, Weber J, Bai M, Jimenez RC, Sachsenberg T, Pfeuffer J, Alvarez RV, Griss J, Nesvizhskii AI, Perez-Riverol Y. BioContainers: an open-source and community-driven framework for software standardization. Bioinformatics. 2017 Aug 15;33(16):2580-2582. doi: 10.1093/bioinformatics/btx192. PubMed PMID: 28379341; PubMed Central PMCID: PMC5870671.

- [Docker](https://dl.acm.org/doi/10.5555/2600239.2600241)

  > Merkel, D. (2014). Docker: lightweight linux containers for consistent development and deployment. Linux Journal, 2014(239), 2. doi: 10.5555/2600239.2600241.

- [Singularity](https://pubmed.ncbi.nlm.nih.gov/28494014/)

  > Kurtzer GM, Sochat V, Bauer MW. Singularity: Scientific containers for mobility of compute. PLoS One. 2017 May 11;12(5):e0177459. doi: 10.1371/journal.pone.0177459. eCollection 2017. PubMed PMID: 28494014; PubMed Central PMCID: PMC5426675.

GitHub Events

Total
  • Create event: 16
  • Release event: 1
  • Issues event: 13
  • Watch event: 15
  • Delete event: 18
  • Issue comment event: 23
  • Push event: 120
  • Pull request review comment event: 35
  • Pull request event: 52
  • Pull request review event: 49
  • Fork event: 3
Last Year
  • Create event: 16
  • Release event: 1
  • Issues event: 13
  • Watch event: 15
  • Delete event: 18
  • Issue comment event: 23
  • Push event: 120
  • Pull request review comment event: 35
  • Pull request event: 52
  • Pull request review event: 49
  • Fork event: 3

Issues and Pull Requests

Last synced: 6 months ago

All Time
  • Total issues: 7
  • Total pull requests: 26
  • Average time to close issues: 19 days
  • Average time to close pull requests: 8 days
  • Total issue authors: 3
  • Total pull request authors: 4
  • Average comments per issue: 0.0
  • Average comments per pull request: 0.54
  • Merged pull requests: 16
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 7
  • Pull requests: 26
  • Average time to close issues: 19 days
  • Average time to close pull requests: 8 days
  • Issue authors: 3
  • Pull request authors: 4
  • Average comments per issue: 0.0
  • Average comments per pull request: 0.54
  • Merged pull requests: 16
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • JudithBernett (8)
  • famosab (1)
  • PascalIversen (1)
  • edmundmiller (1)
Pull Request Authors
  • JudithBernett (19)
  • nf-core-bot (9)
  • PascalIversen (1)
  • picciama (1)
Top Labels
Issue Labels
enhancement (5) bug (2)
Pull Request Labels

Dependencies

.github/workflows/awsfulltest.yml actions
  • actions/upload-artifact v3 composite
  • seqeralabs/action-tower-launch v2 composite
.github/workflows/awstest.yml actions
  • actions/upload-artifact v3 composite
  • seqeralabs/action-tower-launch v2 composite
.github/workflows/branch.yml actions
  • mshick/add-pr-comment v1 composite
.github/workflows/ci.yml actions
  • actions/checkout v3 composite
  • nf-core/setup-nextflow v1 composite
.github/workflows/clean-up.yml actions
  • actions/stale v7 composite
.github/workflows/fix-linting.yml actions
  • actions/checkout v3 composite
  • actions/setup-node v3 composite
.github/workflows/linting.yml actions
  • actions/checkout v3 composite
  • actions/setup-node v3 composite
  • actions/setup-python v4 composite
  • actions/upload-artifact v3 composite
  • mshick/add-pr-comment v1 composite
  • nf-core/setup-nextflow v1 composite
  • psf/black stable composite
.github/workflows/linting_comment.yml actions
  • dawidd6/action-download-artifact v2 composite
  • marocchino/sticky-pull-request-comment v2 composite
pyproject.toml pypi