ML Pipeline for Drug Discovery

A comprehensive machine learning pipeline for drug discovery and virtual screening, featuring model training, selection, fusion, and deployment capabilities.

🚀 Features

• End-to-end ML Pipeline: Complete workflow from data preparation to model deployment
• Model Training & Selection: Support for multiple ML algorithms with automated model selection
• Model Fusion: Advanced ensemble methods for improved performance
• Virtual Screening: High-throughput screening of chemical compounds
• MLflow Integration: Experiment tracking and model versioning
• Cloud Storage: Automatic artifact upload to Google Cloud Storage
• Conformal Prediction: Uncertainty quantification for predictions
• Chemistry Filters: Built-in molecular property filters

🛠️ Installation

Prerequisites

• Python 3.8+
• pip or conda package manager
• Git

Setup

1. Clone the repository

bash git clone <repository-url>

1. Create virtual environment

bash python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate

1. Install dependencies bash pip install -r requirements.txt pip install -r requirements_mapie_conformal.txt #install this to use conformal

🚀 Quick Start

Basic Usage

```bash

Run with default configuration

python -m src

Run with custom configuration

python -m src --config path/to/your/config.yaml

Run with verbose logging

python -m src --verbose ```

🔄 Pipeline Overview

The ML pipeline consists of 10 main steps:

1. Configuration Loading

• Loads YAML configuration file
• Sets up run folder and logging

2. Data Preparation

• Creates balanced datasets
• Handles data preprocessing
• Feature engineering

3. Model Training

• Trains multiple ML models
• Supports various algorithms (RF, SVM, XGBoost, Neural Networks)
• Cross-validation and hyperparameter tuning

4. Model Testing

• Evaluates trained models on test sets
• Generates performance metrics
• Creates evaluation reports

5. Model Selection

• Selects best performing models
• Supports multiple evaluation criteria
• Generates radar charts for visualization

6. Model Fusion

• Combines predictions from multiple models
• Ensemble methods for improved performance
• Optional step based on configuration

7. Virtual Screening

• Screens large compound libraries
• Applies chemistry filters
• Generates screening reports

8. MLflow Logging

• Logs experiments and models
• Tracks metrics and parameters
• Model versioning and registry

9. Artifact Upload

• Uploads results to Google Cloud Storage
• Organizes artifacts for easy access

10. Cleanup

• Removes run_name files
• Configurable cleanup options (default True, if you want to keep runfile set `cleanupafter_runtoTrue` )

⚙️ Configuration

Main Configuration File (config_loader.yaml)

```yaml

Data Configuration

proteinname: "WDR91" # Target Name istrain: True # Running train phase (False:for no, True: for yes) istest: True # Running test phase (False:for no, True: for yes) isscreen: True # Running screen phase (False:for no, True: for yes)

train_data: - ./data/TrainFiles/company1.parquet

testdata: - ./data/TestFiles/sampleddatatest1.parquet

desiredcolumns: - ECFP4 # Correct format: [ECFP4], and [ECFP4, ECFP6, ...] if multuple columns labelcolumntrain: LABEL labelcolumntest: LABEL nrowstrain: None # integer or None nrowstest: None featurefusionmethod: None # options: None, All, Pairwise balanceflag: False # Creating blanced train sets (True/False) balanceratios: # balanceratios: [1, 2, 4, 8] - 1 # Ratio of positive to negative samples in the balanced dataset

Model Configuration

desired_models: - lgbm

hyperparameterstuning: False # (N:for no, Y: for yes) tfmodels: - tfff - tfcnn1D

Specifying hyperparameters

hyperparameters: tfff: inputshape: 2048 hiddenunits: - 128 - 64 learningrate: 0.0005

Training Configuration

Nfold: 2

Conformal Prediction

conformalprediction: False # Running conformal prediction (N:for no, Y: for yes) conformaltestsize: 0.3 conformalconfidence_level: 0.95

Model selection

trainfileformodelselection: [] # If empty, the top model by evaluation columns and result on the evaluation set is selected. Example: trainfileformodelselection: WDR91SGC.parquet evaluationfileformodelselection: [] # If empty, the top model by evaluation columns is selected. Exmple: evaluationfileformodelselection: evaluation.parquet evaluationcolumn: - TestHitsAt200 - TestHitsAt500 crossvalidationcolumn: - CVHitsAt200 - CVPrecision - CVRecall - CVAccuracy - CVPlatePPV

Model Fusion

Fusion: True # Running model fusion (N:for no, Y: for yes) numtopmodels: 2

Cloud Storage

bucketname: "your-gcs-bucket" prefixname: "mlflow-artifacts"

Cleanup

cleanupafterrun: true ```

Environment Variables (.env)

📖 Usage

Command Line Interface

```bash

Basic usage

python -m src

With custom config

python -m src --config configs/custom_config.yaml

With verbose logging

python -m src --verbose

Help

python -m src --help ```

📁 Project Structure

---

├── LICENSE
├── Makefile           <- Makefile with commands like `make data` or `make train`
├── README.md          <- The top-level README for developers using this project.
├── data
│   ├── external       <- Data from third party sources.
│   ├── interim        <- Intermediate data that has been transformed.
│   ├── processed      <- The final, canonical data sets for modeling.
│   └── raw            <- The original, immutable data dump.
│
├── docs               <- A default Sphinx project; see sphinx-doc.org for details
│
├── models             <- Trained and serialized models, model predictions, or model summaries
│
├── notebooks          <- Jupyter notebooks. Naming convention is a number (for ordering),
│                         the creator's initials, and a short `-` delimited description, e.g.
│                         `1.0-jqp-initial-data-exploration`.
│
├── references         <- Data dictionaries, manuals, and all other explanatory materials.
│
├── reports            <- Generated analysis as HTML, PDF, LaTeX, etc.
│   └── figures        <- Generated graphics and figures to be used in reporting
│
├── requirements.txt   <- The requirements file for reproducing the analysis environment, e.g.
│                         generated with `pip freeze > requirements.txt`
│
├── setup.py           <- makes project pip installable (pip install -e .) so src can be imported
├── src                <- Source code for use in this project.
│   ├── __init__.py    <- Makes src a Python module
│   │
│   ├── data           <- Scripts to download or generate data
│   │   └── make_dataset.py
│   │
│   ├── features       <- Scripts to turn raw data into features for modeling
│   │   └── build_features.py
│   │
│   ├── models         <- Scripts to train models and then use trained models to make
│   │   │                 predictions
│   │   ├── predict_model.py
│   │   └── train_model.py
│   │
│   └── visualization  <- Scripts to create exploratory and results oriented visualizations
│       └── visualize.py
│
└── tox.ini            <- tox file with settings for running tox; see tox.readthedocs.io

---

Project based on the cookiecutter data science project template. #cookiecutterdatascience

📊 Monitoring & Logging

MLflow UI

```bash

Start MLflow server

mlflow server --host 0.0.0.0 --port 5000

Access UI at http://127.0.0.1:5000/

```

🔧 Customization

Adding New Models

1. Create model class in src/models/
2. Add to configuration in config_loader.yaml
3. Update training pipeline in train_model.py

Custom Evaluation Metrics

1. Add metric function to src/utils/eval_utils11.py
2. Update configuration to include new metric
3. Modify selection criteria as needed

Custom Screening Filters

1. Add filter function to src/screening/clustering.py
2. Update configuration to enable new filter
3. Test with sample data

🐛 Troubleshooting

Common Issues

1. Memory Issues

yaml # Reduce batch size in config nrows_train: 10000 nrows_test: 5000

1. MLflow Connection Issues bash # Check MLflow server status mlflow server --help

🤝 Contributing

We welcome contributions! Please follow these steps:

1. Fork the repository
2. Create feature branch: git checkout -b feature/amazing-feature
3. Commit changes: git commit -m 'Add amazing feature'
4. Push to branch: git push origin feature/amazing-feature
5. Open Pull Request

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• MLflow for experiment tracking
• scikit-learn for machine learning algorithms
• RDKit for cheminformatics
• Google Cloud for storage solutions

---

Happy Drug Discovery! 🧬💊