phylo-diffusion
Code Repository for Phylo-Diffusion. Accepted at ECCV 2024
Science Score: 62.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
✓CITATION.cff file
Found CITATION.cff file -
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
✓Academic publication links
Links to: arxiv.org -
○Academic email domains
-
✓Institutional organization owner
Organization imageomics has institutional domain (imageomics.osu.edu) -
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (9.4%) to scientific vocabulary
Repository
Code Repository for Phylo-Diffusion. Accepted at ECCV 2024
Basic Info
- Host: GitHub
- Owner: Imageomics
- License: mit
- Language: Python
- Default Branch: main
- Homepage: https://imageomics.github.io/phylo-diffusion/
- Size: 1.71 MB
Statistics
- Stars: 4
- Watchers: 2
- Forks: 1
- Open Issues: 1
- Releases: 0
Metadata Files
README.md
Hierarchical Conditioning of Diffusion Models Using Tree-of-Life for Studying Species Evolution (ECCV 2024)
This repo is the official implementation of "Hierarchical Conditioning of Diffusion Models Using Tree-of-Life for Studying Species Evolution"
Paper | Project Page | Demo
Requirements
A suitable conda environment named phylo_diffusion can be created
and activated with:
conda env create -f environment.yaml
conda activate phylo_diffusion
Model Training
For training the models, we can use the following command:
python main.py --name <mddel_name> --logdir <path_to_logdir> --base <yaml_config_path> --postfix <file_postfix_name> -t True --gpus <comma-separated GPU indices>
We first need to train the base autoencoder model. <!-- This can be trained using the following command: -->
Dataset
Please find the Fish Dataset at link
Trained Models and related files
Please find the trained models at link
Sampling Images
python scripts/trait_masking.py --config_path <path_to_config_file> --ckpt_path <path_to_saved_model> --node_dict <path_to_hierarchical_node_dict> --output_dir_name <output_dir_name>
Citation
@inproceedings{khurana2024hierarchical,
title={Hierarchical Conditioning of Diffusion Models Using Tree-of-Life for Studying Species Evolution},
author={Khurana, Mridul and Daw, Arka and Maruf, M and Uyeda, Josef C and Dahdul, Wasila and Charpentier, Caleb and Bak{\i}{\c{s}}, Yasin and Bart Jr, Henry L and Mabee, Paula M and Lapp, Hilmar and others},
booktitle={European Conference on Computer Vision},
pages={137--153},
year={2024},
organization={Springer}
}
Acknowledgments
The code base is borrowed from the original implementation of Latent Diffusion Models [1] available at LDM code. Please consider citing LDM as well.
References
[1] Rombach, Robin, et al. "High-resolution image synthesis with latent diffusion models." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.
Owner
- Name: Imageomics Institute
- Login: Imageomics
- Kind: organization
- Website: https://imageomics.osu.edu
- Twitter: imageomics
- Repositories: 4
- Profile: https://github.com/Imageomics
Citation (CITATION.cff)
abstract: "A central problem in biology is to understand how organisms evolve and adapt to their environment by acquiring variations in the observable characteristics or traits of species across the tree of life. With the growing availability of large-scale image repositories in biology and recent advances in generative modeling, there is an opportunity to accelerate the discovery of evolutionary traits automatically from images. Toward this goal, we introduce Phylo-Diffusion, a novel framework for conditioning diffusion models with phylogenetic knowledge represented in the form of HIERarchical Embeddings (HIER-Embeds). We also propose two new experiments for perturbing the embedding space of Phylo-Diffusion: trait masking and trait swapping, inspired by counterpart experiments of gene knockout and gene editing/swapping. Our work represents a novel methodological advance in generative modeling to structure the embedding space of diffusion models using tree-based knowledge. Our work also opens a new chapter of research in evolutionary biology by using generative models to visualize evolutionary changes directly from images. We empirically demonstrate the usefulness of Phylo-Diffusion in capturing meaningful trait variations for fishes and birds, revealing novel insights about the biological mechanisms of their evolution."
authors:
- family-names: Khurana
given-names: Mridul
cff-version: 1.0.0
date-released: "2024-07-31"
# identifiers:
# - description: "The GitHub release URL of tag <version>."
# type: url
# value: "https://github.com/Imageomics/<repo>/releases/tag/<tag-name>"
# - description: "The GitHub URL of the commit tagged with <tag-name>."
# type: url
# value: "https://github.com/Imageomics/<repo>/tree/<commit-hash>"
keywords:
- "diffusion models"
- "hierarchical conditioning"
- evolution
- imageomics
- "knowledge-guided"
- CV
- biology
- images
- multimodal
license: MIT
message: "If you find this software helpful in your research, please cite both the software and our paper."
repository-code: "https://github.com/Imageomics/phylo-diffusion"
title: "Phylo-Diffusion"
version: 1.0.0
type: software
preferred-citation:
type: article
authors:
- family-names: Khurana
given-names: Mridul
- family-names: Daw
given-names: Arka
- family-names: Maruf
given-names: "M."
- family-names: Uyeda
given-names: "Josef C."
- family-names: Dahdul
given-names: Wasila
- family-names: Charpentier
given-names: Caleb
- family-names: Bakış
given-names: "Yasin"
- family-names: "Bart Jr."
given-names: "Henry L."
- family-names: Mabee
given-names: "Paula M."
- family-names: Lapp
given-names: Hilmar
- family-names: Balhoff
given-names: "James P."
- family-names: Chao
given-names: "Wei-Lun"
- family-names: Stewart
given-names: Charles
- family-names: "Berger-Wolf"
given-names: Tanya
- family-names: Karpatne
given-names: Anuj
title: "Hierarchical Conditioning of Diffusion Models Using Tree-of-Life for Studying Species Evolution"
year: 2024
journal: arXiv
doi: 10.48550/arXiv.2408.00160
references:
- authors:
- family-names: Rombach
given-names: Robin
- family-names: Blattmann
given-names: Andreas
- family-names: Lorenz
given-names: Dominik
- family-names: Esser
given-names: Patrick
- family-names: Ommer
given-names: Bjorn
title: "High-resolution image synthesis with latent diffusion models"
# version:
type: article
# doi:
# date-released:
GitHub Events
Total
- Issues event: 1
- Watch event: 3
- Issue comment event: 1
- Push event: 5
Last Year
- Issues event: 1
- Watch event: 3
- Issue comment event: 1
- Push event: 5
Dependencies
- albumentations ==0.4.3
- einops ==0.3.0
- imageio ==2.9.0
- imageio-ffmpeg ==0.4.2
- numpy *
- omegaconf ==2.0.0
- open_clip_torch *
- opencv-python ==4.1.2.30
- pudb ==2019.2
- pytorch-lightning ==1.4.2
- scikit-learn *
- seaborn ==0.9.0
- streamlit >=0.73.1
- test-tube >=0.7.5
- torch-fidelity ==0.3.0
- torchmetrics ==0.6.0
- tqdm ==4.64.1
- transformers ==4.35.0
- wandb ==0.13.5
- numpy *
- torch *
- tqdm *