https://github.com/cloneofsimo/minmomentmatching

Science Score: 36.0%

This score indicates how likely this project is to be science-related based on various indicators:

○
CITATION.cff file
✓
codemeta.json file
Found codemeta.json file
✓
.zenodo.json file
Found .zenodo.json file
○
DOI references
✓
Academic publication links
Links to: arxiv.org
○
Academic email domains
○
Institutional organization owner
○
JOSS paper metadata
○
Scientific vocabulary similarity
Low similarity (6.9%) to scientific vocabulary

Last synced: 10 months ago · JSON representation

Repository

Basic Info

Host: GitHub
Owner: cloneofsimo
Language: Python
Default Branch: main
Size: 8.79 KB

Statistics

Stars: 10
Watchers: 1
Forks: 0
Open Issues: 0
Releases: 0

Created over 1 year ago · Last pushed over 1 year ago

Metadata Files

Readme

README.md

Moment Matching Distillation for Diffusion Models

This repository implements "Multistep Distillation of Diffusion Models via Moment Matching" by Google, using LLaMA-DiT architecture and Flow matching scheduler.

Requirements

bash pip install torch torchvision tqdm tensorboard pillow click

Training

First train the teacher model:

bash python train_student.py --cifar --epochs 100 --batch_size 64

Then train the student models using moment matching:

bash python train.py --cifar --epochs 100 --batch_size 256 --init_ckpt_dir checkpoints/model_epoch_44.pt

Algorithm 2

The training process follows Algorithm 2 from the paper:

Sample time s < t and noise z_t at time t
Get student prediction x_student from z_t
Sample z_s using x_student
Get auxiliary prediction x_aux and teacher prediction x_teacher from z_s
Alternate between:
- Training student by minimizing E[x_student * (x_aux - x_teacher)]
- Training auxiliary by minimizing ||x_student - x_aux||^2 + ||x_teacher - x_aux||^2

Architecture

Uses LLaMA-DiT (DiT with LLaMA-style blocks) as the base architecture
Rectified Flow (Sub-VP) as the diffusion scheduler
Supports both MNIST and CIFAR-10 datasets

Monitoring

Training progress can be monitored via TensorBoard: bash tensorboard --logdir checkpoints/logs

This will show: - Training loss - Generated samples - Sampling process GIFs

Citation

bibtex @article{deepmind2024moment, title={Multistep Distillation of Diffusion Models via Moment Matching}, author={DeepMind}, journal={arXiv preprint arXiv:2406.04103}, year={2024} }

License

MIT

Owner

Name: Simo Ryu
Login: cloneofsimo
Kind: user
Company: Corca AI

Website: https://fb.com/MLPaperFetchingCat
Twitter: cloneofsimo
Repositories: 10
Profile: https://github.com/cloneofsimo

Cats are Turing machines cloneofsimo@gmail.com

GitHub Events

Total

Watch event: 10
Push event: 1
Create event: 2

Last Year

Watch event: 10
Push event: 1
Create event: 2

Issues and Pull Requests

Last synced: about 1 year ago

All Time

Total issues: 0
Total pull requests: 0
Average time to close issues: N/A
Average time to close pull requests: N/A
Total issue authors: 0
Total pull request authors: 0
Average comments per issue: 0
Average comments per pull request: 0
Merged pull requests: 0
Bot issues: 0
Bot pull requests: 0

Past Year

Issues: 0
Pull requests: 0
Average time to close issues: N/A
Average time to close pull requests: N/A
Issue authors: 0
Pull request authors: 0
Average comments per issue: 0
Average comments per pull request: 0
Merged pull requests: 0
Bot issues: 0
Bot pull requests: 0

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Open Source Science