minrf
Minimal implementation of scalable rectified flow transformers, based on SD3's approach
Science Score: 54.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
✓CITATION.cff file
Found CITATION.cff file -
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
✓Academic publication links
Links to: arxiv.org -
○Committers with academic emails
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (11.8%) to scientific vocabulary
Repository
Minimal implementation of scalable rectified flow transformers, based on SD3's approach
Basic Info
Statistics
- Stars: 599
- Watchers: 8
- Forks: 52
- Open Issues: 11
- Releases: 0
Metadata Files
README.md
Minimal Implementation of Scalable Rectified Flow Transformers
Left is the naive RF, right is the logit-normal time-sampling RF. Both are trained on MNIST.
This repository contains a minimal implementation of the rectified flow models. I've taken SD3 approach of training along with LLaMA-DiT architecture. Unlike my previous repo this time I've decided to split the file into 2: The model implementation and actual code, but you don't have to look at the model code.
Everything is still self-contained, minimal, and hopefully easy to hack. There is nothing complicated goin on if you understood the math.
1. Simple Rectified Flow, for beginners
Install torch, pil, torchvision
pip install torch torchvision pillow
Run
bash
python rf.py
to train the model on MNIST from scratch.
If you are cool and want to train CIFAR instead, you can do that.
bash
python rf.py --cifar
On 63'th epoch, your output should be something like:
2. Massive Rectified Flow, muP Support
This is for gigachads who wants to train Imagenet instead. Don't worry! IMO Imagenet is the new MNIST, and we will use my imagenet.int8 dataset for this.
First go to advanced dir, download the dataset.
bash
cd advanced
pip install hf_transfer # just do install this.
bash download.sh
This shouldn't take more than 5 min if your network is decent.
Run
bash
bash run.sh
to train the model. This will train Imagenet from scratch, do a muP grid search to find the aligned basin for the loss function, you unlock the zero-shot LR transfer for Rectified Flow models!
This uses multiple techniques and codebases I have developed over the year. Its a natural mixture of min-max-IN-dit, min-max-gpt, ez-muP
Citations
If you use this material, please cite this repository with the following:
bibtex
@misc{ryu2024minrf,
author = {Simo Ryu},
title = {minRF: Minimal Implementation of Scalable Rectified Flow Transformers},
year = 2024,
publisher = {Github},
url = {https://github.com/cloneofsimo/minRF},
}
Owner
- Name: Simo Ryu
- Login: cloneofsimo
- Kind: user
- Company: Corca AI
- Website: https://fb.com/MLPaperFetchingCat
- Twitter: cloneofsimo
- Repositories: 10
- Profile: https://github.com/cloneofsimo
Cats are Turing machines cloneofsimo@gmail.com
Citation (CITATION.cff)
cff-version: 1.2.0 message: "Citations would be appreciated if you end up using this tool! I currently go by Simo Ryu" authors: - family-names: "Ryu" given-names: "Simo" orcid: "https://orcid.org/0009-0008-0017-2677" title: "minRF: Minimal Implementation of Scalable Rectified Flow Transformers" version: 0.0.1 date-released: 2024-05 url: "https://github.com/cloneofsimo/lora"
GitHub Events
Total
- Issues event: 1
- Watch event: 150
- Pull request event: 2
- Fork event: 21
Last Year
- Issues event: 1
- Watch event: 150
- Pull request event: 2
- Fork event: 21
Issues and Pull Requests
Last synced: 11 months ago
All Time
- Total issues: 10
- Total pull requests: 3
- Average time to close issues: 4 days
- Average time to close pull requests: 3 months
- Total issue authors: 10
- Total pull request authors: 3
- Average comments per issue: 0.9
- Average comments per pull request: 0.0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 9
- Pull requests: 3
- Average time to close issues: 4 days
- Average time to close pull requests: 3 months
- Issue authors: 9
- Pull request authors: 3
- Average comments per issue: 0.78
- Average comments per pull request: 0.0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- zaptrem (1)
- mephisto28 (1)
- win10ogod (1)
- George0726 (1)
- breaddaerb (1)
- cloneofsimo (1)
- Xynonners (1)
- nicolas-dufour (1)
- simulateuser538 (1)
- Xact-sniper (1)
Pull Request Authors
- montyanderson (2)
- julien-blanchon (1)
- luke-a-thompson (1)