https://github.com/bytedance/multi-reward-editing
Multi-Reward as Condition for Instruction-Based Image Editing
Science Score: 23.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
✓Academic publication links
Links to: arxiv.org -
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (9.9%) to scientific vocabulary
Keywords
Repository
Multi-Reward as Condition for Instruction-Based Image Editing
Basic Info
Statistics
- Stars: 28
- Watchers: 1
- Forks: 0
- Open Issues: 0
- Releases: 0
Topics
Metadata Files
README.md
Multi-Reward as Condition for Instruction-Based Image Editing
🔮 Welcome to the official code repository for Multi-Reward as Condition for Instruction-Based Image Editing. We're excited to share our work with you, please bear with us as we prepare code. Stay tuned for the reveal!
Architecture
Implementation
Dataset Preparation
Download the dataset from Hugging Face
- Training Images: InstructPix2Pix
- Training Reward: RewardEdit-20K
- Test Images and Instructions: Real-Edit
Model Preparation
Download the model from Hugging Face
- Basic modules: stable-diffusion-v1-5, sdxl-vae-fp16-fix, CLIP-ViT-H-14-laion2B-s32B-b79K, clip-vit-large-patch14-336
- Our trained models: stage1instructpix2pix, stage2rewardinstruct_pix2pix
Requirements
```shell
Python 3.9, PyTorch 2.1.0 with CUDA 12.2
pip3 install -r requirements.txt ```
Training and Evaluation
Please utilize the script provided below: ```shell
Training
python3 -m torch.distributed.launch \ --nnodes $WORKERNUM \ --noderank $ID \ --nprocpernode $WORKERGPU \ --masteraddr $METISWORKER0HOST \ --masterport $PORT \ trainsd15.py \ --stage=2 \ --pretrainedmodelnameorpath=$STAGE1MODELPATH \ --resolution=256 --randomflip \ --trainbatchsize=4 --gradientaccumulationsteps=4 \ --maxtrainsteps=10000 --checkpointingsteps=500 \ --learningrate=5e-5 --lrwarmupsteps=0 \ --conditioningdropoutprob=0.05 \ --imageencoderpath=./CLIP-ViT-H-14-laion2B-s32B-b79K \ --outputdir=$OUTPUTDIR
Evaluation
python3 eval.py --modelpath=$STAGE2MODEL_PATH ```
Citation
If you find this project useful in your research, please consider citing:
@article{gu2024multi,
title={Multi-Reward as Condition for Instruction-based Image Editing},
author={Gu, Xin and Li, Ming and Zhang, Libo and Chen, Fan and Wen, Longyin and Luo, Tiejian and Zhu, Sijie},
journal={arXiv preprint arXiv:2411.04713},
year={2024}
}
Owner
- Name: Bytedance Inc.
- Login: bytedance
- Kind: organization
- Location: Singapore
- Website: https://opensource.bytedance.com
- Twitter: ByteDanceOSS
- Repositories: 255
- Profile: https://github.com/bytedance
GitHub Events
Total
- Issues event: 10
- Watch event: 41
- Issue comment event: 1
- Push event: 1
- Public event: 1
Last Year
- Issues event: 10
- Watch event: 41
- Issue comment event: 1
- Push event: 1
- Public event: 1
Dependencies
- accelerate *
- bytedlogid *
- datasets *
- diffusers ==0.29.0
- httpx ==0.23.0
- numpy ==1.23.2
- openai *
- sentencepiece ==0.1.99
- torchmetrics ==1.0.0
- transformers ==4.43.0
- xformers ==0.0.22.post7