https://github.com/bytedance/gr-mg
Official implementation of GR-MG
Science Score: 23.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
✓Academic publication links
Links to: arxiv.org, scholar.google -
○Committers with academic emails
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (11.2%) to scientific vocabulary
Keywords
Repository
Official implementation of GR-MG
Basic Info
- Host: GitHub
- Owner: bytedance
- License: apache-2.0
- Language: Python
- Default Branch: main
- Homepage: https://gr-mg.github.io/
- Size: 1.06 MB
Statistics
- Stars: 79
- Watchers: 1
- Forks: 6
- Open Issues: 2
- Releases: 0
Topics
Metadata Files
README.md
GR-MG
This repo contains code for the paper:
Leveraging Partially Annotated Data via Multi-Modal Goal Conditioned Policy
Peiyan Li, Hongtao Wu*‡, Yan Huang*, Chilam Cheang, Liang Wang, Tao Kong
*Corresponding author ‡ Project lead
🌐 Project Website | 📄 Paper
News
- (🔥 New) (2024.12.18) Our paper was accepted by IEEE Robotics and Automation Letter (RA-L) !
- (🔥 New) (2024.08.27) We have released the code and checkpoints of GR-MG ! ## Preparation Note: We only test GR-MG with CUDA 12.1 and python 3.9
```bash
clone this repository
git clone https://github.com/bytedance/GR-MG.git cd GR_MG
install dependencies for goal image generation model
bash ./goal_gen/install.sh
install dependencies for multi-modal goal conditioned policy
bash ./policy/install.sh
``
Download the pretrained [InstructPix2Pix](https://huggingface.co/timbrooks/instruct-pix2pix) weights from Huggingface and save them inresources/IP2P/.
Download the pretrained MAE encoder [mae_pretrain_vit_base.pth ](https://dl.fbaipublicfiles.com/mae/pretrain/mae_pretrain_vit_base.pth) and save it inresources/MAE/`.
Download and unzip the CALVIN dataset.
Checkpoints
Training
1. Train Goal Image Generation Model
```bash
modify the variables in the script before you execute the following instruction
bash ./goalgen/trainip2p.sh ./goal_gen/config/train.json ```
2. Pretrain Multi-modal Goal Conditioned Policy
We use the method described in GR-1 and pretrain our policy with Ego4D videos. You can download the pretrained model checkpoint here. You can also pretrain the policy yourself using the scripts we provide. Before doing this, you'll need to download the Ego4D dataset.
```bash
pretrain multi-modal goal conditioned policy
bash ./policy/main.sh ./policy/config/pretrain.json ```
3. Train Multi-modal Goal Conditioned Policy
After pretraining, modify the pretrainedmodelpath in /policy/config/train.json and execute the following instruction to train the policy.
```bash
train multi-modal goal conditioned policy
bash ./policy/main.sh ./policy/config/train.json ```
Evaluation
To evaluate our model on CALVIN, you can execute the following instruction: ```bash
Evaluate GR-MG on CALVIN
bash ./evaluate/eval.sh ./policy/config/train.json
``
In theeval.sh` script, you can specify which goal image generation model and policy to use. Additionally, we provide multi-GPU evaluation code, allowing you to evaluate different training epochs of the policy simultaneously.
Contact
If you have any questions about the project, please contact peiyan.li@cripac.ia.ac.cn.
Acknowledgements
We thank the authors of the following projects for making their code and dataset open source:
Citation
If you find this project useful, please star the repository and cite our paper:
@article{li2025gr,
title={GR-MG: Leveraging Partially-Annotated Data Via Multi-Modal Goal-Conditioned Policy},
author={Li, Peiyan and Wu, Hongtao and Huang, Yan and Cheang, Chilam and Wang, Liang and Kong, Tao},
journal={IEEE Robotics and Automation Letters},
year={2025},
publisher={IEEE}
}
Owner
- Name: Bytedance Inc.
- Login: bytedance
- Kind: organization
- Location: Singapore
- Website: https://opensource.bytedance.com
- Twitter: ByteDanceOSS
- Repositories: 255
- Profile: https://github.com/bytedance
GitHub Events
Total
- Issues event: 13
- Watch event: 46
- Issue comment event: 16
- Push event: 6
- Fork event: 7
Last Year
- Issues event: 13
- Watch event: 46
- Issue comment event: 16
- Push event: 6
- Fork event: 7
Committers
Last synced: about 1 year ago
Top Committers
| Name | Commits | |
|---|---|---|
| lipeiyan-bd | l****9@b****m | 11 |
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: about 1 year ago
All Time
- Total issues: 14
- Total pull requests: 0
- Average time to close issues: 5 days
- Average time to close pull requests: N/A
- Total issue authors: 11
- Total pull request authors: 0
- Average comments per issue: 2.21
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 14
- Pull requests: 0
- Average time to close issues: 5 days
- Average time to close pull requests: N/A
- Issue authors: 11
- Pull request authors: 0
- Average comments per issue: 2.21
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- JLUtangchuan (3)
- COST-97 (2)
- Stardust-hyx (1)
- RanCao2018 (1)
- passage2016 (1)
- ye450450 (1)
- charlesliangcai (1)
- zijunfdu (1)
- JYounngS (1)
- danzel-crazy (1)
- dingjiansw101 (1)