OVGNet: An Unified Visual-Linguistic Framework for Open-Vocabulary Robotic Grasping

Meng Li · Qi Zhao · Shuchang Lyu · Chunlei Wang · Yujing Ma · Guangliang Cheng · Chenguang Yang

Highlight!!!!

This repo is the implementation of "OVGNet: An Unified Visual-Linguistic Framework for Open-Vocabulary Robotic Grasping". we refer to Vision-Language-Grasping, GroundingDINO, VL-Grasp. Many thanks to these excellent repos.

Demo Setting

• Novel indicates the unseen objects in training.
• Base denotes the seen objects in training.
• Battery and power drill are novel classes, which belong to hard task.
• Apple and pear are base classes, which belong to simple task.

Demo Video

Demo

Dataset

• OVGrasping follows GroundingDINO data format.
• The OVGrapsing dataset comprises 117 categories and 63,385 instances.
• Instances are sourced from three distinct origins: RoboRefIt, GraspNet, simulated environment.
• The dataset is divided into two categories: the base category consists 51,857 instances, and the novel category comprises 11,528 instances.

Installation

• Ubantu==18.04
• Python==3.9
• Torch==1.11, Torchvision==0.12.0
• CUDA==11.3
• checkpoint==OVGANet
• assets==assets

please add the assets into OVGNet folder
please ensure the CUDA version is 11.3 conda create -n OVGNet python=3.9 conda activate OVGNet pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0 --extra-index-url https://download.pytorch.org/whl/cu113

cd /OVGNet/ pip install -r requirments.txt

cd graspnet/graspnet/pointnet2 python setup.py install cd graspnet/graspnet/knn python setup.py install cd groundingdino pip install -e .

Run

cd /OVGNet/ python test.py --testing_case_dir ./test_cases/simple/apple --pretrain ./checkpoint/OVGANet

Test on OVGrasping

cd /OVGNet/test_vg/ python test_vg.py --c ./config/cfg_odvg.py --datasets ./config/datasets_vg_example.json --pretrain_model_path OVGNet/checkpoint/OVGANet

Cite

``` @InProceedings{Li2024IROS, author = {Li Meng and Zhao Qi and Lyu Shuchang and Wang Chunlei and Ma Yujing and Cheng Guangliang and Yang Chenguang}, title = {OVGNet: A Unified Visual-Linguistic Framework for Open-Vocabulary Robotic Grasping}, year = {2024}, eprint = {2407.13175}, archivePrefix = {arXiv}, primaryClass = {cs.RO}, url = {https://arxiv.org/abs/2407.13175}, }