OSSID: Online Self-Supervised Instance Detection (and for) Pose Estimation

OSSID is a self-supervised pipeline for object instance detection by, and for, 6D pose estimation. The results of a zero-shot 6D pose estimation method (Zephyr) used to finetune a zero-shot detector online. Then the detection results in turn provide object bounding boxes and reduce the search space for pose estimation. OSSID enables detection and pose estimation to adapt to new objects and environments online, without the need of no manual annotation. OSSID achieve the state-of-the-art results on YCB-V and LM-O datasets.

[Arxiv] [Project Page] [Video]

Setup

1. Clone this repo. Update the path to this repo as OSSID_ROOT in config.py. git clone git@github.com:r-pad/OSSID_code.git

2. We recommend building the environment and installing all required packages using Anaconda. Also pip install this repo cd OSSID_code conda env create -n ossid --file env.yml conda activate ossid pip install -e .

3. Clone and pip install the following libraries. Update the path to bop_toolkit source code as BOP_TOOLKIT_PATH in config.py. git clone git@github.com:r-pad/quat_math.git cd quat_math; pip install .; cd .. git clone git@github.com:r-pad/object_pose_utils.git cd object_pose_utils; pip install .; cd .. git clone git@github.com:r-pad/bop_toolkit.git # We have only tested with our own fork of bop_toolkit cd bop_toolkit; pip install -e .; cd ..

4. Set up Zephyr from this link. Note that we will use the same conda environment, so please follow instructions below (steps 2, 3, and 4 of the original instructions).

``` sudo apt-get install build-essential cmake libopencv-dev python-numpy

git clone --recurse-submodules git@github.com:r-pad/zephyr.git cd zephyr

mkdir build cd build cmake .. -DPYTHONEXECUTABLE=$(python -c "import sys; print(sys.executable)") -DPYTHONINCLUDEDIR=$(python -c "from distutils.sysconfig import getpythoninc; print(getpythoninc())") -DPYTHONLIBRARY=$(python -c "import distutils.sysconfig as sysconfig; print(sysconfig.getconfigvar('LIBDIR'))") make; make install

cd .. # move to the root folder of zephyr pip install -e . ```

1. MVTec Halcon needs to be set up to run the entire pipeline. This repo is tested with Halcon student edition (version=21.05), while it should work with newer version. Remeber to update the python binding mvtec-halcon accordingly if you use a newer version (By default, mvtec-halcon==21050.0.0 is installed). After installatio of Halcon, you might need to run

source /path/to/halcon/.profile_halcon

1. (For BOP Challenge evaluation) Setup the renderer needed for BOP evaluation as instructed here. This repo is tested with the C++ Renderer (bop_renderer). Note that you may need to change the python version in CMakeLists.txt from 3.6 to 3.8. Update config.py in bop_toolkit accordingly. Update BOP_RESULTS_FOLDER in OSSID config.py to be the same as results_path in bop_toolkit config.py.

2. (For detection mAP evaluation) Clone this Github repo. Update its path as MAP_CODE_ROOT in config.py.

Download data, checkpoints

1. Download YCB-V and LM-O datasets from BOP challenge. The "Base Archive", "Object models" and "BOP'19/20 test images" are sufficient for running evaluation. Unzip them according to the structure described in BOP Challenge into a folder and update BOP_DATASET_ROOT.

2. Download the checkpoints and preprocessed data. From Google Drive, download all the files and placed them in python/ossid/ckpts and python/ossid/data folders accordingly. Please refer to the readme.md files in these two folders for the description of each file. Note that ycbv_grid.zip needs to be unzipped at BOP_DATASETS_ROOT/ycbv/ folder.

Experiments

The following commands can be used to reproduce the results reported in our paper. First change the directory to python/ossid/scripts/ and then run the following commands.

In case of out of GPU memory error, please consider setting --finetune_batch_size 4 or a smaller number.

If you are using student license, Halcon may not be able to run in multiple processes (HALCON error #2348: Activation error). Therefore please run the following commans sequentially.

• OSSID(Ours) on LM-O in Table 1. python online_learning.py \ --dataset_name lmo \ --n_local_test 10 \ --finetune_interval 32 \ --finetune_epochs 1 \ --always_dtoid_mask \ --use_pretrained_dtoid \ --exp_name lmo_final_main

• OSSID(Ours) on YCB-V in Table 1. python online_learning.py \ --dataset_name ycbv \ --n_local_test 10 \ --finetune_interval 32 \ --finetune_epochs 1 \ --use_sift_hypos \ --finetune_warmup 128 \ --always_dtoid_mask \ --fast \ --exp_name ycbv_final_main

• OSSID(w/Conf.Filter) on LM-O in Table 1. python online_learning.py \ --dataset_name lmo \ --n_local_test 10 \ --finetune_interval 32 \ --finetune_epochs 1 \ --use_pretrained_dtoid \ --exp_name lmo_final_conf

• OSSID(w/Conf.Filter) on YCB-V in Table 1. python online_learning.py \ --dataset_name ycbv \ --n_local_test 10 \ --finetune_interval 32 \ --finetune_epochs 1 \ --use_sift_hypos \ --finetune_warmup 128 \ --fast \ --exp_name ycbv_final_conf

• OSSID(Transductive) on LM-O in Table 1. python online_learning.py \ --dataset_name lmo \ --n_local_test 10 \ --use_offline_dtoid \ --no_finetune \ --exp_name lmo_final_transductive

• OSSID(Transductive) on YCB-V in Table 1. python online_learning.py \ --dataset_name ycbv \ --n_local_test 10 \ --use_sift_hypos \ --use_offline_dtoid \ --no_finetune \ --fast \ --exp_name ycbv_final_transductive

• OSSID(Oracle) on LM-O in Table 1. python online_learning.py \ --dataset_name lmo \ --n_local_test 10 \ --finetune_interval 32 \ --finetune_epochs 1 \ --always_dtoid_mask \ --use_pretrained_dtoid \ --use_oracle_gt \ --exp_name lmo_final_oracle

• OSSID(Oracle) on YCB-V in Table 1. python online_learning.py \ --dataset_name ycbv \ --n_local_test 10 \ --finetune_interval 32 \ --finetune_epochs 1 \ --use_sift_hypos \ --finetune_warmup 128 \ --always_dtoid_mask \ --fast \ --use_oracle_gt \ --exp_name ycbv_final_oracle

Cite

If you find this codebase useful in your research, please consider citing:

``` @ARTICLE{ral2022ossid, author={Gu, Qiao and Okorn, Brian and Held, David}, journal={IEEE Robotics and Automation Letters}, title={OSSID: Online Self-Supervised Instance Detection by (And For) Pose Estimation}, year={2022}, volume={7}, number={2}, pages={3022-3029}, doi={10.1109/LRA.2022.3145488}}

@inproceedings{icra2022ossid, title={OSSID: Online Self-Supervised Instance Detection by (and for) Pose Estimation}, author={Gu, Qiao and Okorn, Brian and Held, David}, booktitle={2022 International Conference on Robotics and Automation (ICRA)}, year={2022}} ```

Reference

• We used the PPF implementation provided in MVTec HALCON software for pose hypothese generation. It is a commercial software but provides free license for student.