A High-Resolution Dataset for Instance Detection with Multi-View Instance Capture

[**Qianqian Shen**](https://shenqq377.github.io/)¹ · [**Yunhan Zhao**](https://yunhan-zhao.github.io/)² · [**Nahyun Kwon**](https://nahyunkwon.github.io/)³ · [**Jeeeun Kim**](https://github.com/qubick)³ · [**Yanan Li**](https://yananlix1.github.io/)¹ · [**Shu Kong**](https://aimerykong.github.io/)^{3,4,5 1}Zhejiang Lab ²UC Irvine ³Texas A&M University ⁴University of Macau ⁵Institute of Collaborative

The paper has been accepted by NeurIPS (Datasets and Benchmarks) 2023.

Dataset

The InsDet datase is a high-resolution real-world dataset for Instance Detection with Multi-view Instance Capture.
We provide an InsDet-mini for demo and visualization, and the full dataset InsDet-FULL.

The full dataset contains 100 objects with multi-view profile images in 24 rotation positions (per 15°), 160 testing scene images with high-resolution, and 200 pure background images. The mini version contains 5 objects, 10 testing scene images, and 10 pure background images.

Details

The Objects contains: - 000avedashampoo - images: raw RGB images (e.g., "images/001.jpg") - masks: segmentation masks generated by GrabCut Annotation Toolbox (e.g., "masks/001.png") -

$\vdots$

Tip: The first three digits specify the instance id.

The Scenes contains: - easy - leisure_zone - raw RGB images with 6144×8192 pixels (e.g. “office001/rgb_000.jpg”) - bounding box annotation for objects in test scenes generated by labelImg toolbox and using PascalVOC format (e.g. “office_001/rgb_000.xml”) - meeting_room - office_002 - pantry_room_002 - sink - hard - office_001 - pantry_room_001

Tip: Each bounding box is specified by [xmin, ymin, xmax, ymax].

The Background contains 200 pure background images that do not include any instances from Objects folder.

Code

The project is built on detectron2, segment-anything, and DINOv2.
<!-- Detectron2 provides end-to-end detectors implementation and metric evaluation. Segment-anything is an off-the-shelf class-agnostic segmentation model that we used to produce instance proposals. DINOv2 is a self-supervised vision foundation model that we used to extract feature representation. -->

Demo

The Jupyter notebooks files demonstrate our non-learned method using SAM and DINOv2. We choose light pretrained models of SAM (vitl) and DINOv2 (dinov2vits14) for efficiency. <!-- | Pretrained Model | # of params | AP | AP50 | AP75 | | :--- | :---: | :---:| :---:| :---:| | ViT-S/14 distilled | 21M |41.61 |49.10 |45.95 | |ViT-B/14 distilled | 86M |41.89 |49.39 |46.30 | |ViT-L/14 distilled | 300M |43.33 |50.80 |47.84 | |ViT-g/14 | 1,100M |44.65 |53.47 |49.11 | -->

Citation

If you find our project useful, please consider citing: bibtex @inproceedings{shen2023high, title={A high-resolution dataset for instance detection with multi-view object capture}, author={Shen, Qianqian and Zhao, Yunhan and Kwon, Nahyun and Kim, Jeeeun and Li, Yanan and Kong, Shu}, booktitle={NeurIPS Datasets & Benchmark Track}, year={2023}