EMIFF: Enhanced Multi-scale Image Feature Fusion for Vehicle-Infrastructure Cooperative 3D Object Detection

Project page | Paper | VIMI |

EMIFF: Enhanced Multi-scale Image Feature Fusion for Vehicle-Infrastructure Cooperative 3D Object Detection Zhe Wang, Siqi Fan, Xiaoliang Huo, Tongda Xu, Yan Wang, Jingjing Liu, Yilun Chen, Ya-Qin Zhang.ICRA 2024.

This repository contains the official Pytorch implementation of training & evaluation code and the pretrained models for EMIFF/VIMI.

Abstract

In autonomous driving, cooperative perception makes use of multi-view cameras from both vehicles and infrastructure, providing a global vantage point with rich semantic context of road conditions beyond a single vehicle viewpoint. Currently, two major challenges persist in vehicle-infrastructure cooperative 3D (VIC3D) object detection: $1)$ inherent pose errors when fusing multi-view images, caused by time asynchrony across cameras; $2)$ information loss in transmission process resulted from limited communication bandwidth. To address these issues, we propose a novel camera-based 3D detection framework for VIC3D task, Enhanced Multi-scale Image Feature Fusion (EMIFF). To fully exploit holistic perspectives from both vehicles and infrastructure, we propose Multi-scale Cross Attention (MCA) and Camera-aware Channel Masking (CCM) modules to enhance infrastructure and vehicle features at scale, spatial, and channel levels to correct the pose error introduced by camera asynchrony. We also introduce a Feature Compression (FC) module with channel and spatial compression blocks for transmission efficiency. Experiments show that EMIFF achieves SOTA on DAIR-V2X-C datasets, significantly outperforming previous early-fusion and late-fusion methods with comparable transmission costs.

Methods

Get Started

• Installation
• Prepare Dataset
• Train and Test

Benchmark and Model Zoo

Modality:Image

| Fusion | Method| Dataset | AP-3D (IoU=0.5) | AP-BEV (IoU=0.5) |Config|DownLoad| | :-----: | :--------: | :-------: | :----: | :----: | :----: | :-----: |
| Only-Veh | ImvoxelNet | VIC-Sync | 7.29 | 8.85 | config |\ | | Only-Inf | ImvoxelNet | VIC-Sync | 8.66 | 14.41 | config |\ | | Late-Fusion | ImvoxelNet | VIC-Sync | 11.08 | 14.76 | \ | \ | | Early-Fusion | BEVFormer_S | VIC-Sync | 8.80 | 13.45 | config | model/log|
| Early-Fusion | ImVoxelNet | VIC-Sync | 12.72 | 18.17 | config | model/log|
| Intermediate-Fusion| EMIFF | VIC-Sync | 15.61 | 21.44 | config | model/log |

We evaluate Only-Veh/Only-Inf/Late-Fusion model following OpenDAIRV2X.

Acknowledgement

This project is not possible without the following codebases. * OpenDAIRV2X * MMDetection3D <!-- * pypcd -->

Citation

If you find our work useful in your research, please consider citing:

``` @misc{wang2023vimi, title={VIMI: Vehicle-Infrastructure Multi-view Intermediate Fusion for Camera-based 3D Object Detection}, author={Zhe Wang and Siqi Fan and Xiaoliang Huo and Tongda Xu and Yan Wang and Jingjing Liu and Yilun Chen and Ya-Qin Zhang}, year={2023}, eprint={2303.10975}, archivePrefix={arXiv}, primaryClass={cs.CV} }

@inproceedings{wang2024emiff, title={EMIFF: Enhanced Multi-scale Image Feature Fusion for Vehicle-Infrastructure Cooperative 3D Object Detection}, author={Zhe Wang and Siqi Fan and Xiaoliang Huo and Tongda Xu and Yan Wang and Jingjing Liu and Yilun Chen and Ya-Qin Zhang}, booktitle = {2024 IEEE International Conference on Robotics and Automation (ICRA)}, year = {2024}} } ```