hi-lo-sensing

High and Low Resolution Tradeoffs in Roadside Multimodal Sensing.

https://github.com/asu-suo-lab/hi-lo-sensing

Repository

High and Low Resolution Tradeoffs in Roadside Multimodal Sensing.

https://github.com/ASU-Suo-Lab/Hi-Lo-Sensing/blob/main/

[![arXiv](https://img.shields.io/badge/arXiv-Paper-.svg)](https://arxiv.org/abs/2410.01250)
# High and Low Resolution Tradeoffs in Roadside Multimodal Sensing

## Overview
- [Introduction](https://github.com/ASU-Suo-Lab/Hi-Lo-Sensing#introduction)
- [Main Results](https://github.com/ASU-Suo-Lab/Hi-Lo-Sensing#main-results)
- [Installation](https://github.com/ASU-Suo-Lab/Hi-Lo-Sensing#installation)
- [Data Preparation](https://github.com/ASU-Suo-Lab/Hi-Lo-Sensing#dataset-preparation)

## Introduction
Balancing cost and performance is crucial when choosing high- versus low-resolution point-cloud roadside sensors. Inspired by human multi-sensory integration, we propose a modular framework to assess whether reductions in spatial resolution can be compensated by informational richness in detecting traffic participants. Extensive experimental testing on the proposed framework shows that fusing velocity-encoded radar with low-resolution LiDAR yields marked gains (`14 AP` for pedestrians and an overall improvement of `1.5 mAP` across six categories) at `lower cost` than high-resolution LiDAR alone. 

![Pipeline](resources/conclusion.jpg)

Additionally, the roadside sensor placement locations, tilt angles, types, and configuration, will influence point cloud density and distribution across the coverage area. We realized a simulation tool that builds on integer programming and can automatically iterate and compare different multimodal sensor placement strategies against sensing coverage and cost jointly.

![Pipeline](resources/digital_map.jpg)

## Main results
We compared two LiDAR-only algorithms, [Pointpillars](https://arxiv.org/abs/1812.05784) and [Centerpoint](https://arxiv.org/abs/2006.11275), and four multimodal perception algorithms based on LiDAR and radar fusion, such as PointPillars_LR, [L4DR](https://arxiv.org/abs/2408.03677), [Bi-LRFusion](https://arxiv.org/abs/2408.03677) and [LiRaFusion](https://arxiv.org/abs/2402.11735).
### 3D Object Detection (on our multimodal dataset test)
|  Model  |  Modality | Resolution | mAP | Car | Truck |  Bus | G.C. | Ped. | Motor. |
|---------|-----------|------------|---- |-----|-------|------|-----------|------|--------|
|  Pointpillars    |  L  |  216beam
132beam
164beam  | 89.1
86.1
91.5 | 98.9
98.1
98.5   | 91.3
96.0
90.2   | 58.9
58.2
91.8   | 98.8
98.2
98.9   | 89.1
72.6
75.4   | 97.4
93.5
94.1   |
|  CenterPoint     |  L  |  216beam
132beam
164beam  | 94.6
91.6
92.5 | 97.8
96.7
96.7   | 92.2
93.3
93.3   | 97.8
97.8
97.8   | 95.5
95.5
95.8   | 88.9
74.4
78.9   | 95.6
92.2
92.2   |
|  Pointpillars_LR | L+R |  216beam
132beam
164beam  | 98.1
95.1
95.9 | 98.9
97.8
97.8   | 98.8
99.8
99.9   | 100.0
100.0
100.0 | 98.9
99.9
98.9  | 94.3
78.6
84.1   | 97.8
94.3
94.4   |
|  L4DR            | L+R |  216beam
132beam
164beam  | 96.3
93.2
93.8 | 98.9
98.7
98.8   | 94.8
97.3
97.6   | 99.0
99.0
99.0   | 98.9
99.1
99.0   | 88.4
71.0
73.7   | 97.7
94.4
94.4   |
|  Bi-LRFusion     | L+R |  216beam
132beam
164beam  | 98.3
95.2
95.9 |98.9
97.8
97.8   | 99.8
99.9
98.9   | 100.0
100.0
100.0 | 98.9
100.0
100.0 | 94.4
80.0
84.4  | 97.8
93.3
94.4   | 
|  LiRaFusion      | L+R |  216beam
132beam
164beam  | 97.4
94.5
95.6 |98.3
97.2
97.9   | 97.6
97.6
98.5   | 98.9
98.9
98.9   | 98.6
98.5
98.9   | 94.4
80.5
85.4   | 96.8
93.9
94.1   |
### Multi-modal Sensor Placement Optimization (based on the Test Bed in Sun Lakes, AZ, USA)
Without a standardized placement procedure and published sensor poses, mAP and other metrics are measured in incomparable setups and cannot reveal true model quality. Emphasizing roadside perception placement algorithms gives the research community a unified, repeatable geometric benchmark. Researchers can then evaluate LiDAR-only, radar-only, and multimodal-fusion methods on the same coverage-versus-cost baseline, ensuring that algorithm improvements.

![Pipeline](resources/placement.png)

## Usage
### Installation
Please refer to [mmdetection3d](https://github.com/open-mmlab/mmdetection3d) for installation.

### Dataset Preparation
You can download our [multi-modal dataset here](https://www.dropbox.com/scl/fi/nlrxxomnymx1r4n9c6742/ITSC.zip?rlkey=ni0ayjafmldf3n2qoh17apvt3&st=se0j0plj&dl=0) and unzip all the zip files. We have provided offline processed annotation files and the prefix of our .pkl files is "nuscenes_mini_infos_train(val/test)"

To our knowledge, this is **the first large-scale benchmark with multi-resolution and multi-modal dataset** for roadside perception, allowing fair comparison of multi-modal detection and fusion methods. The data are organised similarly as in [nuScenes](https://www.nuscenes.org/), allowing immediate use with existing pipelines. We can see an overview of the registered 3D box labels and their properties in the following table. It is particularly noteworthy that there are many golf carts in the real world. This type of object is hardly considered by any other perception dataset. We import it into CARLA simulation through 3D modeling and collect corresponding data for training and test.
| Class       | Labels | AvgLength | AvgWidth | AvgHeight |
|-------------|-------:|-----------:|----------:|-----------:|
| Car         | 39,169 | 4.56 | 2.06 | 1.56 |
| Truck       | 3,076  | 8.89 | 3.24 | 3.85 |
| Motorcycle  | 32,969 | 2.19 | 0.81 | 1.57 |
| Bus         | 6,032  | 5.56 | 2.31 | 2.52 |
| Pedestrian  | 51,689 | 0.50 | 0.50 | 1.73 |
| GolfCart   | 2,398  | 3.73 | 1.61 | 1.96 |
| Total       | 135,333 |  |  |  |

### Training
To train the model, use the following example command:
```
# single-gpu training
python ./tools/train.py /configs/L4DR_carla.py
```
Here are the supported models now, please change the `data_root` in config file to where you save data in your PC.

* bi_lrfusion_carla.py

* L4DR_carla.py

* lrfusion_carla.py

* yihong_carla.py

* pointpillars_hv_secfpn_1xb6_nus-3d-6class.py

* centerpoint_voxel0075_second_secfpn_1xb4-cyclic-20e_nus-6class.py

### Testing
```
# single-gpu testing
python ./tools/test.py  
```

## Acknowledgments
Our repo uses code from a few open source repositories. Without the efforts of these folks (and their willingness to release their implementations), our repo would not be possible. We thanks these authors for their efforts!
* [mmdetection3d](https://github.com/open-mmlab/mmdetection3d)
* [L4DR](https://github.com/ylwhxht/L4DR)
* [LiRaFusion](https://github.com/Song-Jingyu/LiRaFusion)
* [Bi-LRFusion](https://github.com/jessiew0806/bi-lrfusion)

Owner

GitHub Events