depthfusion
Science Score: 36.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
✓Academic publication links
Links to: arxiv.org, scholar.google -
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (9.7%) to scientific vocabulary
Repository
Basic Info
- Host: GitHub
- Owner: Mingqj
- License: apache-2.0
- Language: Python
- Default Branch: main
- Size: 17.6 KB
Statistics
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
- Releases: 0
Metadata Files
README.md
DepthFusion: Depth-Aware Hybrid Feature Fusion for LiDAR-Camera 3D Object Detection
📖 About
This repository represents the official implementation of the paper titled "DepthFusion: Depth-Aware Hybrid Feature Fusion for LiDAR-Camera 3D Object Detection".
We propose a novel depth encoding strategy to guide multi-modal fusion in 3D object detection. By encoding depth, our method adaptively adjusts the modality weights at both the global and local feature levels, enabling more effective LiDAR-camera fusion across varying depth ranges.
📢 Updates
- [2025.05.06] - Released the code and model weights for DepthFusion.
- [2025.02.24] - Submitted DepthFusion to IEEE Transactions on Multimedia (TMM).
💾 Main Results
| Config | mAP | NDS | Backbone | Image size | Latency | FPS | Model| |:-----------------------------------------------------------------:|:----:|:----:|:--------:|:----------------:|:-------:|:----:|:----:| | DepthFusion-Light | 69.8 | 73.3 | ResNet18 | 256 $\times$ 704 | 72.4 | 13.8 | GoogleDrive | | DepthFusion-Base | 71.2 | 74.0 | ResNet50 | 320 $\times$ 800 | 114.9 | 8.7 | GoogleDrive | | DepthFusion-Large | 72.3 | 74.4 | SwinTiny | 384 $\times$ 1056 | 175.4 | 5.7 | GoogleDrive |
Get Started
🛠️ Installation and Data Preparation
step 1. Please prepare environment as that in Docker and run:
shell
pip install -r requirements.txt
We use the following main environment:
shell
torch 1.10.0+cu111
torchvision 0.10.0+rocm4.1
mmcls 0.25.0
mmcv-full 1.5.3
mmdet 2.25.1
mmdet3d 1.0.0rc4
mmsegmentation 0.25.0
step 2. Prepare DepthFusion repo by running:
shell script
git clone https://github.com/Mingqj/DepthFusion.git
cd DepthFusion
pip install -v -e .
step 3. Prepare nuScenes dataset and create the pkl for DepthFusion by running:
shell
python tools/create_data_bevdet.py
step 4. Arrange the folder as:
shell script
DepthFusion
└──data
└── nuscenes
├── v1.0-trainval
├── sweeps
├── samples
└── gts
🏋️ Train DepthFusion model
```shell
single gpu
python tools/train.py /configs/depthfusion/depthfuison-light.py # light version python tools/train.py /configs/depthfusion/depthfuison-base.py # base version python tools/train.py /configs/depthfusion/depthfuison-large.py # large version
multiple gpu
./tools/disttrain.sh /configs/depthfusion/depthfuison-light.py 8 # light version ./tools/disttrain.sh /configs/depthfusion/depthfuison-base.py 8 # base version ./tools/dist_train.sh /configs/depthfusion/depthfuison-large.py 8 # large version
```
📋 Test DepthFusion model
```shell
single gpu
python tools/test.py /configs/depthfusion/depthfuison-light.py $checkpoint --eval mAP # light version python tools/test.py /configs/depthfusion/depthfuison-base.py $checkpoint --eval mAP # base version python tools/test.py /configs/depthfusion/depthfuison-large.py $checkpoint --eval mAP # large version
multiple gpu
./tools/disttest.sh /configs/depthfusion/depthfuison-light.py $checkpoint 8 --eval mAP # light version ./tools/disttest.sh /configs/depthfusion/depthfuison-base.py $checkpoint 8 --eval mAP # base version ./tools/dist_test.sh /configs/depthfusion/depthfuison-large.py $checkpoint 8 --eval mAP # large version ```
👀 Visualize the predicted result.
```shell
light version
python tools/test.py /configs/depthfusion/depthfuison-light.py $checkpoint --format-only --eval-options jsonfileprefix=$savepath python tools/analysistools/vis.py $savepath/ptsbbox/resultsnusc.json
base version
python tools/test.py /configs/depthfusion/depthfuison-base.py $checkpoint --format-only --eval-options jsonfileprefix=$savepath python tools/analysistools/vis.py $savepath/ptsbbox/resultsnusc.json
large version
python tools/test.py /configs/depthfusion/depthfuison-large.py $checkpoint --format-only --eval-options jsonfileprefix=$savepath python tools/analysistools/vis.py $savepath/ptsbbox/resultsnusc.json ```
❛❛❞ Citation
bibtex
@article{ji2025depthfusion,
title={DepthFusion: Depth-Aware Hybrid Feature Fusion for LiDAR-Camera 3D Object Detection},
author={Mingqian Ji and Jian Yang and Shanshan Zhang},
journal={arXiv preprint arXiv:2505.07398},
year={2025},
}
❤️ Acknowledgement
We thank these great works and open-source codebases: MMDetection3D, BEVDet, CenterPoint, Lift-Splat-Shoot, Swin Transformer, BEVFusion, BEVDepth, nuScenes-C
Owner
- Name: Doubleit
- Login: Mingqj
- Kind: user
- Repositories: 1
- Profile: https://github.com/Mingqj
GitHub Events
Total
- Issues event: 4
- Watch event: 3
- Issue comment event: 2
- Push event: 77
- Create event: 2
Last Year
- Issues event: 4
- Watch event: 3
- Issue comment event: 2
- Push event: 77
- Create event: 2
Dependencies
- nvcr.io/nvidia/tensorrt 22.07-py3 build
- pytorch/pytorch ${PYTORCH}-cuda${CUDA}-cudnn${CUDNN}-devel build
- docutils ==0.16.0
- m2r *
- mistune ==0.8.4
- myst-parser *
- sphinx ==4.0.2
- sphinx-copybutton *
- sphinx_markdown_tables *
- mmcv-full >=1.4.8,<=1.6.0
- mmdet >=2.24.0,<=3.0.0
- mmsegmentation >=0.20.0,<=1.0.0
- open3d *
- spconv *
- waymo-open-dataset-tf-2-1-0 ==1.2.0
- mmcv >=1.4.8
- mmdet >=2.24.0
- mmsegmentation >=0.20.1
- torch *
- torchvision *
- distortion *
- einops *
- h5py *
- imgaug *
- lyft_dataset_sdk *
- networkx >=2.2,<2.3
- numba ==0.53.0
- numpy *
- nuscenes-devkit *
- plyfile *
- scikit-image *
- tensorboard *
- trimesh >=2.35.39,<2.35.40
- asynctest * test
- codecov * test
- flake8 * test
- interrogate * test
- isort * test
- kwarray * test
- pytest * test
- pytest-cov * test
- pytest-runner * test
- ubelt * test
- xdoctest >=0.10.0 test
- yapf * test