Introduction

💥GussianPretrain introduces 3D Gaussian Splatting technology into vision pre-training task for the first time. Which demonstrates remarkable effectiveness and robustness, achieving significant improvements across various 3D perception tasks, including 3D object detection, HD map reconstruction, and occupancy prediction, with efficiency and lower memory consumption.💥

Qualitative Rendered Visualization

Image and Video DEMO

Rendered Image Visualization.

Framework Modules Analysis and Rendered Video Visualization.

https://github.com/user-attachments/assets/4d77ea40-0567-461e-94f5-f499198f7f8a

News

• [2025-03-05] 🚀 We incorporate our method with LiDAR modality.
• [2025-01-31] 🚀 Complement rendered visualization images and video for better clear the reconstruction performance of our approach.

• [2025-01-01] 💥 The experiments setting of UVTR-CS config and weight also released which not achieved in the paper.

• [2025-01-01] 🚀 The complete code and associated weights have been released. By the way, Happy New Year to everyone! 💥.

• [2024-11-20] The codebase is initialed. We are diligently preparing for a clean, optimized version. Stay tuned for the complete code release, which is coming soon..

Overview

💥The architecture of proposed GaussianPretrain. Given multi-view images, we first extract valid mask patches using the mask generator with the LiDAR Depth Guidance strategy. Subsequently, a set of learnable 3D Gaussian anchors is generated using ray-based guidance and conceptualized as volumetric LiDAR points. Finally, the reconstruction signals of RGB, Depth, and Occupancy are decoded based on the predicted Gaussian anchor parameters.

Main Results

3D Object Detection

HD-Map Reconstruction

Occupancy Predict

Getting Started

Installation

This project is based on MMDetection3D, which can be constructed as follows.

• Install PyTorch v1.9.1 and mmDetection3D v0.17.3 following the instructions.
• Install the required environment

``` conda create -n gaussianpretrain python=3.8 conda activate gaussianpretrain conda install pytorch==1.9.1 torchvision==0.10.1 torchaudio==0.9.1 cudatoolkit=11.1 -c pytorch -c conda-forge

pip install mmcv-full==1.3.11 -f https://download.openmmlab.com/mmcv/dist/cu111/torch1.9/index.html pip install mmdet==2.14.0 mmsegmentation==0.14.1 tifffile-2021.11.2 numpy==1.19.5 protobuf==3.19.4 scikit-image==0.19.2 pycocotools==2.0.0 nuscenes-devkit==1.0.5 spconv-cu111 gpustat numba scipy pandas matplotlib Cython shapely loguru tqdm future fire yacs jupyterlab scikit-image pybind11 tensorboardX tensorboard easydict pyyaml open3d addict pyquaternion awscli timm typing-extensions==4.7.1

cd GaussianPretrain python setup.py develop cd projects/mmdet3d_plugin/ops/diff-gaussian-rasterization python setup.py develop ```

Data Preparation

Please follow the instruction of UVTR and PanoOCC to prepare the dataset.

Training & Testing

You can train and eval the model following the instructions. For example: ```

run gaussian pretrain on 8 GPUS

bash tools/disttrain.sh projects/mmdet3dplugin/configs/gaussianpretrain/gp0.075convnext.py 8

run downstream task ft on 8 GPUS

bash tools/disttrain.sh projects/mmdet3dplugin/configs/gaussianpretrain/uvtrdnft.py 8

run eval

python tools/test.py $config $ckpt --eval bbox ```

Weights

1. Object Detection

| Method | Pretrained ckpt | Config | NDS | mAP | Model | |---------------|-----|--------------|-------|------|-------- | UVTR-C+GP | Pretrained |UVTR-C | 47.2 | 41.7 | Google | UVTR-C+GP | Pretrained |UVTR-CS | 50.0 | 42.3 | Google

2. HD-Map Reconstruction

| Method | Pretrained ckpt | Config | mAP | Model |--------------------|----------------|--------|---------|-------- | MapTR-tiny†+GP | Pretrained |MapTR-tiny | 42.42 | Google

3. Occupancy Predict

| Method | Pretrained ckpt | Config | mIoU | Model | |--------|-----------------------------------------------------------------------------------------------------|---------------|---------|---------------- |BEVFormerOCC+GP| Pretrained | BEVFormerOCC | 24.21 | Google |PanoOCC+GP| Pretrained | PanoOCC | 42.62 | Google

TODO

• streampetr version will publish soon.
• Project Page.

Acknowledgement

This project is mainly based on the following codebases. Thanks for their great works! - UVTR - UniPAD - MMDetection3D