Data

Tools

Indexes

Applications

Experiments

Open Source Science

4dmos

Receding Moving Object Segmentation in 3D LiDAR Data Using Sparse 4D Convolutions (RAL 2022)

https://github.com/prbonn/4dmos

Science Score: 62.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ✓
    CITATION.cff file
    Found CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ✓
    Committers with academic emails
    2 of 2 committers (100.0%) from academic institutions
  • ✓
    Institutional organization owner
    Organization prbonn has institutional domain (www.ipb.uni-bonn.de)
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (17.6%) to scientific vocabulary

Keywords

4d deep-learning lidar minkowski minkowskiengine mos moving object point-cloud point-cloud-processing segmentation
Last synced: 6 months ago · JSON representation ·

Repository

Receding Moving Object Segmentation in 3D LiDAR Data Using Sparse 4D Convolutions (RAL 2022)

Basic Info
Statistics
  • Stars: 311
  • Watchers: 9
  • Forks: 29
  • Open Issues: 0
  • Releases: 2
Topics
4d deep-learning lidar minkowski minkowskiengine mos moving object point-cloud point-cloud-processing segmentation
Created over 3 years ago · Last pushed 6 months ago
Metadata Files
Readme License Citation

README.md

Receding Moving Object Segmentation in 3D LiDAR Data Using Sparse 4D Convolutions

example Our moving object segmentation on the unseen SemanticKITTI test sequences 18 and 21. Red points are predicted as moving.

Please find the corresponding video here.

Given a sequence of point clouds, our method segments moving (red) from non-moving (black) points.

We first create a sparse 4D point cloud of all points in a given receding window. We use sparse 4D convolutions from the MinkowskiEngine to extract spatio-temporal features and predict per-points moving object scores.

Important Update

Please also check out our latest map-based moving object segmentation pipeline MapMOS.

The current state of the repository is improved by internally aligning the scans using KISS-ICP. Also, the build system and pipeline have been updated after the release of our newer method MapMOS, so you can run it on most point cloud data formats. If you want to reproduce the original results from the paper, this version is tagged under 0.1. You can checkout by

bash git checkout v0.1

Installation

First, make sure the MinkowskiEngine is installed on your system, see here for more details.

Next, clone our repository bash git clone git@github.com:PRBonn/4DMOS && cd 4DMOS

and install with bash make install

or bash make install-all if you want to install the project with all optional dependencies (needed for the visualizer). In case you want to edit the Python code, install in editable mode: bash make editable

How to Use It

Just type

bash mos4d_pipeline --help to see how to run 4DMOS.

Check the Download section for a pre-trained model. Like KISS-ICP, our pipeline runs on a variety of point cloud data formats like bin, pcd, ply, xyz, rosbags, and more. To visualize these, just type

bash mos4d_pipeline --visualize /path/to/weights.ckpt /path/to/data

Want to evaluate with ground truth labels? Because these labels come in all shapes, you need to specify a dataloader. This is currently available for SemanticKITTI, NuScenes, HeLiMOS, and our labeled KITTI Tracking sequence 19 and Apollo sequences (see [Downloads](#downloads)).

Training

To train our approach, you need to first cache your data. To see how to do that, just cd into the 4DMOS repository and type

bash python3 scripts/precache.py --help

After this, you can run the training script. Again, --help shows you how: bash python3 scripts/train.py --help

Want to verify the cached data? You can inspect the cached training samples by using the script `python3 scripts/cache_to_ply.py --help`.
Want to change the logging directory? The training log and checkpoints will be saved by default to the current working directory. To change that, export the `export LOGS=/your/path/to/logs` environment variable before running the training script.

HeLiMOS

To train on the HeLiMOS data with different sensor configurations, use the following commands:

shell python3 scripts/precache.py /path/to/HeLiMOS helimos /path/to/cache --config config/helimos/*_training.yaml python3 scripts/train.py /path/to/HeLiMOS helimos /path/to/cache --config config/helimos/*_training.yaml

by replacing the paths and the config file names. To evaluate for example on the Velodyne test data, run

shell mos4d_pipeline /path/to/weights.ckpt /path/to/HeLiMOS --dataloader helimos -s Velodyne/test.txt

Evaluation and Visualization

We use the SemanticKITTI API to evaluate the intersection-over-union (IOU) of the moving class as well as to visualize the predictions. Clone the repository in your workspace, install the dependencies and then run the following command to visualize your predictions for e.g. sequence 8:

cd semantic-kitti-api ./visualize_mos.py --sequence 8 --dataset /path/to/dataset --predictions /path/to/4DMOS/predictions/ID/POSES/labels/STRATEGY/

Benchmark

To submit the results to the LiDAR-MOS benchmark, please follow the instructions here.

Downloads

Publication

If you use our code in your academic work, please cite the corresponding paper:

bibtex @article{mersch2022ral, author = {B. Mersch and X. Chen and I. Vizzo and L. Nunes and J. Behley and C. Stachniss}, title = {{Receding Moving Object Segmentation in 3D LiDAR Data Using Sparse 4D Convolutions}}, journal={IEEE Robotics and Automation Letters (RA-L)}, year = 2022, volume = {7}, number = {3}, pages = {7503--7510}, codeurl = {https://github.com/PRBonn/4DMOS}, }

Acknowledgments

This implementation is heavily inspired by KISS-ICP.

License

This project is free software made available under the MIT License. For details see the LICENSE file.

Owner

  • Name: Photogrammetry & Robotics Bonn
  • Login: PRBonn
  • Kind: organization
  • Email: cyrill.stachniss@igg.uni-bonn.de
  • Location: Bonn

Photogrammetry & Robotics Lab at the University of Bonn

Citation (CITATION.cff) 

cff-version: 1.2.0
preferred-citation:
  title: "Receding Moving Object Segmentation in 3D LiDAR Data Using Sparse 4D Convolutions"
  doi: "110.1109/LRA.2022.3183245"
  year: "2022"
  type: article
  journal: "IEEE Robotics and Automation Letters (RA-L)"
  url: https://www.ipb.uni-bonn.de/pdfs/mersch2022ral.pdf
  codeurl: https://github.com/PRBonn/4DMOS
  authors:
    - family-names: Mersch
      given-names: Benedikt
    - family-names: Chen
      given-names: Xieyuanli
    - family-names: Vizzo
      given-names: Ignacio
    - family-names: Nunes
      given-names: Lucas
    - family-names: Behley
      given-names: Jens
    - family-names: Stachniss
      given-names: Cyrill

GitHub Events

Total
  • Issues event: 9
  • Watch event: 28
  • Delete event: 8
  • Issue comment event: 33
  • Push event: 10
  • Pull request event: 16
  • Fork event: 1
  • Create event: 10
Last Year
  • Issues event: 9
  • Watch event: 28
  • Delete event: 8
  • Issue comment event: 33
  • Push event: 10
  • Pull request event: 16
  • Fork event: 1
  • Create event: 10

Committers

Last synced: 9 months ago

All Time
  • Total Commits: 36
  • Total Committers: 2
  • Avg Commits per committer: 18.0
  • Development Distribution Score (DDS): 0.028
Past Year
  • Commits: 14
  • Committers: 1
  • Avg Commits per committer: 14.0
  • Development Distribution Score (DDS): 0.0
Top Committers
Name Email Commits
Benedikt Mersch m****h@i****e 35
Xieyuanli Chen x****n@i****e 1
Committer Domains (Top 20 + Academic)
igg.uni-bonn.de: 2

Issues and Pull Requests

Last synced: 6 months ago

All Time
  • Total issues: 38
  • Total pull requests: 22
  • Average time to close issues: 9 days
  • Average time to close pull requests: 3 days
  • Total issue authors: 27
  • Total pull request authors: 1
  • Average comments per issue: 4.37
  • Average comments per pull request: 0.0
  • Merged pull requests: 22
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 6
  • Pull requests: 15
  • Average time to close issues: 7 days
  • Average time to close pull requests: 5 days
  • Issue authors: 5
  • Pull request authors: 1
  • Average comments per issue: 6.83
  • Average comments per pull request: 0.0
  • Merged pull requests: 15
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • E4CEeg (3)
  • haider8645 (3)
  • iwander-all (2)
  • momoshiki1 (2)
  • shayannikoohemat (2)
  • MaxShi007 (2)
  • Gatsby23 (2)
  • lingheng123 (2)
  • Lizhijian-123 (1)
  • Dejavu-h (1)
  • lxdunan (1)
  • Dreamdusty (1)
  • JulesSanchez (1)
  • pengboyemo (1)
  • liamlin5566 (1)
Pull Request Authors
  • benemer (26)
Top Labels
Issue Labels
Pull Request Labels

Packages

  • Total packages: 2
  • Total downloads: unknown
  • Total dependent packages: 0
    (may contain duplicates)
  • Total dependent repositories: 0
    (may contain duplicates)
  • Total versions: 2
proxy.golang.org: github.com/PRBonn/4DMOS
  • Versions: 1
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent packages count: 6.3%
Average: 6.5%
Dependent repos count: 6.7%
Last synced: 6 months ago
proxy.golang.org: github.com/prbonn/4dmos
  • Versions: 1
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent packages count: 6.3%
Average: 6.5%
Dependent repos count: 6.7%
Last synced: 6 months ago