Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/cv-stuttgart/m-fuse

[WACV 2023] M-FUSE: Multi-frame Fusion for Scene Flow Estimation

https://github.com/cv-stuttgart/m-fuse

Science Score: 10.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ○
    codemeta.json file
  • ○
    .zenodo.json file
  • ○
    DOI references
  • ✓
    Academic publication links
    Links to: arxiv.org
  • ○
    Academic email domains
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (9.6%) to scientific vocabulary

Keywords

computervision sceneflow
Last synced: 5 months ago · JSON representation

Repository

[WACV 2023] M-FUSE: Multi-frame Fusion for Scene Flow Estimation

Basic Info
  • Host: GitHub
  • Owner: cv-stuttgart
  • License: bsd-3-clause
  • Language: Python
  • Default Branch: main
  • Homepage:
  • Size: 31.3 KB
Statistics
  • Stars: 13
  • Watchers: 4
  • Forks: 2
  • Open Issues: 0
  • Releases: 0
Topics
computervision sceneflow
Created over 3 years ago · Last pushed over 3 years ago
Metadata Files
Readme License

README.md

M-FUSE: Multi-frame Fusion for Scene Flow Estimation

This repository contains the official code for our paper

M-FUSE: Multi-frame Fusion for Scene Flow Estimation
L. Mehl, A. Jahedi, J. Schmalfuss, A. Bruhn
Winter Conference on Applications of Computer Vision (WACV), 2023.

@inproceedings{Mehl2023, title={{M-FUSE}: Multi-frame Fusion for Scene Flow Estimation}, author={Mehl, Lukas and Jahedi, Azin and Schmalfuss, Jenny and Bruhn, Andr{\'e}s}, booktitle={Proc. Winter Conference on Applications of Computer Vision (WACV)}, year={2023} }

Code Overview: - data_readers: code related to data handling - mfuse: model definitions - scripts: scripts for training, evaluation, submission

Setup

  • Install all required python packages: pytorch, numpy, scipy, opencv, tqdm, scikit-sparse, pypng
  • Install the lietorch package. See https://github.com/princeton-vl/lietorch for details.

The code was tested with Python 3.9, PyTorch 1.10.2, CUDA 11.6

Datasets

Download the KITTI scene flow dataset with the multi-frame extension from http://www.cvlibs.net/datasets/kitti/evalsceneflow.php and make sure that it is located in the directory $DATASETS/kitti15 where $DATASETS is an environment variable.

Create disparity files for KITTI using the LEAStereo code https://github.com/XuelianCheng/LEAStereo and put them into $DATASETS/kitti15/training/disp_lea and $DATASETS/kitti15/testing/disp_lea respectively. You can also download precomputed results of LEAStereo for the train and testing split.

Usage

After training M-FUSE on the KITTI dataset for 50K steps, results can be evaluated using python scripts/evaluation_fusion.py --model=<path-to-checkpoint>.pth

A submission for the KITTI benchmark can be created using python scripts/kitti_submission_fusion.py --model=<path-to-checkpoint>.pth

Our resulting checkpoint can be downloaded here, which yields an SF-all error of 4.83 on the KITTI benchmark.

Training

  1. Retrain the RAFT-3D model raft3d_bilaplacian on the FlyingThings3D dataset for 200K steps with their provided code https://github.com/princeton-vl/RAFT-3D or use their checkpoint.

  2. Train our fusion model: python scripts/train_fusion.py --ckpt_r3d=<path-to-pretrained-r3d>

Owner

  • Name: Computer Vision Group
  • Login: cv-stuttgart
  • Kind: organization

Computer Vision Group at the University of Stuttgart

GitHub Events

Total
  • Watch event: 1
Last Year
  • Watch event: 1