https://github.com/christophreich1996/smurf
PyTorch port (inference only) of the paper "SMURF: Self-Teaching Multi-Frame Unsupervised RAFT with Full-Image Warping" [CVPR 2021].
Science Score: 10.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
○codemeta.json file
-
○.zenodo.json file
-
○DOI references
-
✓Academic publication links
Links to: arxiv.org -
○Committers with academic emails
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (12.2%) to scientific vocabulary
Keywords
Repository
PyTorch port (inference only) of the paper "SMURF: Self-Teaching Multi-Frame Unsupervised RAFT with Full-Image Warping" [CVPR 2021].
Basic Info
- Host: GitHub
- Owner: ChristophReich1996
- License: cc-by-4.0
- Language: Python
- Default Branch: main
- Homepage: https://openaccess.thecvf.com/content/CVPR2021/papers/Stone_SMURF_Self-Teaching_Multi-Frame_Unsupervised_RAFT_With_Full-Image_Warping_CVPR_2021_paper.pdf
- Size: 3.71 MB
Statistics
- Stars: 3
- Watchers: 2
- Forks: 0
- Open Issues: 0
- Releases: 0
Topics
Metadata Files
README.md
SMURF: Self-Teaching Multi-Frame Unsupervised RAFT with Full-Image Warping
Unofficial PyTorch (inference only) reimplementation of the CVPR 2021 paper SMURF: Self-Teaching Multi-Frame Unsupervised RAFT with Full-Image Warping by Austin Stone et al. (Google Research).
Figure taken from paper.
Requirements
To perform inference with the SMURF RAFT model it is just required to install PyTorch and TorchVision.
See requirements.txt for details.
Note: to convert the original checkpoints it is also required to install Tensorflow 2.0.
Port weights
As the official SMURF implementation does not provide a license for their checkpoints this repo does not include the
converted PyTorch checkpoints. However, you can convert the official Tensorflow checkpoints with the
provided convert_weights_to_pt.py script.
You can download the original checkpoints here.
For converting an original checkpoint (e.g. Kitti checkpoint) to PyTorch just run:
bash
python convert_weights_to_pt.py --tf_checkpoint "path/to/checkpoint/smurf-kitti-smurf-ckpt-31" --pt_checkpoint_path "smurf_kitti.pt"
Perform inference
To load the converted checkpoint and perform inference you can run:
```python from typing import List
import torch.nn as nn import torchvision from torch import Tensor
from smurf import raft_smurf
Load images
image1: Tensor = torchvision.io.readimage("toydata/reds/00000000.png", mode=torchvision.io.ImageReadMode.RGB) image2: Tensor = torchvision.io.readimage("toydata/reds/00000004.png", mode=torchvision.io.ImageReadMode.RGB)
Normalize images to the pixe range of [-1, 1]
image1 = 2.0 * (image1 / 255.0) - 1.0 image2 = 2.0 * (image2 / 255.0) - 1.0
Init SMURF RAFT model
model: nn.Module = raftsmurf(checkpoint="smurfkitti.pt")
Predict optical flow
optical_flow: List[Tensor] = model(image1[None], image2[None]) ```
The resulting flow should look like this:
A full inference script with flow visualization is provided in perform_inference.py.
Implementation details
This implementation is mainly based on TorchVision's RAFT implementation (BSD 3-Clause License). However, the official SMURF RAFT implementation contains minor implementation differences. This implementation has modified TorchVision's RAFT implementation to match the official SMURF RAFT implementation. For more details please refer to this GitHub issue.
Reference
bibtex
@inproceedings{Stone2021,
title={SMURF: Self-Teaching Multi-Frame Unsupervised RAFT with Full-Image Warping},
author={Stone, Austin and Maurer, Daniel and Ayvaci, Alper and Angelova, Anelia and Jonschkowski, Rico},
booktitle={CVPR},
year={2021}
}
bibtex
@inproceedings{Nah2019,
author={Nah, Seungjun and Baik, Sungyong and Hong, Seokil and Moon, Gyeongsik and Son, Sanghyun and Timofte, Radu and Lee, Kyoung Mu},
title={NTIRE 2019 Challenge on Video Deblurring and Super-Resolution: Dataset and Study},
booktitle={CVPRW},
year={2019}
}
Owner
- Name: Christoph Reich
- Login: ChristophReich1996
- Kind: user
- Location: Germany
- Company: Technical University of Munich
- Website: christophreich1996.github.io
- Twitter: ChristophR1996
- Repositories: 41
- Profile: https://github.com/ChristophReich1996
ELLIS Ph.D. Student @ Technical University of Munich, Technische Universität Darmstadt & University of Oxford | Prev. NEC Labs
GitHub Events
Total
- Watch event: 4
- Fork event: 1
Last Year
- Watch event: 4
- Fork event: 1
Committers
Last synced: about 2 years ago
Top Committers
| Name | Commits | |
|---|---|---|
| ChristophReich1996 | c****h@g****t | 13 |
| Christoph Reich | 3****6 | 4 |
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: about 2 years ago
All Time
- Total issues: 0
- Total pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Total issue authors: 0
- Total pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
Pull Request Authors
Top Labels
Issue Labels
Pull Request Labels
Dependencies
- torch >=2.0.0
- torchvision >=0.15