Introduction

Video prediction ('VP') is the task of predicting future frames given some context frames.

Like with most Computer Vision sub-domains, scientific contributions in this field exhibit a high variance in the following aspects: - Training protocol (dataset usage, when to backprop, value ranges etc.) - Technical details of model implementation (deep learning framework, package dependencies etc.) - Benchmark selection and execution (this includes the choice of dataset, number of context/predicted frames, skipping frames in the observed sequences etc.) - Evaluation protocol (metrics chosen, variations in implementation/reduction modes, different ways of creating visualizations etc.)

Furthermore, while many contributors nowadays do share their code, seemingly minor missing parts such as dataloaders etc. make it much harder to assess, compare and improve existing models.

This repo aims at providing a suite that facilitates scientific work in the subfield, providing standardized yet customizable solutions for the aspects mentioned above. This way, validating existing VP models and creating new ones hopefully becomes much less tedious.

Installation

Requires pip and python >= 3.6 (code is tested with version 3.8).

Note: If using zsh while installing the package with extras, be sure to quote the arguments like i.e. like this: pip install -e '.[dev]'.

Usage

Notes:

• Use VPSuite.list_available_models() and VPSuite.list_available_datasets() to get an overview of which models and datasets are currently covered by the framework.
• All training, testing and hyperparametrization calls can be heavily configured (adjusting training hyperparameters, logging behavior etc, ...). For a comprehensive list of all adjustable run configuration parameters see the documentation of the vp_suite.defaults package.

Customization

This package is designed with quick extensibility in mind. See the sections below for how to add new components (models, model blocks, datasets or measures).

Notes:

• If you omit the docstring for a particular attribute/method/field, the docstring of the base class is used for documentation.
• If implementing components that originate from publications/public repositories, please override the corresponding constants to specify the source! Additionally, if you want to write automated tests checking implementation equality, have a look at how tests/test_impl_match.py fetches the tests of tests/test_impl_match/ and executes these tests.
• Basic unit tests for models, datasets and measures are executed on all registered models - you don't need to write such basic tests for your custom components! Same applies for documentation: The tables that list available components are filled automatically.

Contributing

This project is always open to extension! It grows especially powerful with more models and datasets, so if you've made your code work on custom models/datasets/metrics/etc., feel free to submit a merge request!

Other kinds of contributions are also very welcome - just check the open issues on the tracker or open up a new issue there.

Unit Testing

When submitting a merge request, please make sure all tests run through (execute from root folder): python -m pytest --runslow --cov=vp_suite -rs Note: this is the easiest way to run all tests without import hassles. You will need to have vp-suite installed in development mode, though (see here).

API Documentation

The official API documentation is updated automatically upon push to the main branch. If you want to build the documentation locally, make sure you've installed the package accordingly and execute the following: cd docs/ bash assemble_docs.sh

Citing

Please consider citing if you find our findings or our repository helpful. @article{karapetyan_VideoPrediction_2022, title = {Video Prediction at Multiple Scales with Hierarchical Recurrent Networks}, author = {Karapetyan, Ani and Villar-Corrales, Angel and Boltres, Andreas and Behnke, Sven}, journal={arXiv preprint arXiv:2203.09303}, year={2022} }

Acknowledgements

• Project structure is inspired by segmentation_models.pytorch.
• Sphinx-autodoc templates are inspired by the QNET repository.

All other sources are acknowledged in the documentation of the respective point of usage (to the best of our knowledge).

License

This project comes with an MIT License, except for the following components:

• Module vp_suite.measure.fvd.pytorch_i3d (Apache 2.0 License, taken and modified from here)

Disclaimer

I do not host or distribute any dataset. For all provided dataset functionality, I trust you have the permission to download and use the respective data.