A Toolbox for Video Restoration

Authors

Jing Lin, Yuanhao Cai, Xiaowan Hu, Haoqian Wang, Youliang Yan, Xueyi Zou, Henghui Ding, Yulun Zhang, Radu Timofte, and Luc Van Gool

News

• 2022.12.08 : Pretrained model, training/testing log, visual results of FGST on GoPro and DVD dataset are released. S2SVR will be provided later.🔥

• 2022.11.30 : Data preparation codes of GoPro and DVD are provided. :high_brightness:

• 2022.08.05 : Pretrained model of FGST on GOPRO dataset is released. :dizzy:

• 2022.05.14 : Our FGST and S2SVR are accepted by ICML2022. :rocket:

| Super-Resolution | Deblur | Compressed Video Enhancement | | :--------------------------------------------------: | :------------------------------------------------------: | :----------------------------------------------------------: | | | | |

Papers

• Flow-Guided Sparse Transformer for Video Deblurring (ICML 2022)
• Unsupervised Flow-Aligned Sequence-to-Sequence Learning for Video Restoration (ICML 2022)

| Method | Dataset | Pretrained Model | Training Log | Testing Log | Visual Result | Quantitative Result | | :--------------------------------------: | :----------------------------------------------------------: | :----------------------------------------------------------: | :----------------------------------------------------------: | :----------------------------------------------------------: | :----------------------------------------------------------: | :------------------: | | FGST | GoPro | Google Drive / Baidu Disk | Google Drive / Baidu Disk | Google Drive / Baidu Disk | Google Drive / Baidu Disk | 33.02 / 0.947 | | FGST | DVD | Google Drive / Baidu Disk | Google Drive / Baidu Disk | Google Drive / Baidu Disk | Google Drive / Baidu Disk | 33.50 / 0.945 |

Note: access code for Baidu Disk is VR11

1. Create Environment:

• Python 3 (Recommend to use Anaconda)

• NVIDIA GPU + CUDA

• Python packages:

shell pip install torchvision==0.9.0 torch==1.8.0 torchaudio==0.8.0 pip install -r requirements.txt pip install openmim mim install mmcv-full==1.5.0 pip install -v -e . pip install cupy-cuda101==7.7.0

2. Prepare Dataset:

Download the datasets (GOPRO,DVD,REDS,VIMEO,MFQE-v2) and and recollect them as the following form:

shell |--VR-Baseline |--data |-- GoPro |-- test |-- train |-- DVD |-- quantitative_datasets |-- GT |-- LQ |-- qualitative_datasets |-- REDS |-- train_sharp_bicubic |-- train_sharp |-- VIMEO |-- BIx4 |-- GT |-- MFQEV2 |-- test |-- train

You can run the following command to recollect GoPro and DVD dataset:

```shell cd VR-Baseline/data_preparation

recollect GoPro dataset

python GoProUtil.py --inputpath INPUTPATH --savepath SAVE_PATH

recollect DVD dataset

python DVDUtil.py --inputpath INPUTPATH --savepath SAVE_PATH ```

You need to replace INPUT_PATH and SAVE_PATH with your own path.

3. Training:

```shell cd VR_Baseline

training FGST on GoPro dataset

bash tools/disttrain.sh configs/FGSTdeblur_gopro.py 8

training FGST on DVD dataset

bash tools/disttrain.sh configs/DVDdeblur_gopro.py 8

training S2SVR on GoPro dataset

bash tools/disttrain.sh configs/S2SVRdeblur_gopro.py 8

training S2SVR on REDS dataset

bash tools/disttrain.sh configs/S2SVRsr_reds4.py 8

training S2SVR on VIMEO dataset

bash tools/disttrain.sh configs/S2SVRsr_vimeo.py 8

training S2SVR on MFQEv2 dataset

bash tools/disttrain.sh configs/S2SVRvqe_mfqev2.py 8 ```

The training log, trained model will be available in VR-Baseline/experiments/ .

4. Testing:

Download pretrained model and run the following command.

To test on benchmark:

```shell cd VR_Baseline

testing FGST on GoPro dataset

bash tools/disttrain.sh configs/FGSTdeblurgoprotest.py 8

testing FGST on DVD dataset

bash tools/disttrain.sh configs/FGSTdeblurdvdtest.py 8 ```

5. TODO

These works are mostly done during the internship at HUAWEI Noah's Ark Lab. Due to the limitation of company regulations, the original pre-trained models can not be transferred and published here. We will retrain more models and open-source them when we have enough GPUs as soon as possible.

• [ ] More data preparation codes
• [ ] More Pretrained Models
• [ ] Inference Results
• [ ] MFQEv2 dataloader

6. Acknowledgement.

We refer to codes from BasicVSR++ and mmediting. Thanks for their awesome works.

7. Citation

If this repo helps you, please consider citing our works:

```shell

FGST

@inproceedings{fgst, title={Flow-Guided Sparse Transformer for Video Deblurring}, author={Lin, Jing and Cai, Yuanhao and Hu, Xiaowan and Wang, Haoqian and Yan, Youliang and Zou, Xueyi and Ding, Henghui and Zhang, Yulun and Timofte, Radu and Van Gool, Luc}, booktitle={ICML}, year={2022} }

S2SVR

@inproceedings{seq2seq, title={Unsupervised Flow-Aligned Sequence-to-Sequence Learning for Video Restoration}, author={Lin, Jing and Hu, Xiaowan and Cai, Yuanhao and Wang, Haoqian and Yan, Youliang and Zou, Xueyi and Zhang, Yulun and Van Gool, Luc}, booktitle={ICML}, year={2022} } ```