Super-Resolution Residual Convolutional Generative Adversarial Network

Update: Our improved Real-Image SR method titled "Deep Cyclic Generative Adversarial Residual Convolutional Networks for Real Image Super-Resolution (SRResCycGAN)" is appeared in the AIM-2020 ECCV workshop. [Project Website]

An official PyTorch implementation of the SRResCGAN model as described in the paper Deep Generative Adversarial Residual Convolutional Networks for Real-World Super-Resolution. This work is participated in the CVPRW NTIRE 2020 RWSR challenges on the Real-World Super-Resolution.

✨ Visual examples:

• Abstract
• Spotlight Video
• Paper
• Pre-trained Models
• Citation
• Quick Test
• Train models
• SRResCGAN Architecture
• Quantitative Results
• Visual Results
• Code Acknowledgement

Abstract

Most current deep learning based single image super-resolution (SISR) methods focus on designing deeper / wider models to learn the non-linear mapping between low-resolution (LR) inputs and the high-resolution (HR) outputs from a large number of paired (LR/HR) training data. They usually take as assumption that the LR image is a bicubic down-sampled version of the HR image. However, such degradation process is not available in real-world settings i.e. inherent sensor noise, stochastic noise, compression artifacts, possible mismatch between image degradation process and camera device. It reduces significantly the performance of current SISR methods due to real-world image corruptions. To address these problems, we propose a deep Super-Resolution Residual Convolutional Generative Adversarial Network (SRResCGAN) to follow the real-world degradation settings by adversarial training the model with pixel-wise supervision in the HR domain from its generated LR counterpart. The proposed network exploits the residual learning by minimizing the energy-based objective function with powerful image regularization and convex optimization techniques. We demonstrate our proposed approach in quantitative and qualitative experiments that generalize robustly to real input and it is easy to deploy for other down-scaling operators and mobile/embedded devices.

Spotlight Video

Paper

Pre-trained Models

| |DSGAN|SRResCGAN| |---|:---:|:---:| |NTIRE2020 RWSR|Source-Domain-Learning|SR-learning|

BibTeX

@InProceedings{Umer_2020_CVPR_Workshops,
    author = {Muhammad Umer, Rao and Luca Foresti, Gian and Micheloni, Christian},
    title = {Deep Generative Adversarial Residual Convolutional Networks for Real-World Super-Resolution},
    booktitle = {The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
    month = {June},
    year = {2020}
    }

Cite the code repos

Use the following CITATION.cff file to cite the SRResCGAN code repository.

Quick Test

This model can be run on arbitrary images with a Docker image hosted on Replicate: https://beta.replicate.ai/RaoUmer/SRResCGAN. Below are instructions for how to run the model without Docker:

Dependencies

• Python 3.7 (version >= 3.0)
• PyTorch >= 1.0 (CUDA version >= 8.0 if installing with CUDA.)
• Python packages: pip install numpy opencv-python

Test models

1. Clone this github repository. git clone https://github.com/RaoUmer/SRResCGAN cd SRResCGAN cd srrescgan_code_demo
2. Place your own low-resolution images in ./srrescgan_code_demo/LR folder. (There are two sample images i.e. 0815 and 0829).
3. Download pretrained models from Google Drive. Place the models in ./srrescgan_code_demo/trained_nets_x4. We provide two models with source domain learning and SR learning.
4. Run test. We provide SRResCGAN/SRResCGAN+ and you can config in the test_srrescgan.py (without self-ensemble strategy) / test_srrescgan_plus.py (with self-ensemble strategy). python test_srrescgan.py python test_srrescgan_plus.py
5. The results are in ./srrescgan_code_demo/sr_results_x4 folder.

Train models

• The SR training code is available in the training_codes directory.

SRResCGAN Architecture

Overall Representative diagram

SR Generator Network

Quantitative Results

| _{Dataset (HR/LR pairs)} | _{SR methods} | _#Params | _PSNR↑ | _SSIM↑ | _LPIPS↓ | _Artifacts | |:---:|:---:|:---:|:---:|:---:|:---:|:---:| | _Bicubic| _EDSR| _43M |_24.48|_0.53|_0.6800|_{Sensor noise (σ = 8)}| | _Bicubic| _EDSR| _43M |_23.75|_0.62|_0.5400|_{JPEG compression (quality=30)}| | _Bicubic| _ESRGAN| _16.7M |_17.39|_0.19|_0.9400|_{Sensor noise (σ = 8)}| | _Bicubic| _ESRGAN| _16.7M |_22.43|_0.58|_0.5300|_{JPEG compression (quality=30)}| | _CycleGAN| _ESRGAN-FT| _16.7M |_22.42|_0.55|_0.3645|_{Sensor noise (σ = 8)}| | _CycleGAN| _ESRGAN-FT| _16.7M |_22.80|_0.57|_0.3729|_{JPEG compression (quality=30)}| | _DSGAN| _ESRGAN-FS| _16.7M |_22.52|_0.52|_0.3300|_{Sensor noise (σ = 8)}| | _DSGAN| _ESRGAN-FS| _16.7M |_20.39|_0.50|_0.4200|_{JPEG compression (quality=30)}| | _DSGAN| _{SRResCGAN (ours)}| _380K |_25.46|_0.67|_0.3604|_{Sensor noise (σ = 8)}| | _DSGAN| _{SRResCGAN (ours)}| _380K |_23.34|_0.59|_0.4431|_{JPEG compression (quality=30)}| | _DSGAN| _{SRResCGAN+ (ours)}| _380K |_26.01|_0.71|_0.3871|_{Sensor noise (σ = 8)}| | _DSGAN| _{SRResCGAN+ (ours)}| _380K |_23.69|_0.62|_0.4663|_{JPEG compression (quality=30)}| | _DSGAN| _{SRResCGAN (ours)}| _380K |_25.05|_0.67|_0.3357|_{unknown (validset)}| | _DSGAN| _{SRResCGAN+ (ours)}| _380K |_25.96|_0.71|_0.3401|_{unknown (validset)}| | _DSGAN| _ESRGAN-FS| _16.7M |_20.72|_0.52|_0.4000|_{unknown (testset)}| | _DSGAN| _{SRResCGAN (ours)}| _380K |_24.87|_0.68|_0.3250|_{unknown (testset)}|

The NTIRE2020 RWSR Challenge Results (Track-1)

| _Team | _PSNR↑ | _SSIM↑ | _LPIPS↓ | _MOS↓ | |:---:|:---:|:---:|:---:|:---:| | _{Impressionism}| _{24.67 (16)} |_{0.683 (13)}|_{0.232 (1)}|_2.195| | _{Samsung-SLSI-MSL}| _{25.59 (12)} |_{0.727 (9)} |_{0.252 (2)}|_2.425| | _BOE-IOT-AIBD| _{26.71 (4)} |_{0.761 (4)}|_{0.280 (4)}|_2.495| | _MSMers| _{23.20 (18)} |_{0.651 (17)}|_{0.272 (3)}|_2.530| | _KU-ISPL| _{26.23 (6)} |_{0.747 (7)}|_{0.327 (8)}|_2.695| | _InnoPeak-SR| _{26.54 (5)} |_{0.746 (8)}|_{0.302 (5)}|_2.740| | _ITS425| _{27.08 (2)} |_{0.779 (1)}|_{0.325 (6)}|_2.770| | _{MLP-SR (ours)}| _{24.87 (15)} |_{0.681 (14)}|_{0.325 (7)}|_2.905| | _Webbzhou| _{26.10 (9)} |_{0.764 (3)}|_{0.341 (9)}|_-| | _SR-DL| _{25.67 (11)} |_{0.718 (10)}|_{0.364 (10)}|_-| | _TeamAY| _{27.09 (1)} |_{0.773 (2)}|_{0.369 (11)}|_-| | _{BIGFEATURE-CAMERA}| _{26.18 (7)} |_{0.750 (6)}|_{0.372 (12)}|_-| | _{BMIPL-UNIST-YH-1}| _{26.73 (3)} |_{0.752 (5)}|_{0.379 (13)}|_-| | _SVNIT1-A| _{21.22 (19)} |_{0.576 (19)}|_{0.397 (14)}|_-| | _KU-ISPL2| _{25.27 (14)} |_{0.680 (15)}|_{0.460 (15)}|_-| | _SuperT| _{25.79 (10)} |_{0.699 (12)}|_{0.469 (16)}|_-| | _GDUT-wp| _{26.11 (8)} |_{0.706 (11)}|_{0.496 (17)}|_-| | _SVNIT1-B| _{24.21 (17)} |_{0.617 (18)}|_{0.562 (18)} |_-| | _SVNIT2| _{25.39 (13)} |_{0.674 (16)}|_{0.615 (19)}|_-| | _AITA-Noah-A| _{24.65 (-)} |_{0.699 (-)}|_{0.222 (-)}|_2.245| | _AITA-Noah-B| _{25.72 (-)} |_{0.737 (-)}|_{0.223 (-)}|_2.285| | _Bicubic| _{25.48 (-)} |_{0.680 (-)}|_{0.612 (-)}|_3.050| | _{ESRGAN Supervised}| _{24.74 (-)} |_{0.695 (-)}|_{0.207 (-)}|_2.300|

Visual Results

Validation-set (Track-1)

You can download all the SR resutls of our method on the validation-set from Google Drive: SRResCGAN, SRResCGAN+.

Test-set (Track-1)

You can download all the SR resutls of our method on the test-set from Google Drive: SRResCGAN, SRResCGAN+.

Real-World Smartphone images (Track-2)

You can download all the SR resutls of our method on the smartphone images from Google Drive: SRResCGAN, SRResCGAN+.

Code Acknowledgement

The training and testing codes are somewhat based on ESRGAN, DSGAN, and deep_demosaick.