DLGSANet: Lightweight Dynamic Local and Global Self-Attention Networks for Image Super-Resolution

Project Page | Paper (ArXiv) | Supplemental Material

This repository is the official pytorch implementation of our paper, DLGSANet: Lightweight Dynamic Local and Global Self-Attention Networks for Image Super-Resolution.

Xiang Li¹, Jinshan Pan¹, Jinhui Tang¹, Jiangxin Dong¹

¹IMAG Lab, Nanjing University of Science and Technology

Abstract: We propose an effective lightweight dynamic local and global self-attention network (DLGSANet) to solve image super-resolution. Our method explores the properties of Transformers while having low computational costs. Motivated by the network designs of Transformers, we develop a simple yet effective multi-head dynamic local self-attention (MHDLSA) module to extract local features efficiently. In addition, we note that existing Transformers usually explore all similarities of the tokens between the queries and keys for the feature aggregation. However, not all the tokens from the queries are relevant to those in keys, using all the similarities does not effectively facilitate the high-resolution image reconstruction. To overcome this problem, we develop a sparse global self-attention (SparseGSA) module to select the most useful similarity values so that the most useful global features can be better utilized for the high-resolution image reconstruction. We develop a hybrid dynamic-Transformer block(HDTB) that integrates the MHDLSA and SparseGSA for both local and global feature exploration. To ease the network training, we formulate the HDTBs into a residual hybrid dynamic-Transformer group (RHDTG). By embedding the RHDTGs into an end-to-end trainable network, we show that our proposed method has fewer network parameters and lower computational costs while achieving competitive performance against state-of-the-art ones in terms of accuracy.

Framework

---

Contents

The contents of this repository are as follows:

1. Dependencies
2. Train
3. Test

Dependencies

• Python
• Pytorch (1.11 or 1.13)
• basicsr
• cupy-cuda

extra infos:

• BasicSR
• cupy

For more details of the dependencies, please refer to requirements.txt

Train

```

For X2

sh ./demosbatchfile/SISRClassicDIV2K/trainSISRClassicDIV2KLarge90C6G4BDLGSANetSRx2scratchimgsize48lr5e_4.sh

For X3

sh ./demosbatchfile/SISRClassicDIV2K/trainSISRClassicDIV2KLarge90C6G4BDLGSANetSRx3scratchimgsize48lr5e_4.sh

For X4

sh ./demosbatchfile/SISRClassicDIV2K/trainSISRClassicDIV2KLarge90C6G4BDLGSANetSRx4scratchimgsize48lr5e_4.sh ```

Test

```

For X2

sh ./demosbatchfile/SISRClassicDIV2K/testSISRClassicDIV2KLarge90C6G4BDLGSANetSRx2scratchimgsize48lr5e_4.sh

For X3

sh ./demosbatchfile/SISRClassicDIV2K/testSISRClassicDIV2KLarge90C6G4BDLGSANetSRx3scratchimgsize48lr5e_4.sh

For X4

sh ./demosbatchfile/SISRClassicDIV2K/testSISRClassicDIV2KLarge90C6G4BDLGSANetSRx4scratchimgsize48lr5e_4.sh

```

---

Results

• Pretrained models and visual results

| Degradation | Model Zoo | Visual Results | | :----- |:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:| | BI-Efficient SR | Google Drive | Google Drive | | BI-Classic SR | To-Do | To-Do | | BI-Classic SR (x4) | Google Drive / Baidu Netdisk code:IMAG | Google Drive / Baidu Netdisk code:IMAG |

• Lightweight models PSNR

Unfortunately, the lightweight models' pretrain models were lost. Additionally, the github project has not been updated for a year due to personal reasons. Recently, I rebuilt the framework and used a single 3090GPU to retrain the lightweight models using the same training settings as the paper (16 batchsize). The re-trained model and those in the study have a slightly varied PSNR (within 0.05dB) due to differences in devices and versions of Pytorch/Cuda. New pre-train models and visual results will be added on Baidu Netdisk and Google Drive. We recommend either training on your own for research purposes or using the new data from re-trained lightweight models.

DLGSANet-Tiny:

| model-scale | Set5 | Set14 | BSDS100 | Urban100 | Manga109 | |:-----------------|:------------------:|:------------------:|:------------------:|:------------------:|:-------------------:| | Tiny-x2(paper) | 38.16 / 0.9611 | 33.92 / 0.9202 | 32.26 / 0.9007 | 32.82 / 0.9343 | 39.14 / 0.9777 | | Tiny-x2 | 38.1581 / 0.9615 | 33.8906 / 0.9200 | 32.2828 / 0.9017 | 32.8461 / 0.9343 | 39.1326 / 0.9780 | | Tiny-x3(paper) | 34.63 / 0.9288 | 30.57 / 0.8459 | 29.21 / 0.8083 | 28.69 / 0.8630 | 34.10 / 0.9480 | | Tiny-x3 | 34.6197 / 0.9293 | 30.5370 / 0.8469 | 29.2335 / 0.8100 | 28.7829 / 0.8645 | 34.0463 / 0.9477 | | Tiny-x4(paper) | 32.46 / 0.8984 | 28.79 / 0.7861 | 27.70 / 0.7408 | 26.55 / 0.8002 | 30.98 / 0.9137 | | Tiny-x4 | 32.4957 / 0.8992 | 28.7738 / 0.7862 | 27.7217 / 0.7426 | 26.5675 / 0.8006 | 30.9556 / 0.9142 |

DLGSANet-Light:

| model-scale | Set5 | Set14 | BSDS100 | Urban100 | Manga109 | |:-----------------|:------------------:|:------------------:|:------------------:|:------------------:|:-------------------:| | Light-x2(paper) | 38.20 / 0.9612 | 33.89 / 0.9203 | 32.30 / 0.9012 | 32.94 / 0.9355 | 39.29 / 0.9780 | | Light-x2 | 38.1577 / 0.9615 | 34.0453 / 0.9216 | 32.3058 / 0.9020 | 32.9323 / 0.9354 | 39.1995 / 0.9780 | | Light-x3(paper) | 34.70 / 0.9295 | 30.58 / 0.8465 | 29.24 / 0.8089 | 28.83 / 0.8653 | 34.16 / 0.9483 | | Light-x3 | 34.6697 / 0.9298 | 30.5621 / 0.8466 | 29.2484 / 0.8101 | 28.8239 / 0.8655 | 34.1938 / 0.9483 | | Light-x4(paper) | 32.54 / 0.8993 | 28.84 / 0.7871 | 27.73 / 0.7415 | 26.66 / 0.8033 | 31.13 / 0.9161 | | Light-x4 | 32.5333 / 0.8998 | 28.6401 / 0.7864 | 27.7299 / 0.7434 | 26.6702 / 0.8036 | 31.0196 / 0.9154 |

Visual Results

---

To Do

Release pre-trained models of regular models

Release the visual results of BI super-resolution

Citation

If this work is helpful for your research, please consider citing the following BibTeX entry. @article{li2023dlgsanet, title={DLGSANet: Lightweight Dynamic Local and Global Self-Attention Networks for Image Super-Resolution}, author={Li, Xiang and Pan, Jinshan and Tang, Jinhui and Dong, Jiangxin}, journal={arXiv preprint arXiv:2301.02031}, year={2023}, }

Acknowledgement

The foundation for the training process is BasicSR , which profited from the outstanding contribution of XPixelGroup .

The following research forms the foundation for the MHDLSA implementation:

On the Connection between Local Attention and Dynamic Depth-wise Convolution paper github

And the following research forms the foundation for the SparseGSA implementation:

Restormer: Efficient Transformer for High-Resolution Image Restoration paper github

Improving Image Restoration by Revisiting Global Information Aggregation paper github

Contact

This repo is currently maintained by Xiang Li (@neonleexiang) and is for academic research use only.