SMFANet: A Lightweight Self-Modulation Feature Aggregation Network for Efficient Image Super-Resolution

[Paper]   [Supp]  

Mingjun Zheng, Long Sun, Jiangxin Dong, and Jinshan Pan
IMAG Lab, Nanjing University of Science and Technology

**Network architecture of the proposed SMFANet. The proposed SMFANet consists of a shallow feature extraction module, feature modulation blocks, and a lightweight image reconstruction module. Feature modulation block contains one self-modulation feature aggregation (SMFA) module and one partial convolution-based feed-forward network (PCFN).

News

• [2024-09-26] The paper is available Here.
• [2024-08-04] We add the scripts for feature visualization, chart plotting and efficiency metric measurement.
• [2024-07-16] We add Hugging Face Demo.
• [2024-07-01] Our SMFANet is accepted by ECCV 2024.

- [2024-06-25] Our SMFANet places 2nd and 3rd in the Parameters and FLOPs sub-track of the NTIRE2024 ESR.

Requirements

• Python 3.8, PyTorch >= 1.8
• BasicSR 1.4.2
• Platforms: Ubuntu 18.04, cuda-11

Installation

```

Clone the repo

git clone https://github.com/Zheng-MJ/SMFANet.git

Install dependent packages

cd SMFANet conda create --name smfan python=3.8 conda activate smfan pip install -r requirements.txt

Install BasicSR

python setup.py develop ``` You can also refer to this INSTALL.md for installation

Data Preparation

Please refer to datasets/REDAME.md for data preparation.

Training

Run the following commands for training:

```

train SMFANet for x4 effieicnt SR

python basicsr/train.py -opt options/train/SMFANet/SMFANetDIV2K100w_x4SR.yml

train SMFANet+ for x4 effieicnt SR

python basicsr/train.py -opt options/train/SMFANet/SMFANetplusDIV2K100wx4SR.yml ```

Testing

• Download the testing dataset.
• Run the following commands: # test SMFANet for x4 efficient SR python basicsr/test.py -opt options/test/SMFANet_DF2K_x4SR.yml
• The test results will be in './results'.

Pretrained Model & Visual Results

Google Drive | Huggingface

TensorRT Optimization

• The script for exporting TensorRT model is available at to_tensorrt/READEME.md

Hugging Face Demo

• The Hugging Face Demo is available here.

Plotting Script

• The script for feature visualization and chart plotting is available at plt/README.md.

Experimental Results

• Comparison with CNN-based lightweight SR methods

• Comparison with ViT-based lightweight SR methods

• Memory and running time comparisons on x4 SR

• Visual comparisons for x4 SR on the Urban100 dataset

• Comparison of local attribution maps (LAMs) and diffusion indices (DIs)

• The power spectral density (PSD) visualizations of feature

Citation

If this work is helpful for your research, please consider citing the following BibTeX entry.

@inproceedings{smfanet, title={SMFANet: A Lightweight Self-Modulation Feature Aggregation Network for Efficient Image Super-Resolution}, author={Zheng, Mingjun and Sun, Long and Dong, Jiangxin and Pan, Jinshan}, booktitle={ECCV}, year={2024} }

Acknowledgement

This code is based on BasicSR toolbox. Thanks for the awesome work.

Contact

If you have any questions, please feel free to reach me out at mingjunzheng@njust.edu.cn