386-more-convnets-in-the-2020s-scaling-up-kernels-beyond-51x51-using-sparsity

https://github.com/szu-advtech-2023/386-more-convnets-in-the-2020s-scaling-up-kernels-beyond-51x51-using-sparsity

Repository

https://github.com/SZU-AdvTech-2023/386-More-ConvNets-in-the-2020s-Scaling-up-Kernels-Beyond-51x51-Using-Sparsity/blob/main/

# [shift-wise conv](https://github.com/lidc54/shift-wiseConv)
Official PyTorch implementation of 
[Shift-ConvNets: Small Convolutional Kernel with Large Kernel Effects]()




We propose the **Shift-Wise** operator, a ConvNet model that uses large convolutional kernels to capture long-range sparse dependencies.

More code will be released later.


## Installation

The code is tested used CUDA 11.7, cudnn 8.2.0, PyTorch 1.10.0.

Create an new conda virtual environment
```
conda create -n shiftWise python=3.8 -y
conda activate shiftWise
```

Install [Pytorch](https://pytorch.org/)>=1.10.0. For example:
```
conda install pytorch==1.10.0 torchvision==0.11.0 torchaudio==0.10.0 cudatoolkit=11.3 -c pytorch -c conda-forge
pip install timm tensorboardX six
```

## evaluation
```
# to test on imagenet with SLaK_reg_tiny
python main.py --model SLaK_reg_tiny --eval true --Decom True --kernel_size 51 49 47 13 5 --width_factor 1.3 --resume checkpoints/SLaK_tiny_checkpoint.pth --input_size 224 --drop_path 0.2 --data_path /workspace/public_data/imagenet2012/

# train tiny pw-random model
python -m torch.distributed.launch --nproc_per_node=3 main.py  --Decom True --sparse --width_factor 1.3 -u 2000 --sparsity 0.4 --sparse_init snip  --prune_rate 0.5 --growth random --epochs 300 --model SLaK_pw_tiny --drop_path 0.1 --batch_size 64 --lr 4e-3 --update_freq 21 --model_ema true --model_ema_eval true --data_path /workspace/public_data/imagenet2012/ --num_workers 40 --kernel_size 51 49 43 13 5 --output_dir checkpoints/pw_random

# train tiny pw-weight model
python -m torch.distributed.launch --nproc_per_node=3 main.py  --Decom True --sparse --width_factor 1.3 -u 2000 --sparsity 0.4 --sparse_init snip  --prune_rate 0.5 --growth random --epochs 300 --model SLaK_pww_tiny --drop_path 0.1 --batch_size 48 --lr 4e-3 --update_freq 28 --model_ema true --model_ema_eval true --data_path /workspace/public_data/imagenet2012/ --num_workers 40 --kernel_size 51 49 45 13 5 --output_dir checkpoints/pw_weight
```


## ToDo List
- [] 51*5   
- []  
  - [] shuffle
  - [] 
  - [] resnetidentity
- [] padding
  - [] k-1
  - [] k//2
- [] 
- [] taichishift
- [] 

## 
 **** | ****                   | **acc@1&120 epochs** | **acc@1&300 epochs** | ****                      
:------:|:------------------------:|:--------------------:|:--------------------:|:---------------------------:
 **1**  | SLaK-T baseline          | 81.6                 | 82.5                 |                             
 **2**  | SlaK-T                 |                      | 82.482               |                             
 **3**  | 51*5       |                      | 82.482               | models/SLaK_reg.py          
 **4**  |                |                      | 82.27                | models/SLaK_pw_wt.py        
 **5**  | shuffle          |                      | 82.27                | models/SLaK_pw.py           
 **6**  | paddingKernelK-1 |                      | 82.38                | models/SLaK_pw_fixpad.py    
 **7**  | pad=k-1            | 81.36                |                      | models/SLaK_pw_fixpad.py    
 **8**  | pad=k//1           | 81.26                |                      | models/SLaK_pw_fixpad.py    
 **9**  |                       | **82.05**            |                      | models/SLaK_pw_fixpadRN.py  


## macs & params
- install packages
```
pip install thop  pip install --upgrade git+https://github.com/Lyken17/pytorch-OpCounter.gitpytorch
pip install ptflops  pip install --upgrade git+https://github.com/sovrasov/flops-counter.pytorch.git
```
- to get info of SLaK_tiny
```
# SLaK_tiny
 python flopses.py --Decom True  --sparse --width_factor 1.3 -u 2000 --sparsity 0.4 --sparse_init snip --prune_rate 0.5 --growth random  --epochs 120 --model SLaK_tiny  --drop_path 0.1 --batch_size 64  --lr 4e-3 --update_freq 8 --model_ema false --model_ema_eval false  --data_path /home/lili/cc/dataset/imagenet2012 --num_workers 60 --kernel_size 51 49 47 13 5
# result 
Computational complexity:       8.74 GMac
Number of parameters:           50.7 M
----------1. macs:8.74 GMac; params:50.7 M--------------
[INFO] Register count_convNd() for .
[INFO] Register zero_ops() for .
[INFO] Register count_normalization() for .
[INFO] Register count_linear() for .
[INFO] Register count_normalization() for .
----------2. macs:8738130546.0; params:50676034.0--------------
```
- to get info of SLaK_gr_sw_tiny
```
# SLaK_gr_sw_tiny
python flopses.py --Decom True  --sparse --width_factor 1.3 -u 2000 --sparsity 0.4 --sparse_init snip --prune_rate 0.5 --growth random  --epochs 120 --model SLaK_gr_sw_tiny  --drop_path 0.1 --batch_size 16  --lr 4e-3 --update_freq 8 --model_ema false --model_ema_eval false  --data_path /home/lili/cc/dataset/imagenet2012 --num_workers 60 --use_amp true --kernel_size 51 49 45 13 5
#result
Computational complexity:       7.72 GMac
Number of parameters:           48.01 M
----------1. macs:7.72 GMac; params:48.01 M--------------
[INFO] Register count_convNd() for .
[INFO] Register zero_ops() for .
[INFO] Register count_normalization() for .
[INFO] Register count_linear() for .
[INFO] Register count_normalization() for .
----------2. macs:7701405072.0; params:47985520.0--------------

#width_factor 1.0
Computational complexity:       4.66 GMac
Number of parameters:           28.74 M
----------1. macs:4.66 GMac; params:28.74 M--------------
[INFO] Register count_convNd() for .
[INFO] Register zero_ops() for .
[INFO] Register count_normalization() for .
[INFO] Register count_linear() for .
[INFO] Register count_normalization() for .
----------2. macs:4645198236.0; params:28723543.0--------------
```
 **name**   | **macs** | **params** 
:----------:|:--------:|:----------:
  **SLaK-T**  | 50.7 M   | 8.74 G       
 **ours-1.3** | 48.01 M  | 7.72 G  
 **ours-1.2** | 40.98 M  | 6.61 G      
 **ours-1.1** | 34.57 M  | 5.59 G     
 **ours-1.0** | 28.74 M  | 4.66 G     

## based on repository [Sparse Large Kernel Network - SLaK](https://github.com/VITA-Group/SLaK)

 **name**   | **resolution** | **acc@1**       | **model**                                                                                          
:----------:|:--------------:|:---------------:|:--------------------------------------------------------------------------------------------------:
 **SLaK-T** | 224x224        | 82.5(300epoch)  | [Google Drive](https://drive.google.com/file/d/1Iut2f5FMS_77jGPYoUJDQzDIXOsax1u4/view?usp=sharing) 
 **SLaK-T** | 224x224        | 81.6(120epochs) | [Surf Drive](https://surfdrive.surf.nl/files/index.php/s/WiQYWNclJ9bW5XV)                          

## cite
If you find this repository useful, please consider giving a star star and cite our paper.

Owner

Citation (citation.txt)

@article{REPO386,
    author = "Liu, Shiwei and Chen, Tianlong and Chen, Xiaohan and Chen, Xuxi and Xiao, Qiao and Wu, Boqian and Pechenizkiy, Mykola and Mocanu, Decebal and Wang, Zhangyang",
    journal = "arXiv preprint arXiv:2207.03620",
    title = "{More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity}",
    year = "2022"
}

GitHub Events