SuperYOLO: Super Resolution Assisted Object Detection in Multimodal Remote Sensing Imagery

⭐ This code has been completely released ⭐

⭐ our article ⭐

⭐ We also finish the work about the quantization based on SuperYOLO： Guided Hybrid Quantization for Object Detection in Multimodal Remote Sensing Imagery via One-to-one Self-teaching!!!⭐

If our code is helpful to you, please cite:

``` @ARTICLE{10075555, author={Zhang, Jiaqing and Lei, Jie and Xie, Weiying and Fang, Zhenman and Li, Yunsong and Du, Qian}, journal={IEEE Transactions on Geoscience and Remote Sensing}, title={SuperYOLO: Super Resolution Assisted Object Detection in Multimodal Remote Sensing Imagery}, year={2023}, volume={61}, number={}, pages={1-15}, doi={10.1109/TGRS.2023.3258666}}

@article{zhang2023guided, title={Guided Hybrid Quantization for Object Detection in Remote Sensing Imagery via One-to-one Self-teaching}, author={Zhang, Jiaqing and Lei, Jie and Xie, Weiying and Li, Yunsong and Yang, Geng and Jia, Xiuping}, journal={IEEE Transactions on Geoscience and Remote Sensing}, year={2023}, publisher={IEEE} }

```

Requirements

python pip install -r requirements.txt

Train

1. Prepare training data

• 1.1 In order to realize the SR assisted branch, the input images of the network are downsampled from 1024 x 1024 size to 512 x 512 during the training process. In the test process, the image size is 512 x 512, which is consistent with the input of other algorithms compared.

• 1.2 Download VEDAI data for our experiment from baiduyun (code: hvi4) or google drive. And the path of dataset is like that

python SuperYOLO ├── dataset │ ├── VEDAI │ │ ├── images │ │ ├── labels │ │ ├── fold01.txt │ │ ├── fold01test.txt │ │ ├── fold02.txt │ │ ├── ..... │ ├── VEDAI_1024 │ │ ├── images │ │ ├── labels

• 1.3 Note that we transform the labels of the dataset to be horizontal boxes by transform code. You shoud run transform.py before training the model. Change the PATH = './dataset/' and then run the code.

2. Begin to train multi images

python python train.py --cfg models/SRyolo_MF.yaml --super --train_img_size 1024 --hr_input --data data/SRvedai.yaml --ch 64 --input_mode RGB+IR+MF

3. Begin to train RGB or IR images

python python train.py --cfg models/SRyolo_noFocus_small.yaml --super --train_img_size 1024 --hr_input --data data/SRvedai.yaml --ch 3 --input_mode RGB

python python train.py --cfg models/SRyolo_noFocus_small.yaml --super --train_img_size 1024 --hr_input --data data/SRvedai.yaml --ch 3 --input_mode IR

4. Begin to train multi images without SR branch

python python train.py --cfg models/SRyolo_MF.yaml --train_img_size 512 --data data/SRvedai.yaml --ch 64 --input_mode RGB+IR+MF

5. Begin to train RGB or IR images without SR branch

python python train.py --cfg models/SRyolo_noFocus_small.yaml --train_img_size 512 --data data/SRvedai.yaml --ch 3 --input_mode RGB

python python train.py --cfg models/SRyolo_noFocus_small.yaml --train_img_size 512 --data data/SRvedai.yaml --ch 3 --input_mode IR

Test

1. Pretrained Checkpoints

You can use our pretrained checkpoints for test process. Download pre-trained model and put it in here.

2. Begin to test

python python test.py --weights runs/train/exp/best.pt --input_mode RGB+IR+MF

Results

| Method | Modality | Car | Pickup | Camping | Truck | Other | Tractor | Boat | Van | mAP50 | Params. $\downarrow$ | GFLOPs $\downarrow$ | |:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:| | YOLOv3 | IR | 80.21 | 67.03 | 65.55 | 47.78 | 25.86 | 40.11 | 32.67 | 53.33 | 51.54 | 61.5351M | 49.55 | | YOLOv3 | RGB | 83.06 | 71.54 | 69.14 | 59.30 | 48.93 | 67.34 | 33.48 | 55.67 | 61.06 | 61.5351M | 49.55 | | YOLOv3 | Multi | 84.57 | 72.68 | 67.13 | 61.96 | 43.04 | 65.24 | 37.10 | 58.29 | 61.26 | 61.5354M | 49.68 | | YOLOv4 | IR | 80.45 | 67.88 | 68.84 | 53.66 | 30.02 | 44.23 | 25.40 | 51.41 | 52.75 | 52.5082M | 38.16 | | YOLOv4 | RGB | 83.73 | 73.43 | 71.17 | 59.09 | 51.66 | 65.86 | 34.28 | 60.32 | 62.43 | 52.5082M | 38.16 | | YOLOv4 | Multi | 85.46 | 72.84 | 72.38 | 62.82 | 48.94 | 68.99 | 34.28 | 54.66 | 62.55 | 52.5085M | 38.23 | | YOLOv5s | IR | 77.31 | 65.27 | 66.47 | 51.56 | 25.87 | 42.36 | 21.88 | 48.88 | 49.94 | 7.0728M | 5.24 | | YOLOv5s | RGB | 80.07 | 68.01 | 66.12 | 51.52 | 45.76 | 64.38 | 21.62 | 40.93 | 54.82 | 7.0728M | 5.24 | | YOLOv5s | Multi | 80.81 | 68.48 | 69.06 | 54.71 | 46.76 | 64.29 | 24.25 | 45.96 | 56.79 | 7.0739M | 5.32 | | YOLOv5m | IR | 79.23 | 67.32 | 65.43 | 51.75 | 26.66 | 44.28 | 26.64 | 56.14 | 52.19 | 21.0659M | 16.13 | | YOLOv5m | RGB | 81.14 | 70.26 | 65.53 | 53.98 | 46.78 | 66.69 | 36.24 | 49.87 | 58.80 | 21.0659M | 16.13 | | YOLOv5m | Multi | 82.53 | 72.32 | 68.41 | 59.25 | 46.20 | 66.23 | 33.51 | 57.11 | 60.69 | 21.0677M | 16.24 | | YOLOv5l | IR | 80.14 | 68.57 | 65.37 | 53.45 | 30.33 | 45.59 | 27.24 | 61.87 | 54.06 | 46.6383M | 36.55 | | YOLOv5l | RGB | 81.36 | 71.70 | 68.25 | 57.45 | 45.77 | 70.68 | 35.89 | 55.42 | 60.81 | 46.6383M | 36.55 | | YOLOv5l | Multi | 82.83 | 72.32 | 69.92 | 63.94 | 48.48 | 63.07 | 40.12 | 56.46 | 62.16 | 46.6046M | 36.70 | | YOLOv5x | IR | 79.01 | 66.72 | 65.93 | 58.49 | 31.39 | 41.38 | 31.58 | 58.98 | 54.18 | 87.2458M | 69.52 | | YOLOv5x | RGB | 81.66 | 72.23 | 68.29 | 59.07 | 48.47 | 66.01 | 39.15 | 61.85 | 62.09 | 87.2458M | 69.52 | | YOLOv5x | Multi | 84.33 | 72.95 | 70.09 | 61.15 | 49.94 | 67.35 | 38.71 | 56.65 | 62.65 | 87.2487M | 69.71 | | SuperYOLO | IR | 87.90 | 81.39 | 76.90 | 61.56 | 39.39 | 60.56 | 46.08 | 71.00 | 65.60 | 4.8256M | 16.61 | | SuperYOLO | RGB | 90.30 | 82.66 | 76.69 | 68.55 | 53.86 | 79.48 | 58.08 | 70.30 | 72.49 | 4.8256M | 16.61 | | SuperYOLO | Multi | 91.13 | 85.66 | 79.30 | 70.18 | 57.33 | 80.41 | 60.24 | 76.50 | 75.09 | 4.8451M | 17.98 |

Time

2023.2.14 open the train.py

2023.2.14 update the new fusion method (MF)

2023.2.16 update the test.py for visualization of detection results

Visualization of results

Acknowledgements

This code is built on YOLOv5 (PyTorch). We thank the authors for sharing the codes.

Licencing

Copyright (C) 2020 Jiaqing Zhang

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3 of the License.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program.

Contact

If you have any questions, please contact me by email (jq.zhangcn@foxmail.com).