UFPMP-Det: Toward Accurate and Efficient Object Detection on Drone Imagery

The repo is the official implementation of UFPMP-Det.

The code of UFP module is at mmdet/core/ufp

The code of MP-Det is at mmdet/models/denseheads/mphead.py

The config of our project is at configs/UFPMP-Det

Install

1. This repo is implemented based on mmdetection. Please install it according to get_start.md.
2. shell pip install nltk pip install albumentations ## Quickstart We provide the Dataset(COCO Format) as follows:
3. VisDrone:链接：https://pan.baidu.com/s/1FfAsAApHZruucO5A2QgQAg 提取码：qrvs
4. UAVDT:链接：链接：https://pan.baidu.com/s/1KLmU5BBWwgtFbuZa7QWavw 提取码：z08x

We provide the checkpoint as follows: - VisDrone Coarse-Det:链接: https://pan.baidu.com/s/1jK3bqImDGSwqRJGVXinS0w 提取码: nab3 - VisDrone MP-Det ResNet50: 链接: https://pan.baidu.com/s/1zOoJVO2fPejnzM9KioZLuQ 提取码: m7rj

Training

This repo is only supposed single GPU.

Prepare

Build by yourself: We provide two data set conversion tools.

```shell

conver VisDrone to COCO

python UFPMP-Det-Tools/build_dataset/VisDrone2COCO.py

conver UAVDT to COCO

python UFPMP-Det-Tools/build_dataset/UAVDT2COCO.py

build UFP dataset(VisDrone)

CUDAVISIBLEDEVICES=2 python UFPMP-Det-Tools/builddataset/UFPVisDrone2COCO.py \ ./configs/UFPMP-Det/coarsedet.py \ ./workdirs/coarsedet/epoch12.pth \ xxxxxx/dataset/COCO/images/UAVtrain \ xxxxxx/dataset/COCO/annotations/instancesUAVtrainv1.json \ xxxxxx/dataset/COCO/images/instanceUFPUAVtrain/ \ xxxxxx/dataset/COCO/annotations/instanceUFPUAVtrain.json \ --txtpath pathtoVisDroneannotation_dir ```

Download:

In Quick Start

Train Coarse Detector

shell CUDA_VISIBLE_DEVICES=0 python tools/train.py ./configs/UFPMP-Det/coarse_det.py

Train MP-Det

shell CUDA_VISIBLE_DEVICES=0 python tools/train.py ./config/UFPMP-Det/mp_det_res50.py

Test

```shell CUDAVISIBLEDEVICES=2 python UFPMP-Det-Tools/evalscript/ufpmpdeteval.py \ ./configs/UFPMP-Det/coarsedet.py \ ./workdirs/coarsedet/epoch12.pth \ ./configs/UFPMP-Det/mpdetres50.py \ ./workdirs/mpdetres50/epoch12.pth \ XXXXX/dataset/COCO/annotations/instancesUAVval_v1.json \ XXXXX/dataset/COCO/images/UAVval

```

Citation

If you find our paper or this project helps your research, please kindly consider citing our paper in your publication.

@inproceedings{ufpmpdet, title={UFPMP-Det: Toward Accurate and Efficient Object Detection on Drone Imagery}, author={Huang, Yecheng and Chen, Jiaxin and Huang, Di}, booktitle={AAAI Conference on Artificial Intelligence}, year={2022} }

TinyNAS

• This repository is a collection of training-free neural architecture search methods developed by TinyML team, Data Analytics and Intelligence Lab, Alibaba DAMO Academy. Researchers and developers can use this toolbox to design their neural architectures with different budgets on CPU devices within 30 minutes.
  • Training-Free Neural Architecture Evaluation Scores by Entropy DeepMAD(CVPR'23), and by Gradient Zen-NAS(ICCV'21)
  • Joint Quantization and Architecture Search Mixed-Precision Quantization Search(NeurIPS'22)
  • Application : Object Detection MAE-DET(ICML'22)
  • Application : Action Recognition Maximizing Spatio-Temporal Entropy(ICLR'23)

News

• :sunny: Hiring research interns for Neural Architecture Search, Tiny Machine Learning, Computer Vision tasks: xiuyu.sxy@alibaba-inc.com
• :boom: 2023.04: PreNAS: Preferred One-Shot Learning Towards Efficient Neural Architecture Search is accepted by ICML'23.
• :boom: 2023.04: We will give a talk on Zero-Cost NAS at IFML Workshop, April 20, 2023.
• :boom: 2023.03: Code for E3D is now released.
• :boom: 2023.03: The code is refactoried and DeepMAD is supported.
• :boom: 2023.03: DeepMAD: Mathematical Architecture Design for Deep Convolutional Neural Network is accepted by CVPR'23.
• :boom: 2023.02: A demo is available on ModelScope
• :boom: 2023.01: Maximizing Spatio-Temporal Entropy of Deep 3D CNNs for Efficient Video Recognition is accepted by ICLR'23.
• :boom: 2022.11: DAMO-YOLO backbone search is now supported! And paper is on ArXiv now.
• :boom: 2022.09: Mixed-Precision Quantization Search is now supported! The QE-Score paper is accepted by NeurIPS'22.
• :boom: 2022.08: We will give a tutorial on Functional View for Zero-Shot NAS at MLSys'22.
• :boom: 2022.06: Code for MAE-DET is now released.
• :boom: 2022.05: MAE-DET is accepted by ICML'22.
• :boom: 2021.09: Code for Zen-NAS is now released.
• :boom: 2021.07: The inspiring training-free paper Zen-NAS has been accepted by ICCV'21.

Features

• This toolbox consists of multiple modules including the following :
  • Search Searcher module
  • Search Strategy module
  • Models Definition module
  • Score module
  • Search Space module
  • Budgets module
  • Latency Module
  • Population module

It manages these modules with the help of ModelScope Registry and Configuration mechanism.

• The Searcher is defined to be responsible for building and completing the entire search process. Through the combination of these modules and the corresponding configuration files, we can complete backbone search for different tasks (such as classification, detection, etc.) under different budget constraints (such as the number of parameters, FLOPs, delay, etc.).

• Currently supported tasks: For each task, we provide several sample configurations and scripts as follows to help you get started quickly.

  • Classification：Please Refer to Search Space and Config
  • Detection：Please Refer to Search Space and Config
  • Quantization: Please Refer to Search Space and Config

---

Installation

• Please Refer to installation.md

---

How to Use

• Please Refer to get_started.md

---

Results

Results for Classification（Details）

|Backbone|Param (MB)|FLOPs (G)|ImageNet TOP1|Structure|Download| |:----|:----|:----|:----|:----|:----| |DeepMAD-R18|11.69|1.82|77.7%| txt|model| |DeepMAD-R34|21.80|3.68|79.7%| txt|model | |DeepMAD-R50|25.55|4.13|80.6%|txt |model | |DeepMAD-29M-224|29|4.5|82.5%|txt |model | |DeepMAD-29M-288|29|4.5|82.8%|txt |model | |DeepMAD-50M|50|8.7|83.9%|txt |model | |DeepMAD-89M|89|15.4|84.0%|txt |model |
| Zen-NAS-R18-like | 10.8 | 1.7 | 78.44 | txt |model | | Zen-NAS-R50-like | 21.3 | 3.6 | 80.04 | txt |model | | Zen-NAS-R152-like | 53.5 | 10.5 | 81.59 | txt |model |

The official code for Zen-NAS was originally released at https://github.com/idstcv/ZenNAS.

---

Results for low-precision backbones（Details）

|Backbone|Param (MB)|BitOps (G)|ImageNet TOP1|Structure|Download| |:----|:----|:----|:----|:----|:----| |MBV2-8bit|3.4|19.2|71.90%| -| -| |MBV2-4bit|2.3|7|68.90%| -|- | |Mixed19d2G|3.2|18.8|74.80%|txt |model | |Mixed7d0G|2.2|6.9|70.80%|txt |model |

---

Results for Object Detection（Details）

| Backbone | Param (M) | FLOPs (G) | box AP_val | box AP_S | box AP_M | box AP_L | Structure | Download | |:---------:|:---------:|:---------:|:-------:|:-------:|:-------:|:-------:|:--------:|:------:| | ResNet-50 | 23.5 | 83.6 | 44.7 | 29.1 | 48.1 | 56.6 | - | - | | ResNet-101| 42.4 | 159.5 | 46.3 | 29.9 | 50.1 | 58.7 | - | - | | MAE-DET-S | 21.2 | 48.7 | 45.1 | 27.9 | 49.1 | 58.0 | txt |model | | MAE-DET-M | 25.8 | 89.9 | 46.9 | 30.1 | 50.9 | 59.9 | txt |model | | MAE-DET-L | 43.9 | 152.9 | 47.8 | 30.3 | 51.9 | 61.1 | txt |model |

---

Results for Action Recognition (Details）

| Backbone | size | FLOPs (G) | SSV1 Top-1 | SSV1 Top-5 | Structure | |:---------:|:-------:|:-------:|:-------:|:-------:|:--------:| | X3D-S | 160 | 1.9 | 44.6 | 74.4| - | | X3D-S | 224 | 1.9 | 47.3 | 76.6| - | | E3D-S | 160 | 1.9 | 47.1 | 75.6| txt | | E3D-M | 224 | 4.7 | 49.4 | 78.1| txt | | E3D-L | 312 | 18.3 | 51.1 | 78.7| txt |

---

Note： If you find this useful, please support us by citing them. ``` @inproceedings{cvpr2023deepmad, title = {DeepMAD: Mathematical Architecture Design for Deep Convolutional Neural Network}, author = {Xuan Shen and Yaohua Wang and Ming Lin and Yilun Huang and Hao Tang and Xiuyu Sun and Yanzhi Wang}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, year = {2023}, url = {https://arxiv.org/abs/2303.02165} }

@inproceedings{icml23prenas, title={PreNAS: Preferred One-Shot Learning Towards Efficient Neural Architecture Search}, author={Haibin Wang and Ce Ge and Hesen Chen and Xiuyu Sun}, booktitle={International Conference on Machine Learning}, year={2023}, organization={PMLR} }

@inproceedings{iclr23maxste, title = {Maximizing Spatio-Temporal Entropy of Deep 3D CNNs for Efficient Video Recognition}, author = {Junyan Wang and Zhenhong Sun and Yichen Qian and Dong Gong and Xiuyu Sun and Ming Lin and Maurice Pagnucco and Yang Song }, journal = {International Conference on Learning Representations}, year = {2023}, }

@inproceedings{neurips23qescore, title = {Entropy-Driven Mixed-Precision Quantization for Deep Network Design}, author = {Zhenhong Sun and Ce Ge and Junyan Wang and Ming Lin and Hesen Chen and Hao Li and Xiuyu Sun}, journal = {Advances in Neural Information Processing Systems}, year = {2022}, }

@inproceedings{icml22maedet, title={MAE-DET: Revisiting Maximum Entropy Principle in Zero-Shot NAS for Efficient Object Detection}, author={Zhenhong Sun and Ming Lin and Xiuyu Sun and Zhiyu Tan and Hao Li and Rong Jin}, booktitle={International Conference on Machine Learning}, year={2022}, organization={PMLR} }

@inproceedings{iccv21zennas, title = {Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognition}, author = {Ming Lin and Pichao Wang and Zhenhong Sun and Hesen Chen and Xiuyu Sun and Qi Qian and Hao Li and Rong Jin}, booktitle = {2021 IEEE/CVF International Conference on Computer Vision}, year = {2021}, } ```

License

This project is developed by Alibaba and licensed under the Apache 2.0 license.

This product contains third-party components under other open source licenses.

See the NOTICE file for more information.

Generalized Focal Loss V2: Learning Reliable Localization Quality Estimation for Dense Object Detection

GFocalV2 (GFLV2) is a next generation of GFocalV1 (GFLV1), which utilizes the statistics of learned bounding box distributions to guide the reliable localization quality estimation.

Again, GFLV2 improves over GFLV1 about ~1 AP without (almost) extra computing cost! Analysis of GFocalV2 in ZhiHu: 大白话 Generalized Focal Loss V2. You can see more comments about GFocalV1 in 大白话 Generalized Focal Loss(知乎)

---

More news:

[2021.3] GFocalV2 has been accepted by CVPR2021 (pre-review score: 113).

[2020.11] GFocalV1 has been adopted in NanoDet, a super efficient object detector on mobile devices, achieving same performance but 2x faster than YoLoV4-Tiny! More details are in YOLO之外的另一选择，手机端97FPS的Anchor-Free目标检测模型NanoDet现已开源~.

[2020.10] Good News! GFocalV1 has been accepted in NeurIPs 2020 and GFocalV2 is on the way.

[2020.9] The winner (1st) of GigaVision (object detection and tracking) in ECCV 2020 workshop from DeepBlueAI team adopt GFocalV1 in their solutions.

[2020.7] GFocalV1 is officially included in MMDetection V2, many thanks to @ZwwWayne and @hellock for helping migrating the code.

Introduction

Localization Quality Estimation (LQE) is crucial and popular in the recent advancement of dense object detectors since it can provide accurate ranking scores that benefit the Non-Maximum Suppression processing and improve detection performance. As a common practice, most existing methods predict LQE scores through vanilla convolutional features shared with object classification or bounding box regression. In this paper, we explore a completely novel and different perspective to perform LQE -- based on the learned distributions of the four parameters of the bounding box. The bounding box distributions are inspired and introduced as ''General Distribution'' in GFLV1, which describes the uncertainty of the predicted bounding boxes well. Such a property makes the distribution statistics of a bounding box highly correlated to its real localization quality. Specifically, a bounding box distribution with a sharp peak usually corresponds to high localization quality, and vice versa. By leveraging the close correlation between distribution statistics and the real localization quality, we develop a considerably lightweight Distribution-Guided Quality Predictor (DGQP) for reliable LQE based on GFLV1, thus producing GFLV2. To our best knowledge, it is the first attempt in object detection to use a highly relevant, statistical representation to facilitate LQE. Extensive experiments demonstrate the effectiveness of our method. Notably, GFLV2 (ResNet-101) achieves 46.2 AP at 14.6 FPS, surpassing the previous state-of-the-art ATSS baseline (43.6 AP at 14.6 FPS) by absolute 2.6 AP on COCO test-dev, without sacrificing the efficiency both in training and inference.

For details see GFocalV2. The speed-accuracy trade-off is as follows:

Get Started

Please see GETTING_STARTED.md for the basic usage of MMDetection.

Train

```python

assume that you are under the root directory of this project,

and you have activated your virtual environment if needed.

and with COCO datas in 'datas/coco/'

./tools/disttrain.sh configs/gfocal/gfocalr50fpnms2x.py 8 --validate ```

Inference

python ./tools/dist_test.sh configs/gfocal/gfocal_r50_fpn_ms2x.py work_dirs/gfocal_r50_fpn_ms2x/epoch_24.pth 8 --eval bbox

Speed Test (FPS)

python CUDA_VISIBLE_DEVICES=0 python3 ./tools/benchmark.py configs/gfocal/gfocal_r50_fpn_ms2x.py work_dirs/gfocal_r50_fpn_ms2x/epoch_24.pth

Models

For your convenience, we provide the following trained models (GFocalV2). All models are trained with 16 images in a mini-batch with 8 GPUs.

Model | Multi-scale training | AP (minival) | AP (test-dev) | FPS | Link --- |:---:|:---:|:---:|:---:|:---: GFocalR50FPN1x | No | 41.0 | 41.1 | 19.4 | Google GFocalR50FPN2x | Yes | 43.9 | 44.4 | 19.4 | Google GFocalR101FPN2x | Yes | 45.8 | 46.0 | 14.6 | Google GFocalR101dcnv2FPN2x | Yes | 48.0 | 48.2 | 12.7 | Google GFocalX101dcnv2FPN2x | Yes | 48.8 | 49.0 | 10.7 | Google GFocalR2101dcnv2FPN_2x | Yes | 49.9 | 50.5 | 10.9 | Google

[0] The reported numbers here are from new experimental trials (in the cleaned repo), which may be slightly different from the original paper. \ [1] Note that the 1x performance may be slightly unstable due to insufficient training. In practice, the 2x results are considerably stable between multiple runs. \ [2] All results are obtained with a single model and without any test time data augmentation such as multi-scale, flipping and etc.. \ [3] dcnv2 denotes deformable convolutional networks v2. Note that for ResNe(X)t based models, we apply deformable convolutions from stage c3 to c5 in backbones. \ [4] Refer to more details in config files in config/gfocal/. \ [5] FPS is tested with a single GeForce RTX 2080Ti GPU, using a batch size of 1.

Acknowledgement

Thanks MMDetection team for the wonderful open source project!

Citation

If you find GFocal useful in your research, please consider citing:

``` @article{li2020gfl, title={Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection}, author={Li, Xiang and Wang, Wenhai and Wu, Lijun and Chen, Shuo and Hu, Xiaolin and Li, Jun and Tang, Jinhui and Yang, Jian}, journal={arXiv preprint arXiv:2006.04388

    },

year={2020} } ```

``` @article{li2020gflv2, title={Generalized Focal Loss V2: Learning Reliable Localization Quality Estimation for Dense Object Detection}, author={Li, Xiang and Wang, Wenhai and Hu, Xiaolin and Li, Jun and Tang, Jinhui and Yang, Jian}, journal={arXiv preprint arXiv:2011.12885

    },

year={2020} } ```