YOLOFT: An Extremely Small Video Object Detection Benchmark Baseline

:loudspeaker: Introduction

This is the official implementation of the baseline model for XS-VID benchmark.

[news]: We will soon be releasing XS-VIDv2, incorporating many new videos and scenarios, significantly expanding our dataset! Please stay tuned!

:ferris_wheel: Dependencies

• CUDA 11.7
• Python 3.8
• PyTorch 1.12.1(cu116)
• TorchVision 0.13.1(cu116)
• numpy 1.24.4

:openfilefolder: Datasets

Our work is based on the large-scale extremely small video object detection benchmark XS-VID. Download the dataset(s) from corresponding links below. - [Google drive]annotations; images(0-3); images(4-5); - [BaiduNetDisk]annotations and images;

Please choose a download method to download the annotations and all images. Make sure all the split archive files (e.g., images.zip, images.z01, images.z02, etc.) are in the same directory. Use the following command to extract them:

bash unzip images.zip unzip annotations.zip We have released several annotation formats to facilitate subsequent research and use, including COCO, COCOVID, YOLO

Install

This repository is build on Ultralytics 8.0.143 which can be installed by running the following scripts. Please ensure that all dependencies have been satisfied before setting up the environment. ``` scp -r -P 2026 /data/jiahaoguo/datasets/gaode6/annotations/yolo/gaode6rm198exclude14569* jiahaoguo@115.156.158.8:/data/jiahaoguo/datasets/gaode_6/annotations/yolo/

conda create --name yoloft python=3.10 conda activate yoloft pip install torch==2.4.0 torchvision==0.19.0 torchaudio==2.4.0 --index-url https://download.pytorch.org/whl/cu118 git clone https://github.com/gjhhust/YOLOFT cd YOLOFT pip install -r requirements.txt pip install -e . pip install -U openmim mim install mmcv pip install mmcv-full

cd ./ultralytics/nn/modules/ops_dcnv3 python setup.py build install

cd ../altcudacorr_sparse python setup.py build install ```

:hourglass: Data preparation

If you want to use a custom video dataset for training tests, it needs to be converted to yolo format for annotation, and the dataset files are organized in the following format:

data_root_dir/ # Root data directory test.txt # List of test data files, each line contains a relative path to an image file train.txt # List of training data files, each line contains a relative path to an image file images/ # Directory containing image files video1/ # Directory for image files of the first video 0000000.png # First frame image file of the first video 0000001.png # Second frame image file of the first video video2/ # Directory for image files of the second video ... # More image files ... # More video directories labels/ # Directory containing label files video1/ # Directory for label files of the first video 0000000.txt # Label file for the first frame of the first video (matches the image file) 0000001.txt # Label file for the second frame of the first video (matches the image file) video2/ # Directory for label files of the second video ... # More label files ... # More video directories

Note: The name of the image and the name of the label in yolo format must be the same, and the format is frameNumber.png, e.g. "0000001.png and 0000001.txt".

1. XS-VID

https://modelscope.cn/datasets/lanlanlanrr/XS-VID/files

config/dataset/XS-VIDv2.yaml

1 splitlength: [1] trainslit: [0] #0epoch1

2 nloss splitlength: [n] trainslit: [0] #0epochn

3 splitlength: [1, n] trainslit: [0, 10] #110epochn

batchsizeXS-VIDv2.yaml ```yaml splitbatch_dict: 1: 32 2: 15 #s1 3: 8 4: 6 5: 5 6: 4 8: 7 ``` 1batch32215

1. coco ultralytics/data/scripts/get_coco.sh

yoloftyolo 1. nloss0 2. videodatasetsamplermodels/yoloft/detect/train.pybuild dataset 3. config/yoloftonxx/yoloftSdcndys1t.yaml C2fC2fDCNv3 4. mosiabboxsegment 5. 1coco200e -> 2RGB100e -> 3segmentn epoch 4->