APTv2

This repo is the official implementation of "APTv2: Benchmarking Animal Pose Estimation and Tracking with a Large-scale Dataset and Beyond".

Introduction

Animal Pose Estimation and Tracking (APT) is a critical task in detecting and monitoring the keypoints of animals across a series of video frames, which is essential for understanding animal behavior. Past works relating to animals have primarily focused on either animal tracking or single-frame animal pose estimation only, neglecting the integration of both aspects. The absence of comprehensive APT datasets inhibits the progression and evaluation of animal pose estimation and tracking methods based on videos, thereby constraining their real-world applications. To fill this gap, we introduce APTv2, the pioneering large-scale benchmark for animal pose estimation and tracking. APTv2 comprises 2,749 video clips filtered and collected from 30 distinct animal species. Each video clip includes 15 frames, culminating in a total of 41,235 frames. Following meticulous manual annotation and stringent verification, we provide high-quality keypoint and tracking annotations for a total of 84,611 animal instances, split into easy and hard subsets based on the number of instances that exists in the frame. With APTv2 as the foundation, we establish a simple baseline method named ViTPoseTrack and provide benchmarks for representative models across three tracks: (1) single-frame animal pose estimation track to evaluate both intra- and inter-domain transfer learning performance, (2) low-data transfer and generalization track to evaluate the inter-species domain generalization performance, and (3) animal pose tracking track. Our experimental results deliver key empirical insights, demonstrating that APTv2 serves as a valuable benchmark for animal pose estimation and tracking. It also presents new challenges and opportunities for future research.

Supervised Results with Pretrained Models

| method | model | pretrain | AP | config | weight | | :---: | :---: | :---: | :---: | :---: | :---: | | SimBa | Res-50 | IN1k | 64.1 | config | Onedrive | | SimBa | Res-50 | COCO | 67.8 | config | Onedrive | | SimBa | Res-50 | AP-10K | 66.3 | config | Onedrive | | SimBa | Res-101 | IN1k | 65.3 | config | Onedrive | | SimBa | Res-101 | COCO | 68.1 | config | Onedrive | | SimBa | Res-101 | AP-10K | 64.6 | config | Onedrive | | HRNet | HR-w32 | IN1k | 68.5 | config | Onedrive | | HRNet | HR-w32 | COCO | 70.1 | config | Onedrive | | HRNet | HR-w32 | AP-10K | 69.8 | config | Onedrive | | HRNet | HR-w48 | IN1k | 70.1 | config | Onedrive | | HRNet | HR-w48 | COCO | 71.7 | config | Onedrive | | HRNet | HR-w48 | AP-10K | 71.2 | config | Onedrive | | HRFormer | HRFomer-S | IN1k | 67.0 | config | Onedrive | | HRFormer | HRFomer-S | COCO | 69.5 | config | Onedrive | | HRFormer | HRFomer-S | AP-10K | 67.2 | config | Onedrive | | HRFormer | HRFomer-B | IN1k | 69.0 | config | Onedrive | | HRFormer | HRFomer-B | COCO | 69.7 | config | Onedrive | | HRFormer | HRFomer-B | AP-10K | 69.6 | config | Onedrive | | ViTPose | ViTPose-B | IN1k | 72.4 | config | Onedrive | | ViTPose | ViTPose-B | COCO | 72.4 | config | Onedrive | | ViTPose | ViTPose-B | AP-10K | 72.4 | config | Onedrive |

Zero-Shot Results

| model | pretrain | Canidae | Felidae | Hominidae | Cercopithecidae | Ursidae | Bovidae | Average | weight | | :---: | :---: | :---: | :---: | :---: | :---: |:---: | :---: | :---: | :---: | | HRNet-w32 | AP10K | 59.6 | 64.5 | 42.2 | 38.6 | 51.6 | 58.7 | 52.5 | Onedrive | | ViTPose-B | AP10K | 63.9 | 65.5 | 47.5 | 51.0 | 59.0 | 59.0 | 57.7 | Onedrive | | ViTPose-L | AP10K | 66.0 | 69.1 | 59.9 | 61.0 | 62.4 | 61.0 | 63.2 | Onedrive |

Leave-One-Out Results of HRNet-w32 (both easy and hard)

| setting | Canidae | Felidae | Hominidae | Cercopithecidae | Ursidae | Bovidae | weight | | :---: | :---: | :---: | :---: | :---: | :---: |:---: | :---: | | w/o Canidae | 66.1 | 77.8 | 68.8 | 66.3 | 75.4 | 68.3 | Onedrive | | w/o Felidae | 74.4 | 68.1 | 68.9 | 67.8 | 76.1 | 66.8 | Onedrive | | w/o Hominidae | 74.2 | 77.4 | 49.7 | 65.5 | 75.9 | 67.4 | Onedrive | | w/o Cercopithecidae | 75.2 | 79.0 | 68.6 | 45.0 | 76.2 | 67.7 | Onedrive | | w/o Ursidae | 74.2 | 77.9 | 69.9 | 66.4 | 51.3 | 67.6 | Onedrive | | w/o Bovidae | 74.0 | 77.4 | 69.3 | 66.8 | 76.9 | 59.5 | Onedrive |

Leave-One-Out Results of HRNet-w32 (only easy)

| setting | Canidae | Felidae | Hominidae | Cercopithecidae | Ursidae | Bovidae | weight | | :---: | :---: | :---: | :---: | :---: | :---: |:---: | :---: | | w/o Canidae | 55.9 | 68.0 | 63.2 | 60.8 | 61.8 | 55.9 | Onedrive | | w/o Felidae | 62.1 | 59.5 | 64.6 | 60.9 | 61.7 | 56.0 | Onedrive | | w/o Hominidae | 62.6 | 70.1 | 47.0 | 55.3 | 61.3 | 57.5 | Onedrive | | w/o Cercopithecidae | 62.6 | 69.1 | 62.9 | 38.1 | 61.9 | 56.0 | Onedrive | | w/o Ursidae | 62.5 | 69.7 | 64.6 | 60.5 | 45.2 | 55.7 | Onedrive | | w/o Bovidae | 61.8 | 69.5 | 65.0 | 59.9 | 63.4 | 50.9 | Onedrive |

Tracking Results Pretrained on GOT10K

| tracker | Res-50 | Res-101 | HR-w32 | HR-w48 | HRFormer-S | HRFormer-B | ViTPose-B | ViTPose-L | Avg. | weight | | :---: | :---: | :---: | :---: | :---: | :---: |:---: | :---: | :---: | :---: | :---: | | ViTTrack | 65.9 | 66.0 | 68.2 | 69.5 | 67.6 | 67.7 | 70.2 | 72.2 | 68.4 | Onedrive | | ViTPoseTrack-B | 65.3 | 65.6 | 67.9 | 69.1 | 66.9 | 67.1 | 69.7 | 71.9 | 67.9 | Onedrive | | ViTPoseTrack-L | 65.9 | 66.0 | 68.3 | 69.6 | 67.5 | 67.6 | 70.2 | 72.4 | 68.4 | Onedrive |

Installation

Please refer to install.md for installation guide.

Get Started

Download the data for APTv2. After downloading the pretrained models, please conduct the experiments by running. bash bash tools/dist_train.sh <Config PATH> <NUM GPUs> To test the pretrained models performance, please run bash bash tools/dist_test.sh <Config PATH> <Checkpoint PATH> <NUM GPUs>

Acknowledgement

We acknowledge the excellent implementation from mmpose.

Statement

If you are interested in our work, please consider citing the following:

@misc{yang2023aptv2, title={APTv2: Benchmarking Animal Pose Estimation and Tracking with a Large-scale Dataset and Beyond}, author={Yuxiang Yang and Yingqi Deng and Yufei Xu and Jing Zhang}, year={2023}, eprint={2312.15612}, archivePrefix={arXiv}, primaryClass={cs.CV} }