Environment Setting

Conda Environment Settings

1. python.__version__ == 3.9 bash conda create -n ${ENV_NAME} python=3.9
2. CUDA, Pytorch Version

CUDA 11.8, PyTorch 2.0.0 . bash pip install torch==2.0.0+cu118 torchvision==0.15.1+cu118 torchaudio==2.0.1+cu118 --index-url https://download.pytorch.org/whl/cu118

MMAction Installation

Install Dependency bash pip install -U openmim mim install "mmengine==0.10.5" mim install "mmcv==2.1.0" mim install "mmdet==3.2.0" mim install "mmpose==1.3.2"

Build mmaction from source bash pip install -e .

export conda environment yaml conda_environment.yaml .

Train & Test

Train

bash python tools/train.py ${CONFIG_FILE} [optional arguments]

Example bash python tools/train.py pipeline_integration/demo_configs/custom_violence_keypoint_epoch300_batch8.py

Test

bash python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [optional arguments]

Example bash python tools/test.pypipeline_integration/demo_configs/custom_violence_keypoint_epoch300_batch8.py pipeline_integration/weights/best_acc_top1_epoch_264.pth

Pipeline Integration - Code Execution

pipeline_integration .

Notification

Pipeline

Violence Detection Action Recognition Model PoseC3D.

Top-down Pose Estimation Heatmap join temporarily stack action classification , 3.2. From 2D Poses to 3D Heatmap Volumes heatmap keypoint coordinate Pose Heatmap storage space .

In experiments, we find that coordinate-triplets (x,y,c) help save the majority of storage space at the cost of little performance drop.

  Pipeline   , COCO  Keypoint Coordinate    .

Multi-person Inference

Top-down based HRNet single person action inference linear .

Model Configfile

Class , config .

pipeline_integration/extract_pose.py

Pose Extraction Example

bash python pipeline_integration/extract_pose.py --video-path ./pipeline_build/long_subset/1_074_1_01.mp4 Action Recognition Model PoseC3D Human Pose Keypoint pipeline_build/sample/long_subset .

Human Pose Estimation Model HRNet-w32 .

List[List[dict]] Shape

len(List[List[dict]])==num_people, len(List[dict])==num_frames .

dict . python [ [ { 'bbox_scores': array([1.], dtype=float32), 'bboxes': array([[443.81, 159.72, 532.91, 336.49]]), 'keypoint_scores': array([[0.875257 , 0.8032905 , 0.90061617, 0.6363667 , 0.8965926, 0.7831787 , 0.8786333 , 0.322353 , 0.9310938 , 0.22244653, 0.9125471 , 0.7968693 , 0.78160095, 0.87491614, 0.885747 , 0.8881028 , 0.9050592 ]], dtype=float32), 'keypoints': array([[[480.5918 , 179.91736], [480.5918 , 176.46481], [478.8655 , 178.19109], [470.23416, 181.64362], [470.23416, 181.64362], [470.23416, 193.72751], [459.87656, 200.63258], [484.0443 , 210.9902 ], [484.0443 , 217.89528], [501.307 , 205.8114 ], [504.75955, 204.08513], [478.8655 , 238.61052], [465.05536, 243.78932], [497.8545 , 271.40964], [471.96045, 280.041 ], [508.2121 , 311.11383], [470.23416, 319.74518]]], dtype=float32), 'keypoints_visible': array([[0.875257 , 0.8032905 , 0.90061617, 0.6363667 , 0.8965926 , 0.7831787 , 0.8786333 , 0.322353 , 0.9310938 , 0.22244653, 0.9125471 , 0.7968693 , 0.78160095, 0.87491614, 0.885747 , 0.8881028 , 0.9050592 ]], dtype=float32), } ], ... ]

pipeline_integration/inference.py

bash python pipeline_integration/inference.py --pose-file-path pipeline_integration/sample/long_subset/1_071_1_04.pkl --clip-len 48 --predict-step-size 12 --device cuda:0 extract_pose.py pkl input sliding window fashion inference .

Temporal Sliding Window Inference Logic

clip_len     (clip)  sliding window fashion  clip PoseC3d  inference.

(track_id) .

1. (track_id)
   

- temporal sliding window

    - clip_len * frame_interval  sliding window .

    - sliding window clip  ,      .

1. (track_id) clip PoseC3d Model :

- window predict_step_size  frame  PoseC3d Model inference.

-  clip frame index  Action Recognition  

- `pipeline_build/sample/results` pkl  .

Input/Output

: - pipeline_integration/inference.py Pose Data.

: - 'predscore': action class
- 'frameindex': frame index

pkl . List[List[dict]]

len(List[List[dict]])==num_person

len(List[dict]) inference clip .

python [ [ { 'pred_score':array([ 0.00093905,0.00426766,0.28390622,0.01333115,0.00408716, 0.04153658,0.64740133,0.00453098],dtype=float32), 'pred_label':array([6]), 'frame_index':[ 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47 ] }, ..., ] ]