Long Short-Term Transformer for Online Action Detection

Introduction

This is a PyTorch implementation for our NeurIPS 2021 Spotlight paper "Long Short-Term Transformer for Online Action Detection".

Environment

• The code is developed with CUDA 10.2, Python >= 3.7.7, PyTorch >= 1.7.1

0. [Optional but recommended] create a new conda environment.
    ```
    conda create -n lstr python=3.7.7
    ```
    And activate the environment.
    ```
    conda activate lstr
    ```

1. Install the requirements
    ```
    pip install -r requirements.txt
    ```

Data Preparation

Option1: Prepare the features and targets by yourself.

1. Download the THUMOS'14 and TVSeries datasets.

2. Extract feature representations for video frames.

* For **ActivityNet** pretrained features, we use the [`ResNet-50`](https://arxiv.org/pdf/1512.03385.pdf) model for the RGB and optical flow inputs. We recommend to use this [`checkpoint`](https://github.com/open-mmlab/mmaction2/blob/master/configs/recognition/tsn/README.md#activitynet-v13) in [`MMAction2`](https://github.com/open-mmlab/mmaction2).

* For **Kinetics** pretrained features, we use the [`ResNet-50`](https://arxiv.org/pdf/1512.03385.pdf) model for the RGB inputs. We recommend to use this [`checkpoint`](https://github.com/open-mmlab/mmaction2/blob/master/configs/recognition/tsn/tsn_r50_320p_1x1x8_100e_kinetics400_rgb.py) in [`MMAction2`](https://github.com/open-mmlab/mmaction2). We use the [`BN-Inception`](https://arxiv.org/pdf/1502.03167.pdf) model for the optical flow inputs. We recommend to use the model [`here`](https://drive.google.com/drive/folders/1Q8yf2u8YWkva-apAxW_9_TzvLGuWZaix?usp=sharing).

***Note:*** We compute the optical flow using [`DenseFlow`](https://github.com/xumingze0308/denseflow).

1. If you want to use our dataloaders, please make sure to put the files as the following structure:

* THUMOS'14 dataset:
    ```
    $YOUR_PATH_TO_THUMOS_DATASET
    ├── rgb_kinetics_resnet50/
    |   ├── video_validation_0000051.npy (of size L x 2048)
    │   ├── ...
    ├── flow_kinetics_bninception/
    |   ├── video_validation_0000051.npy (of size L x 1024)
    |   ├── ...
    ├── target_perframe/
    |   ├── video_validation_0000051.npy (of size L x 22)
    |   ├── ...
    ```

* TVSeries dataset:
    ```
    $YOUR_PATH_TO_TVSERIES_DATASET
    ├── rgb_kinetics_resnet50/
    |   ├── Breaking_Bad_ep1.npy (of size L x 2048)
    │   ├── ...
    ├── flow_kinetics_bninception/
    |   ├── Breaking_Bad_ep1.npy (of size L x 1024)
    |   ├── ...
    ├── target_perframe/
    |   ├── Breaking_Bad_ep1.npy (of size L x 31)
    |   ├── ...
    ```

1. Create softlinks of datasets: cd long-short-term-transformer ln -s $YOUR_PATH_TO_THUMOS_DATASET data/THUMOS ln -s $YOUR_PATH_TO_TVSERIES_DATASET data/TVSeries

Option2: Directly download the pre-extracted features and targets from TeSTra.

If you want to skip the data preprocessing and quickly try LSTR, please refer to TeSTra. The features and targets there exactly follow LSTR's data structure and should be able to reproduce LSTR's performance. However, if you have any question about the processing of these features and targets, please contact the authors of TeSTra directly.

Training

Training LSTR with 512 seconds long-term memory and 8 seconds short-term memory requires less 3 GB GPU memory.

The commands are as follows.

``` cd long-short-term-transformer

Training from scratch

python tools/trainnet.py --configfile $PATHTOCONFIGFILE --gpu $CUDAVISIBLE_DEVICES

Finetuning from a pretrained model

python tools/trainnet.py --configfile $PATHTOCONFIGFILE --gpu $CUDAVISIBLEDEVICES \ MODEL.CHECKPOINT $PATHTO_CHECKPOINT ```

Online Inference

There are three kinds of evaluation methods in our code.

• First, you can use the config SOLVER.PHASES "['train', 'test']" during training. This process devides each test video into non-overlapping samples, and makes prediction on the all the frames in the short-term memory as if they were the latest frame. Note that this evaluation result is not the final performance, since (1) for most of the frames, their short-term memory is not fully utlized and (2) for simplicity, samples in the boundaries are mostly ignored.

  ``` cd long-short-term-transformer

  Inference along with training

  python tools/trainnet.py --configfile $PATHTOCONFIGFILE --gpu $CUDAVISIBLE_DEVICES \ SOLVER.PHASES "['train', 'test']" ```

• Second, you could run the online inference in batch mode. This process evaluates all video frames by considering each of them as the latest frame and filling the long- and short-term memories by tracing back in time. Note that this evaluation result matches the numbers reported in the paper, but batch mode cannot be further accelerated as descibed in paper's Sec 3.6. On the other hand, this mode can run faster when you use a large batch size, and we recomand to use it for performance benchmarking.

  ``` cd long-short-term-transformer

  Online inference in batch mode

  python tools/testnet.py --configfile $PATHTOCONFIGFILE --gpu $CUDAVISIBLEDEVICES \ MODEL.CHECKPOINT $PATHTOCHECKPOINT MODEL.LSTR.INFERENCEMODE batch ```

• Third, you could run the online inference in stream mode. This process tests frame by frame along the entire video, from the beginning to the end. Note that this evaluation result matches the both LSTR's performance and runtime reported in the paper. It processes the entire video as LSTR is applied to real-world scenarios. However, currently it only supports to test one video at each time.

  ``` cd long-short-term-transformer

  Online inference in stream mode

  python tools/testnet.py --configfile $PATHTOCONFIGFILE --gpu $CUDAVISIBLEDEVICES \ MODEL.CHECKPOINT $PATHTOCHECKPOINT MODEL.LSTR.INFERENCEMODE stream DATA.TESTSESSIONSET "['$VIDEO_NAME']" ```

Evaluation

Evaluate LSTR's performance for online action detection using perframe mAP or mcAP.

cd long-short-term-transformer python tools/eval/eval_perframe --pred_scores_file $PRED_SCORES_FILE

Evaluate LSTR's performance at different action stages by evaluating each decile (ten-percent interval) of the video frames separately.

cd long-short-term-transformer python tools/eval/eval_perstage --pred_scores_file $PRED_SCORES_FILE

Citations

If you are using the data/code/model provided here in a publication, please cite our paper:

@inproceedings{xu2021long,
    title={Long Short-Term Transformer for Online Action Detection},
    author={Xu, Mingze and Xiong, Yuanjun and Chen, Hao and Li, Xinyu and Xia, Wei and Tu, Zhuowen and Soatto, Stefano},
    booktitle={Conference on Neural Information Processing Systems (NeurIPS)},
    year={2021}
}

Security

See CONTRIBUTING for more information.

License

This project is licensed under the Apache-2.0 License.