Reading Between the 🎞️ Frames:
Multi-Modal Depression Detection in Videos from Non-Verbal Cues

Accepted at the 2024 edition of European Conference on Information Retrieval (ECIR).

TL;DR

We extract high-level non-verbal cues using pretrained models, process them using a modality-specific encoder, condition the resulting embeddings with positional and modality embeddings, and process the sequence with a transformer encoder to perform the final classification.

📘 Introduction

Depression, a prominent contributor to global disability, affects a substantial portion of the population. Efforts to detect depression from social media texts have been prevalent, yet only a few works explored depression detection from user-generated video content. In this work, we address this research gap by proposing a simple and flexible multi-modal temporal model capable of discerning non-verbal depression cues from diverse modalities in noisy, real-world videos. We show that, for in-the-wild videos, using additional high-level non-verbal cues is crucial to achieving good performance, and we extracted and processed audio speech embeddings, face emotion embeddings, face, body and hand landmarks, and gaze and blinking information. Through extensive experiments, we show that our model achieves state-of-the-art results on three key benchmark datasets for depression detection from video by a substantial margin.

🛠️ Data Preparation

Downloading the datasets

• For D-Vlog, the features extracted by the authors are publicly available here. Original vlog videos are available upon request. Please contact the original paper authors.

• For DAIC-WOZ and E-DAIC, the features are only available upon request here.

Extracting non-verbal modalities

Implementation Detail

Once all the data has been pre-processed, you should indicate the absule path to the directory where it is stored in the 'configs/env_config.yaml' file for each one of the corresponding datasets.

In addition, you can continue working in the 'landmarks' environment, since it has everything we need for training and evaluating our model:

conda activate landmarks

Modality distributions for D-Vlog

Once extracted, the modalities for D-Vlog should be similar to the following plots (counts / missing frames):

💪 Training and Evaluation

To train and evaluate the models and the results reported in the paper, you can run the following commands:

cd experiments/ bash run-exps.sh

📖 Citation

If you found our work useful, please cite our paper:

Reading Between the Frames: Multi-Modal Non-Verbal Depression Detection in Videos

@InProceedings{gimeno2024videodepression, author="Gimeno-G{\'o}mez, David and Bucur, Ana-Maria and Cosma, Adrian and Mart{\'i}nez-Hinarejos, Carlos-David and Rosso, Paolo", editor="Goharian, Nazli and Tonellotto, Nicola and He, Yulan and Lipani, Aldo and McDonald, Graham and Macdonald, Craig and Ounis, Iadh", title="Reading Between the Frames: Multi-modal Depression Detection in Videos from Non-verbal Cues", booktitle="Advances in Information Retrieval", year="2024", publisher="Springer Nature Switzerland", address="Cham", pages="191--209", isbn="978-3-031-56027-9" }

This repository is based on the Acumen ✨ Template ✨.

📝 License

This work is protected by CC BY-NC-ND 4.0 License (Non-Commercial & No Derivatives)