Data

Tools

Indexes

Applications

Experiments

Open Source Science

multimodal-emotion-recognition-demo

A demo for multi-modal emotion recognition.(多模态情感识别demo)

https://github.com/robin-wzq/multimodal-emotion-recognition-demo

Science Score: 18.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ✓
    CITATION.cff file
    Found CITATION.cff file
  • ○
    codemeta.json file
  • ○
    .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Academic email domains
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (10.3%) to scientific vocabulary
Last synced: 10 months ago · JSON representation ·

Repository

A demo for multi-modal emotion recognition.(多模态情感识别demo)

Basic Info
  • Host: GitHub
  • Owner: Robin-WZQ
  • Language: Python
  • Default Branch: Version-3.0
  • Homepage:
  • Size: 9.3 MB
Statistics
  • Stars: 56
  • Watchers: 1
  • Forks: 5
  • Open Issues: 0
  • Releases: 4
Created almost 5 years ago · Last pushed about 2 years ago
Metadata Files
Readme Citation

readme.md

An End-to-End Visual-Audio Attention Network for Emotion Recognition DEMO

This is the demo implementation of the paper "An End-to-End Visual-Audio Attention Network for Emotion Recognition in User-Generated Videos". [NOT OFFICIAL!!]

Original Paper Project Page | Paper

Requirements

  • PyTorch (ver. 0.4+ required)
  • FFmpeg
  • Python3
  • Pyqt5

Preparation

data（ve8）

If U just want to use the DEMO, this step is not necessary. Download the pre-trained and trained model is enough!!

  • Download the videos here.(offical)
  • video pre-processing using /tools/processing.py(mp4 to jpg+ Add n_frames information + Generate annotation file in json format + mp4 to mp3)
  • Here, We provide the processed dataset, including VideoEmotion8-imgs(splitted by FFmpeg) and VideoEmotion8-mp3, so that you can train your own model easier.

VideoEmotion8-imgs: here (extraction code: fhom)

VideoEmotion8-mp3: here (extraction code: 7tn3)

model

  • resnet-101-kinetics.pth: pre-trained model download here (extraction code:0bi8)
  • save_30.pth: trained model download here (extraction code:uq82)
  • ve8_01.json: download here (extraction code:s567)

Running the code

Assume the strcture of data directories is the following: ```misc ~/ data Joy/ .../(video name) images/(jpg files) mp3/ mp3/(mp3 file) results resnet-101-kinetics.pth save30.pth ve801.json

```

Confirm all options in ~/opts.py. bash python Emotion.py

Result

图片2 See the next section for details.

Tutorial

To see another branch:click here --Tutorial

(Chinese version)

Owner

  • Login: Robin-WZQ
  • Kind: user
  • Location: Beijing

Citation (Citation.txt) 

Most code come from https://github.com/maysonny/VAANet/tree/master.

@inproceedings{Zhao2020AnEV,
  title={An End-to-End Visual-Audio Attention Network for Emotion Recognition in User-Generated Videos},
  author={Sicheng Zhao and Yunsheng Ma and Yang Gu and Jufeng Yang and Tengfei Xing and Pengfei Xu and Runbo Hu and Hua Chai and Kurt Keutzer},
  booktitle={AAAI},
  year={2020}
}

GitHub Events

Total
  • Watch event: 16
Last Year
  • Watch event: 16

Dependencies

requirements.txt pypi
  • Pillow ==7.2.0
  • SoundFile ==0.10.3.post1
  • audioread ==2.1.8
  • cffi ==1.14.0
  • decorator ==4.4.2
  • joblib ==0.15.1
  • librosa ==0.6.1
  • llvmlite ==0.31.0
  • numba ==0.48.0
  • numpy ==1.19.0
  • protobuf ==3.12.2
  • pycparser ==2.20
  • resampy ==0.2.2
  • scikit-learn ==0.23.1
  • scipy ==1.5.0
  • six ==1.15.0
  • tensorboardX ==2.0
  • threadpoolctl ==2.1.0
  • torch ==1.4.0
  • torchvision ==0.5.0