SpeechLLM

SpeechLLM is a multi-modal Language Model (LLM) specifically trained to analyze and predict metadata from a speaker's turn in a conversation. This advanced model integrates a speech encoder to transform speech signals into meaningful speech representations. These embeddings, combined with text instructions, are then processed by the LLM to generate predictions.

The model inputs an speech audio file of 16 KHz and predicts the following: 1. SpeechActivity : if the audio signal contains speech (True/False) 2. Transcript : ASR transcript of the audio 3. Gender of the speaker (Female/Male) 4. Age of the speaker (Young/Middle-Age/Senior) 5. Accent of the speaker (Africa/America/Celtic/Europe/Oceania/South-Asia/South-East-Asia) 6. Emotion of the speaker (Happy/Sad/Anger/Neutral/Frustrated)

Usage

```python

Load model directly from huggingface

from transformers import AutoModel model = AutoModel.frompretrained("skit-ai/speechllm-2B", trustremote_code=True)

model.generatemeta( audiopath="path-to-audio.wav", #16k Hz, mono audiotensor=torchaudio.load("path-to-audio.wav")[1], # [Optional] either audiopath or audiotensor directly instruction="Give me the following information about the audio [SpeechActivity, Transcript, Gender, Emotion, Age, Accent]", maxnewtokens=500, returnspecial_tokens=False )

Model Generation

''' { "SpeechActivity" : "True", "Transcript": "Yes, I got it. I'll make the payment now.", "Gender": "Female", "Emotion": "Neutral", "Age": "Young", "Accent" : "America", } ''' ```

Try the model in Google Colab Notebook. Also, check out our blog on SpeechLLM for end-to-end conversational agents(User Speech -> Response).

Model Weights

We released the speechllm-2B and speechllm-1.5B model checkpoints on huggingface :hugs:. | Model | Speech Encoder | LLM | checkpoint url | |-------------------|-------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------|---------------------------------------------------------------| | speechllm-2B | facebook/hubert-xlarge-ll60k | TinyLlama/TinyLlama-1.1B-Chat-v1.0 | Huggingface | | speechllm-1.5B | microsoft/wavlm-large | TinyLlama/TinyLlama-1.1B-Chat-v1.0 | Huggingface |

Latest Checkpoint Result

speechllm-2B

| Dataset | Type | Word Error Rate | Gender Acc | Age Acc | Accent Acc | |:--------------------------:|:-------------------:|:-------------------:|:--------------:|:-----------:|:--------------:| | librispeech-test-clean | Read Speech | 6.73 | 0.9496 | | | | librispeech-test-other | Read Speech | 9.13 | 0.9217 | | | | CommonVoice test | Diverse Accent, Age | 25.66 | 0.8680 | 0.6041 | 0.6959 |

speechllm-1.5B

| Dataset | Type | Word Error Rate | Gender Acc | Age Acc | Accent Acc | |:--------------------------:|:-------------------:|:-------------------:|:--------------:|:-----------:|:--------------:| | librispeech-test-clean | Read Speech | 11.51 | 0.9594 | | | | librispeech-test-other | Read Speech | 16.68 | 0.9297 | | | | CommonVoice test | Diverse Accent, Age | 26.02 | 0.9476 | 0.6498 | 0.8121 |

Training

Dataset Preparation and Installation

Install the necessary packages in the requirements.txt and take care of CUDA versions. Then prepare the audio dataset similar to datasamples/train.csv and datasamples/dev.csv, if new tasks eg: (noise, environment class) has to be added, then update the dataset.py accordingly. bash pip install requirements.txt

Train

update the config in train.py, such as audioencodername, llm_name, etc and other hyper parameters. bash python train.py

Evaluation

After training, update checkpoint path and test dataset path(similar format to train/dev.csv). bash python test.py

Infer model in Streamlit app

bash streamlit run app.py

Disclaimer

The models provided in this repository are not perfect and may produce errors in Automatic Speech Recognition (ASR), gender identification, age estimation, accent recognition, and emotion detection. Additionally, these models may exhibit biases related to gender, age, accent, and emotion. Please use with caution, especially in production environments, and be aware of potential inaccuracies and biases.

License

This project is released under the Apache 2.0 license as found in the LICENSE file. The released checkpoints, and code are intended for research purpose subject to the license of facebook/hubert-xlarge-ll60k, microsoft/wavlm-large and TinyLlama/TinyLlama-1.1B-Chat-v1.0 models.

Cite

@misc{Rajaa_SpeechLLM_Multi-Modal_LLM, author = {Rajaa, Shangeth and Tushar, Abhinav}, title = {{SpeechLLM: Multi-Modal LLM for Speech Understanding}}, url = {https://github.com/skit-ai/SpeechLLM} }