UTMOSv2: UTokyo-SaruLab MOS Prediction System

🎤✨ Official implementation of ✨🎤
“The T05 System for The VoiceMOS Challenge 2024:
Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speech”
🏅🎉 accepted by IEEE Spoken Language Technology Workshop (SLT) 2024. 🎉🏅

ꔫ･-･ꔫ･-･ꔫ･-･ꔫ･-･ꔫ･-･ꔫ･-･ꔫ･-･ꔫ

✨  UTMOSv2 achieved 1st place in 7 out of 16 metrics  ✨
✨🏆    and 2nd place in the remaining 9 metrics    🏆✨
✨    in the VoiceMOS Challenge 2024 Track1!    ✨

🚀 Quick Prediction

✨ You can easily use the pretrained UTMOSv2 model!

🛠️ Using in your Python code 🛠️

If you want to make predictions using the UTMOSv2 library, follow these steps:

1. Install the UTMOSv2 library from GitHub

bash pip install git+https://github.com/sarulab-speech/UTMOSv2.git

1. Make predictions

   • To predict the MOS of a single .wav file:

   python import utmosv2 model = utmosv2.create_model(pretrained=True) mos = model.predict(input_path="/path/to/wav/file.wav")

• To predict the MOS of all .wav files in a folder:

  python import utmosv2 model = utmosv2.create_model(pretrained=True) mos = model.predict(input_dir="/path/to/wav/dir/")

[!NOTE] Either input_path or input_dir must be specified, but not both.

📜 Using the inference script 📜

If you want to make predictions using the inference script, follow these steps:

1. Clone this repository and navigate to UTMOSv2 folder

bash git clone https://github.com/sarulab-speech/UTMOSv2.git cd UTMOSv2

1. Install Package

bash pip install --upgrade pip # enable PEP 660 support pip install -e .[optional] # install with optional dependencies

1. Make predictions

   • To predict the MOS of a single .wav file:

   bash python inference.py --input_path /path/to/wav/file.wav --out_path /path/to/output/file.csv

• To predict the MOS of all .wav files in a folder:

  bash python inference.py --input_dir /path/to/wav/dir/ --out_path /path/to/output/file.csv

[!NOTE] If you are using zsh, make sure to escape the square brackets like this:

zsh pip install -e '.[optional]'

[!TIP] If --out_path is not specified, the prediction results will be output to the standard output. This is particularly useful when the number of files to be predicted is small.

[!NOTE] Either --input_path or --input_dir must be specified, but not both.

[!NOTE] These methods provide quick and simple predictions. For more accurate predictions and detailed usage of the inference script, please refer to the inference guide.

🤗 You can try a simple demonstration on Hugging Face Space:

⚒️ Train UTMOSv2 Yourself

If you want to train UTMOSv2 yourself, please refer to the training guide. To reproduce the training as described in the paper or used in the competition, please refer to this document.

📂 Used Datasets

Details of the datasets used in this project can be found in the datasets documentation.

🔖 Citation

If you find UTMOSv2 useful in your research, please cite the following paper:

bibtex @inproceedings{baba2024utmosv2, title = {The T05 System for The {V}oice{MOS} {C}hallenge 2024: Transfer Learning from Deep Image Classifier to Naturalness {MOS} Prediction of High-Quality Synthetic Speech}, author = {Baba, Kaito and Nakata, Wataru and Saito, Yuki and Saruwatari, Hiroshi}, booktitle = {IEEE Spoken Language Technology Workshop (SLT)}, year = {2024}, }