Data

Tools

Indexes

Applications

Experiments

Open Source Science

mean-opinion-score

Python library for calculating the mean opinion score and 95% confidence interval of the standard deviation of text-to-speech ratings according to Ribeiro et al. (2011).

https://github.com/stefantaubert/mean-opinion-score

Science Score: 67.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ✓
    CITATION.cff file
    Found CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ✓
    DOI references
    Found 11 DOI reference(s) in README
  • ✓
    Academic publication links
    Links to: zenodo.org
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (11.2%) to scientific vocabulary

Keywords

intelligibility mos naturalness speech-synthesis subjective-evaluation text-to-speech tts
Last synced: 6 months ago · JSON representation ·

Repository

Python library for calculating the mean opinion score and 95% confidence interval of the standard deviation of text-to-speech ratings according to Ribeiro et al. (2011).

Basic Info
  • Host: GitHub
  • Owner: stefantaubert
  • License: mit
  • Language: Python
  • Default Branch: master
  • Homepage:
  • Size: 77.1 KB
Statistics
  • Stars: 24
  • Watchers: 1
  • Forks: 1
  • Open Issues: 0
  • Releases: 2
Topics
intelligibility mos naturalness speech-synthesis subjective-evaluation text-to-speech tts
Created almost 3 years ago · Last pushed about 1 year ago
Metadata Files
Readme License Citation

README.md

mean-opinion-score

PyPI PyPI MIT PyPI PyPI PyPI DOI

Python library for calculating the mean opinion score (MOS) and 95% confidence interval (CI) of the standard deviation (SD) of text-to-speech (TTS) ratings according to "Ribeiro, F., Florêncio, D., Zhang, C., & Seltzer, M. (2011). CrowdMOS: An approach for crowdsourcing mean opinion score studies". To determine CIs, the authors used a two-way random effects model with the variables: diversity of intrinsic sentence quality, diversity of rater preference, and subjective uncertainty.

Installation

sh pip install mean-opinion-score --user

Usage

```py import numpy as np

from meanopinionscore import getci95, getci95default, getmos

_ = np.nan

ratings = np.array([ # columns represent sentences [4, 5, , 4, _, 3], # rater 1 [4, 4, 4, 5, _, 4], # rater 2 [, 3, 5, 4, , 1], # rater 3 [, _, _, _, _, _], # rater 4 ])

mos = getmos(ratings) ci = getci95(ratings) cidefault = getci95_default(ratings)

print(f"MOS: {mos:.2f} ± {ci:.4f}") print(f"MOS: {mos:.2f} ± {ci_default:.4f}")

MOS: 3.85 ± 1.3316

MOS: 3.85 ± 0.5579

```

Dependencies

  • numpy
  • scipy

Contributing

If you notice an error, please don't hesitate to open an issue.

Development setup

```sh

update

sudo apt update

install Python 3.6, 3.7, 3.8, 3.9, 3.10 & 3.11 for ensuring that tests can be run

sudo apt install python3-pip \ python3.6 python3.6-dev python3.6-distutils python3.6-venv \ python3.7 python3.7-dev python3.7-distutils python3.7-venv \ python3.8 python3.8-dev python3.8-distutils python3.8-venv \ python3.9 python3.9-dev python3.9-distutils python3.9-venv \ python3.10 python3.10-dev python3.10-distutils python3.10-venv \ python3.11 python3.11-dev python3.11-distutils python3.11-venv

install pipenv for creation of virtual environments

python3.11 -m pip install pipenv --user

check out repo

git clone https://github.com/stefantaubert/mean-opinion-score.git cd mean-opinion-score

create virtual environment

python3.11 -m pipenv install --dev ```

Running the tests

```sh

first install the tool like in "Development setup"

then, navigate into the directory of the repo (if not already done)

cd mean-opinion-score

activate environment

python3.11 -m pipenv shell

run tests

tox ```

Final lines of test result output:

log py36: OK py37: OK py38: OK py39: OK py310: OK py311: OK congratulations :)

License

MIT License

Acknowledgments

MOS and CI calculation is taken from:

  • Ribeiro, F., Florêncio, D., Zhang, C., & Seltzer, M. (2011). CrowdMOS: An approach for crowdsourcing mean opinion score studies. 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2416–2419. https://doi.org/10.1109/ICASSP.2011.5946971

Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – Project-ID 416228727 – CRC 1410.

Citation

If you want to cite this repo, you can use this BibTeX-entry generated by GitHub (see About => Cite this repository).

txt Taubert, S. (2023). mean-opinion-score (Version 0.0.2) [Computer software]. https://doi.org/10.5281/zenodo.8238259

Changelog

  • v0.0.2 (2023-08-11)
    • Added:
    • commonly used 95% confidence interval calculation
  • v0.0.1 (2023-02-23)
    • Initial release

Owner

  • Name: Stefan Taubert
  • Login: stefantaubert
  • Kind: user
  • Location: Chemnitz, Germany
  • Company: Chemnitz University of Technology

Currently I am working on my PhD about the topic of speech synthesis at Chemnitz University of Technology.

Citation (CITATION.cff) 

cff-version: 1.2.0
title: mean-opinion-score
abstract: Library for calculating the mean opinion score and 95% confidence interval of the standard deviation of text-to-speech ratings according to Ribeiro et al. (2011).
message: >-
  If you use this software, please cite it using the
  metadata from this file.
type: software
authors:
  - email: github@stefantaubert.com
    given-names: Stefan
    family-names: Taubert
    affiliation: Chemnitz University of Technology
    orcid: 'https://orcid.org/0000-0002-4932-2874'
    website: 'https://stefantaubert.com/'
version: 0.0.2
date-released: 2023-08-11
license: MIT
url: https://github.com/stefantaubert/mean-opinion-score
doi: 10.5281/zenodo.8238259

GitHub Events

Total
  • Watch event: 3
  • Push event: 1
Last Year
  • Watch event: 3
  • Push event: 1

Committers

Last synced: 7 months ago

All Time
  • Total Commits: 9
  • Total Committers: 1
  • Avg Commits per committer: 9.0
  • Development Distribution Score (DDS): 0.0
Past Year
  • Commits: 1
  • Committers: 1
  • Avg Commits per committer: 1.0
  • Development Distribution Score (DDS): 0.0
Top Committers
Name Email Commits
Stefan Taubert 2****t 9

Issues and Pull Requests

Last synced: 6 months ago

All Time
  • Total issues: 0
  • Total pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Total issue authors: 0
  • Total pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 0
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
Pull Request Authors
Top Labels
Issue Labels
Pull Request Labels

Packages

  • Total packages: 1
  • Total downloads:
    • pypi 40 last-month
  • Total dependent packages: 1
  • Total dependent repositories: 0
  • Total versions: 2
  • Total maintainers: 1
pypi.org: mean-opinion-score

Library for calculating the mean opinion score and 95% confidence interval of the standard deviation of text-to-speech ratings according to Ribeiro et al. (2011).

  • Versions: 2
  • Dependent Packages: 1
  • Dependent Repositories: 0
  • Downloads: 40 Last month
Rankings
Dependent packages count: 6.6%
Average: 18.6%
Dependent repos count: 30.6%
Maintainers (1)
stefantaubert
Last synced: 6 months ago