tts-mos-test-mturk

Command-line interface (CLI) to evaluate text-to-speech (TTS) mean opinion score (MOS) studies done on Amazon Mechanical Turk (MTurk).

Features

• init: initialize project from .json-file
• masks
  • create: create empty mask
  • mask-workers-by-id: mask workers by their WorkerId
  • mask-workers-by-age-group: mask workers by their age group
  • mask-workers-by-gender: mask workers by their gender
  • mask-workers-by-assignments-count: mask workers by their count of assignments
  • mask-workers-by-masked-ratings-count: mask workers by their count of masked ratings
  • mask-workers-by-correlation: mask workers by their algorithm/sentence correlation
  • mask-workers-by-correlation-percent: mask workers by their algorithm/sentence correlation (percentage-wise)
  • mask-assignments-by-id: mask assignments by their AssignmentId
  • mask-assignments-by-device: mask assignments by their listening device
  • mask-assignments-by-status: mask assignments by their status
  • mask-assignments-by-time: mask assignments by their submit time
  • mask-rating-outliers: mask outlying ratings
  • merge-masks: merge masks together
  • reverse-mask: reverse mask
• stats
  • print-mos: print MOS and CI95
  • print-masking-stats: print masking statistics
  • print-worker-stats: print worker statistics for each algorithm
  • print-assignment-stats: print assignment statistics for each worker
  • print-sentence-stats: print sentence statistics for each algorithm
  • print-data: print all data points
• mturk
  • prepare-approval: generate approval file
  • prepare-rejection: generate rejection file
  • prepare-bonus-payment: generate bonus payment file

Installation

sh pip install tts-mos-test-mturk --user

Usage

```txt usage: mos-cli [-h] [-v] {init,masks,stats,mturk} ...

CLI to evaluate text-to-speech MOS studies done on MTurk.

positional arguments: {init,masks,stats,mturk} description init initialize project from .json-file masks masks commands stats stats commands mturk mturk commands

options: -h, --help show this help message and exit -v, --version show program's version number and exit ```

Project JSON example

json { "algorithms": [ "alg1", "alg2", "alg3", "alg4" ], "files": [ "file1", "file2", "file3" ], "workers": { "worker1": { "gender": "male", "age_group": "18-29", "assignments": { "assignment1": { "device": "headphone", "state": "Approved", "hit": "hit1", "time": "13.07.23 05:08:04", "ratings": [ { "algorithm": "alg1", "file": "file1", "votes": { "naturalness": 3, "intelligibility": 5 } }, { "algorithm": "alg2", "file": "file3", "votes": { "naturalness": 2, "intelligibility": 4 } } ] } } } } }

For a longer example see etc/example.json. It contains 4 algorithms and 120 files which were rated by 36 dummy workers in batches of 10 files per assignment. An example parsing of that file is under etc/example.sh.

Roadmap

• add masks mask-assignments-not-of-last-month/week/day
• make device, state, hit and time optional
• make tax value optional

Dependencies

• numpy
• pandas
• tqdm
• ordered-set>=4.1.0
• mean-opinion-score==0.0.2

Contributing

If you notice an error, please don't hesitate to open an issue.

Development setup

```sh

update

sudo apt update

install Python 3.8, 3.9, 3.10 & 3.11 for ensuring that tests can be run

sudo apt install python3-pip \ python3.8 python3.8-dev python3.8-distutils python3.8-venv \ python3.9 python3.9-dev python3.9-distutils python3.9-venv \ python3.10 python3.10-dev python3.10-distutils python3.10-venv \ python3.11 python3.11-dev python3.11-distutils python3.11-venv

install pipenv for creation of virtual environments

python3.8 -m pip install pipenv --user

check out repo

git clone https://github.com/stefantaubert/tts-mos-test-mturk.git cd tts-mos-test-mturk

create virtual environment

python3.8 -m pipenv install --dev ```

Running the tests

```sh

first install the tool like in "Development setup"

then, navigate into the directory of the repo (if not already done)

cd tts-mos-test-mturk

activate environment

python3.8 -m pipenv shell

run tests

tox ```

Final lines of test result output:

log py38: commands succeeded py39: commands succeeded py310: commands succeeded py311: commands succeeded congratulations :)

License

MIT License

Acknowledgments

• Ribeiro, F., Florêncio, D., Zhang, C., & Seltzer, M. (2011). CrowdMOS: An approach for crowdsourcing mean opinion score studies. 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2416–2419. https://doi.org/10.1109/ICASSP.2011.5946971

Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – Project-ID 416228727 – CRC 1410

Citation

If you want to cite this repo, you can use this BibTeX-entry generated by GitHub (see About => Cite this repository).

txt Taubert, S. (2024). tts-mos-test-mturk (Version 0.0.2) [Computer software]. https://doi.org/10.5281/zenodo.10461539

Changelog

• v0.0.2 (2024-01-05)
  • Bugfix:
  • mask mask-workers-by-correlation-percent: sorting was not always correct
  • Added:
  • mask mask-workers-by-correlation-percent: added option to include masked workers in percentage calculation
  • mturk prepare-bonus-payment: added logging of fees for Mechanical Turk
  • mturk prepare-approval: added logging of fees for Mechanical Turk
  • added mask mask-assignments-by-status
  • added mask mask-assignments-by-time
  • added parsing of HITId
  • added option to mask assignments before preparing rejection CSV
  • added ! before mask name reverses mask on input
  • added masks mask-assignments-by-id
  • added masks mask-workers-by-id
  • Changed:
  • moved template creation and preparation to another repository
  • removed worktime parsing
• v0.0.1 (2023-02-23)
  • Initial release