https://github.com/alan-turing-institute/arc-m4st
Evaluating metrics for speech translation
Science Score: 26.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
○Academic publication links
-
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (9.6%) to scientific vocabulary
Repository
Evaluating metrics for speech translation
Basic Info
- Host: GitHub
- Owner: alan-turing-institute
- License: mit
- Language: Python
- Default Branch: main
- Size: 4.61 MB
Statistics
- Stars: 0
- Watchers: 4
- Forks: 0
- Open Issues: 3
- Releases: 0
Metadata Files
README.md
Metrics for Speech Translation (M4ST)
Evaluation of metrics for Speech Translation
Installation
From source:
bash
git clone https://github.com/alan-turing-institute/ARC-M4ST
cd ARC-M4ST
python -m pip install .
Usage
The code used to generate the experiment results for the main experiments with the WMT, DEMETR, and CallHome datasets is in the scripts directory, which all have readmes describing their data sources and main scripts. It also contains a 4th exploratory experiment looking at the potential impact of transcription errors.
Metric classes for ChrF, BLEU, COMET, MetricX, and BLASER-2.0 are defined in src/m4st/metrics. All of them provide a get_scores method which takes an instance of a TranslationDataset, which takes lists of predictions, and optionally sources, references, and languages (see the docstring of the TranslationDataset class).
Compiling notes
brew install typst
typst compile notes.typ
Ollama
To use the Ollama client, which is one way to corrupt sentences randomly, you need to install https://ollama.com, and run the server.
License
Distributed under the terms of the MIT license. Some of the datasets used and metric code use different permissive licenses - see the relevant files/their documentation.
Owner
- Name: The Alan Turing Institute
- Login: alan-turing-institute
- Kind: organization
- Email: info@turing.ac.uk
- Website: https://turing.ac.uk
- Repositories: 477
- Profile: https://github.com/alan-turing-institute
The UK's national institute for data science and artificial intelligence.
GitHub Events
Total
- Issues event: 26
- Delete event: 7
- Issue comment event: 21
- Member event: 3
- Push event: 115
- Pull request review comment event: 69
- Pull request review event: 49
- Pull request event: 23
- Create event: 15
Last Year
- Issues event: 26
- Delete event: 7
- Issue comment event: 21
- Member event: 3
- Push event: 115
- Pull request review comment event: 69
- Pull request review event: 49
- Pull request event: 23
- Create event: 15
Issues and Pull Requests
Last synced: 10 months ago
All Time
- Total issues: 15
- Total pull requests: 17
- Average time to close issues: about 1 month
- Average time to close pull requests: 21 days
- Total issue authors: 3
- Total pull request authors: 3
- Average comments per issue: 1.0
- Average comments per pull request: 0.88
- Merged pull requests: 9
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 15
- Pull requests: 17
- Average time to close issues: about 1 month
- Average time to close pull requests: 21 days
- Issue authors: 3
- Pull request authors: 3
- Average comments per issue: 1.0
- Average comments per pull request: 0.88
- Merged pull requests: 9
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- klh5 (11)
- boykovdn (2)
- jack89roberts (2)
Pull Request Authors
- jack89roberts (6)
- klh5 (6)
- boykovdn (6)
Top Labels
Issue Labels
Pull Request Labels
Dependencies
- actions/checkout v4 composite
- actions/setup-python v4 composite
- codecov/codecov-action v3.1.4 composite
- pre-commit/action v3.0.0 composite