https://github.com/bramvanroy/charcut
Character-based MT evaluation and difference highlighting
Science Score: 18.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
✓CITATION.cff file
Found CITATION.cff file -
○codemeta.json file
-
○.zenodo.json file
-
○DOI references
-
○Academic publication links
-
○Committers with academic emails
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (13.6%) to scientific vocabulary
Repository
Character-based MT evaluation and difference highlighting
Basic Info
- Host: GitHub
- Owner: BramVanroy
- License: other
- Language: Python
- Default Branch: master
- Size: 92.8 KB
Statistics
- Stars: 1
- Watchers: 0
- Forks: 1
- Open Issues: 2
- Releases: 0
Metadata Files
README.md
CharCut
Character-based MT evaluation and difference highlighting
CharCut compares outputs of MT systems with reference translations. It can compare multiple file pairs simultaneously and produce HTML outputs showing character-based differences along with scores that are directly inferred from the lengths of those differences, thus making the link between evaluation and visualisation straightforward.
The matching algorithm is based on an iterative search for longest common substrings, combined with a length-based threshold that limits short and noisy character matches. As a similarity metric this is not new, but to the best of our knowledge it was never applied to highlighting and scoring of MT outputs. It has the neat effect of keeping character-based differences readable by humans.
Accidentally, the scores inferred from those differences correlate very well with human judgments, similarly to other great character-based metrics like chrF(++) or CharacTER. It was evaluated here:
Adrien Lardilleux and Yves Lepage: "CharCut: Human-Targeted Character-Based MT Evaluation with Loose Differences". In Proceedings of IWSLT 2017.
It is intended to be lightweight and easy to use, so the HTML outputs are, and will be kept, slick on purpose.
Note (Bram Vanroy): the remainder of this README has been changed to reflect the changes I have made to make the package more usable from a Python package perspective, e.g., by using hypotheses/references directly without files.
Installation
shell
pip install charcut
This will install the command calculate-charcut.
Basic usage:
shell
calculate-charcut cand.txt,ref.txt
where cand.txt and ref.txt contain corresponding candidate (MT) and reference (human) segments, 1 per line. Multiple file pairs can be specified on the command line: candidates with references, candidates with other candidates, etc.
By default, only document-level scores are displayed on standard output. To produce an HTML output file, use the -o option:
shell
calculate-charcut cand.txt,ref.txt -o mydiff.html
A few more options are available; call
shell
calculate-charcut -h
to list them.
Consider lowering the -m option value (minimum match size) for non-alphabetical writing systems such as Chinese or Japanese. The default value (3 characters) should be acceptable for most European languages, but depending on the language and data, larger values might produce better looking results.
Modifications by Bram Vanroy
Bram Vanroy made some changes to this package that do not affect the result of the metric but that should improve usability. He also packaged the library for pip and added some tests to ensure the same results with the original library. Code has been rewritten to make it easier to use from within Python without the need of files as input. In Python, the following entry point now exists:
python
def calculate_charcut(
hyps: Union[str, List[str]],
refs: Union[str, List[str]],
html_output_file: str = None,
plain_output_file: str = None,
src_file: str = None,
match_size: int = 3,
alt_norm: bool = False,
verbose: bool = False
) -> Tuple[float, int]:
where hyps and refs are indiviual sentences str or a list of sentences List[str]. This function has the same capabilities and arguments as the command-line script that is available (discussed above). This command line script is now available as an installed entry point rather than a separate Python script. You can call that from the command line with calculate-charcut.
License
Owner
- Name: Bram Vanroy
- Login: BramVanroy
- Kind: user
- Location: Belgium
- Company: @CCL-KULeuven @instituutnederlandsetaal
- Website: https://bramvanroy.github.io/
- Repositories: 29
- Profile: https://github.com/BramVanroy
👋 My name is Bram and I work on natural language processing and machine translation (evaluation) but I also spend a lot of time in this open-source world 🌍
Citation (CITATION)
@inproceedings{lardilleux-lepage-2017-charcut,
title = "{CHARCUT}: Human-Targeted Character-Based {MT} Evaluation with Loose Differences",
author = "Lardilleux, Adrien and
Lepage, Yves",
booktitle = "Proceedings of the 14th International Conference on Spoken Language Translation",
month = dec # " 14-15",
year = "2017",
address = "Tokyo, Japan",
publisher = "International Workshop on Spoken Language Translation",
url = "https://aclanthology.org/2017.iwslt-1.20",
pages = "146--153",
abstract = "We present CHARCUT, a character-based machine translation evaluation metric derived from a human-targeted segment difference visualisation algorithm. It combines an iterative search for longest common substrings between the candidate and the reference translation with a simple length-based threshold, enabling loose differences that limit noisy character matches. Its main advantage is to produce scores that directly reflect human-readable string differences, making it a useful support tool for the manual analysis of MT output and its display to end users. Experiments on WMT16 metrics task data show that it is on par with the best {``}un-trained{''} metrics in terms of correlation with human judgement, well above BLEU and TER baselines, on both system and segment tasks.",
}
GitHub Events
Total
Last Year
Committers
Last synced: over 3 years ago
All Time
- Total Commits: 37
- Total Committers: 3
- Avg Commits per committer: 12.333
- Development Distribution Score (DDS): 0.297
Top Committers
| Name | Commits | |
|---|---|---|
| BramVanroy | B****y@U****e | 26 |
| alardill | 3****l@u****m | 10 |
| Adrien Lardilleux | a****x@e****u | 1 |
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: over 2 years ago
All Time
- Total issues: 2
- Total pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Total issue authors: 2
- Total pull request authors: 0
- Average comments per issue: 0.5
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 1
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 1
- Pull request authors: 0
- Average comments per issue: 1.0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- BramVanroy (1)
- zouharvi (1)
Pull Request Authors
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 1
-
Total downloads:
- pypi 427 last-month
- Total dependent packages: 2
- Total dependent repositories: 4
- Total versions: 2
- Total maintainers: 1
pypi.org: charcut
Character-based MT evaluation and difference highlighting
- Homepage: https://github.com/BramVanroy/CharCut
- Documentation: https://charcut.readthedocs.io/
- License: GPLv3
-
Latest release: 1.1.1
published over 3 years ago