boudams_like_tokenizer
Science Score: 26.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
○Academic publication links
-
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (6.5%) to scientific vocabulary
Repository
Basic Info
- Host: GitHub
- Owner: matgille
- Language: Python
- Default Branch: master
- Size: 199 MB
Statistics
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
- Releases: 0
Metadata Files
README.md
Boudams like segmenter
Fork from Thibault Clrice's Boudams tokenizer Clrice 2020.
See also Ben Trevett's notebook from which part of the code is derived.
What differs from Boudams
Besides <PAD>, <SOS> and <EOS>, four classes are predicted, instead of two. This allows to keep the original segmentation
information:
- Word Boundary > Word Content (remove a space). Class <S-s> Example: fizo< >los
- Word Content > Word boundary (add a space). Class <s-S> Example: d<e>las
- Word Content (do nothing). Class <S-S> Example: caua<u>lleria
- Word Boundary (do nothing). Class <s-s> Example: la< >cauallera
Training a model
The program expects correctly segmented lines of text (with line breaks) as the training data.
Set path to test and train sets in tokeniser.py file.
python tokeniser.py train
Tagging text and xml-tei
The tagger supports txt and xml-tei inputs/outputs (as long as they contain tei:lb elements that mark line begginings).
It allows to produce an XML file with entities to easier
indicate the segmentation information [Stutzmann 2013, Pinche 2017].
References
- Clrice, Thibault. Evaluating Deep Learning Methods for Word Segmentation of Scripta Continua Texts in Old French and Latin. Journal of Data Mining & Digital Humanities 2020 (April 7, 2020). Accessed February 26, 2021. https://jdmdh.episciences.org/6264/pdf.
- Pinche, Ariane. dition Nativement Numrique Des Oeuvres Hagiographiques Li Seint Confessor de Wauchier de Denain, daprs Le Manuscrit 412 de La Bibliothque Nationale de France, 2017. Accessed November 10, 2021. https://hal.archives-ouvertes.fr/hal-01628533.
- Stutzmann, Dominique. Ontologie des formes et encodage des textes manuscrits mdivaux. Document numrique Vol. 16, no. 3 (2013): 8195.
Owner
- Name: Matthias Gille Levenson
- Login: matgille
- Kind: user
- Repositories: 3
- Profile: https://github.com/matgille
PhD. candidate in castilian medieval and computational philology. My other repos are on the Huma-Num gitlab server: https://gitlab.huma-num.fr/mgillelevenson
GitHub Events
Total
- Push event: 12
Last Year
- Push event: 12