🪐 LiBERTus - A Multilingual Language Model for Ancient and Historical Languages

Submission to Task 1 (Constrained) of the SIGTYP 2024 Shared Task on Word Embedding Evaluation for Ancient and Historical Languages. The system is built by first pretraining a multilingual language model and then finetuning it for a downstream task. The submission for Phase 1 and 2 of the Shared Task can be found in the submission_p1 and submission_p2 directories.

📋 project.yml

The project.yml defines the data assets required by the project, as well as the available commands and workflows. For details, see the Weasel documentation.

⏯ Commands

The following commands are defined by the project. They can be executed using weasel run [name]. Commands are only re-run if their inputs have changed.

| Command | Description | | --- | --- | | create-pretraining | Create corpus for multilingual LM pretraining | | create-vocab | Train a tokenizer to create a vocabulary | | pretrain-model | Pretrain a multilingual LM from a corpus | | pretrain-model-from-checkpoint | Pretrain a multilingual LM from a corpus based on a checkpoint | | upload-to-hf | Upload pretrained model and corresponding tokenizer to the HuggingFace repository | | convert-to-spacy-merged | Convert CoNLL-U files into spaCy format for finetuning | | convert-to-spacy | Convert CoNLL-U files into spaCy format for finetuning | | finetune-tok2vec-model | Finetune a tok2vec model given a training and validation corpora | | finetune-trf-model | Finetune a transformer model given a training and validation corpora | | finetune-with-merged-corpus | Finetune a transformer model on the combined training and validation corpora | | package-model | Package model and upload to HuggingFace | | evaluate-model-dev | Evaluate a model on the validation set | | plot-figures | Plot figures for the writeup | | setup-test | Install models from HuggingFace via pip | | download-models-locally | Download models from HuggingFace | | get-test-results | Get results from the test file | | zip-results-p1 | Zip the results into a single file for submission (Phase 1) | | zip-results-p2 | Zip teh results into a single file for submission (Phase 2) |

⏭ Workflows

The following workflows are defined by the project. They can be executed using weasel run [name] and will run the specified commands in order. Commands are only re-run if their inputs have changed.

| Workflow | Steps | | --- | --- | | pretrain | create-pretraining → create-vocab → pretrain-model | | finetune | convert-to-spacy → finetune-trf-model → evaluate-model-dev | | experiment-merged | convert-to-spacy-merged → finetune-with-merged-corpus | | experiment-sampling | create-vocab → pretrain-model | | make-submission-p1 | setup-test → get-test-results → zip-results-p1 | | make-submission-p2 | download-models-locally → zip-results-p2 |

🗂 Assets

The following assets are defined by the project. They can be fetched by running weasel assets in the project directory.

| File | Source | Description | | --- | --- | --- | | assets/train/ | Git | CoNLL-U training datasets for Task 0 (morphology/lemma/POS) | | assets/dev/ | Git | CoNLL-U validation datasets for Task 0 (morphology/lemma/POS) | | assets/test/ | Git | CoNLL-U test datasets for Task 0 (morphology/lemma/POS) |

📄 Cite

If you used any of the code or the models, don't forget to cite

@inproceedings{miranda-2024-allen, title = "{A}llen Institute for {AI} @ {SIGTYP} 2024 Shared Task on Word Embedding Evaluation for Ancient and Historical Languages", author = "Miranda, Lester", booktitle = "Proceedings of the 6th Workshop on Research in Computational Linguistic Typology and Multilingual NLP", month = mar, year = "2024", address = "St. Julian's, Malta", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2024.sigtyp-1.18", pages = "151--159", }