partial-embedding-matrix-adaptation

Vocabulary-level memory efficiency for language model fine-tuning.

https://github.com/mlsw/partial-embedding-matrix-adaptation

Science Score: 41.0%

This score indicates how likely this project is to be science-related based on various indicators:

✓
CITATION.cff file
Found CITATION.cff file
✓
codemeta.json file
Found codemeta.json file
○
.zenodo.json file
○
DOI references
✓
Academic publication links
Links to: arxiv.org
○
Academic email domains
○
Institutional organization owner
○
JOSS paper metadata
○
Scientific vocabulary similarity
Low similarity (14.2%) to scientific vocabulary

Keywords

bert huggingface nlp transformers

Last synced: 6 months ago · JSON representation ·

Repository

Vocabulary-level memory efficiency for language model fine-tuning.

Basic Info

Host: GitHub
Owner: mlsw
License: mit
Language: Python
Default Branch: main
Homepage:
Size: 44.9 KB

Statistics

Stars: 6
Watchers: 2
Forks: 0
Open Issues: 0
Releases: 0

Topics

bert huggingface nlp transformers

Created over 2 years ago · Last pushed 11 months ago

Metadata Files

Readme License Citation

Partial Embedding Matrix Adaptation

Diagram of Partial Embedding Matrix Adaptation

Introduction

This repository contains the implementation for the paper Vocabulary-level Memory Efficiency for Language Model Fine-tuning.

Partial Embedding Matrix Adaptation is a simple technique that can reduce the memory footprint of language model fine-tuning without impacting performance.

Installation

pip install git+https://github.com/mlsw/partial-embedding-matrix-adaptation.git

Usage

Hugging Face Transformers

There is a high-level API for Hugging Face Transformers PyTorch models via the HFEmbeddingPruner class.

```python from partialembeddingmatrix_adaptation import HFEmbeddingPruner

embeddingpruner = HFEmbeddingPruner(model) dataset, _ = embeddingpruner.prepare_model(tokenizer, dataset) ```

Please see examples/distilbert_sst2.py for a complete example. Additionally, the scripts in the utils directory show how to use this API with the Hugging Face Transformers Trainer.

PyTorch

Alternatively, the EmbeddingPruner class can be used directly for PyTorch models. Please see HFEmbeddingPruner for an example of how to use this.

Reproducibility

The following scripts can be used to reproduce the results from the paper. These are adapted from Hugging Face Transformers PyTorch Examples with support for Partial Embedding Matrix Adaptation.

| Task | Script | Documentation | | ---- | ------ | --------------| | GLUE | rungluepema.py | Here | | XNLI | runxnlipema.py | Here |

License

This project is licensed under the terms of the MIT license. Please see LICENSE for more details.

Citation

If you found this work useful, please consider citing our paper:

bibtex @misc{williams-aletras-2025-vocabulary, title={Vocabulary-level Memory Efficiency for Language Model Fine-tuning}, author={Miles Williams and Nikolaos Aletras}, year={2025}, eprint={2309.08708}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2309.08708} }

Owner

Name: Miles Williams
Login: mlsw
Kind: user
Location: United Kingdom
Company: University of Sheffield

Twitter: miles_wil
Repositories: 1
Profile: https://github.com/mlsw

PhD student at @SheffieldNLP

Citation (CITATION.bib)

@misc{williams-aletras-2025-vocabulary,
  title={Vocabulary-level Memory Efficiency for Language Model Fine-tuning}, 
  author={Miles Williams and Nikolaos Aletras},
  year={2025},
  eprint={2309.08708},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2309.08708}
}

GitHub Events

Total

Watch event: 3
Push event: 3

Last Year

Watch event: 3
Push event: 3

Issues and Pull Requests

Last synced: 10 months ago

All Time

Total issues: 0
Total pull requests: 0
Average time to close issues: N/A
Average time to close pull requests: N/A
Total issue authors: 0
Total pull request authors: 0
Average comments per issue: 0
Average comments per pull request: 0
Merged pull requests: 0
Bot issues: 0
Bot pull requests: 0

Past Year

Issues: 0
Pull requests: 0
Average time to close issues: N/A
Average time to close pull requests: N/A
Issue authors: 0
Pull request authors: 0
Average comments per issue: 0
Average comments per pull request: 0
Merged pull requests: 0
Bot issues: 0
Bot pull requests: 0

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Open Source Science