Note: This repository holds the codebase of the Adapters library, which has replaced adapter-transformers. For the legacy codebase, go to: https://github.com/adapter-hub/adapter-transformers-legacy.

Adapters

A Unified Library for Parameter-Efficient and Modular Transfer Learning

adapters is an add-on to HuggingFace's Transformers library, integrating adapters into state-of-the-art language models by incorporating AdapterHub, a central repository for pre-trained adapter modules.

Installation

adapters currently supports Python 3.8+ and PyTorch 1.10+. After installing PyTorch, you can install adapters from PyPI ...

pip install -U adapters

... or from source by cloning the repository:

git clone https://github.com/adapter-hub/adapters.git git checkout adapters cd adapters pip install .

Quick Tour

Load pre-trained adapters:

```python from adapters import AutoAdapterModel from transformers import AutoTokenizer

model = AutoAdapterModel.frompretrained("roberta-base") tokenizer = AutoTokenizer.frompretrained("roberta-base")

model.loadadapter("AdapterHub/roberta-base-pf-imdb", source="hf", setactive=True)

print(model(**tokenizer("This works great!", return_tensors="pt")).logits) ```

Learn More

Adapt existing model setups:

```python import adapters from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained("t5-base")

adapters.init(model)

model.addadapter("myloraadapter", config="lora") model.trainadapter("myloraadapter")

Your regular training loop...

```

Learn More

Flexibly configure adapters:

```python from adapters import ConfigUnion, PrefixTuningConfig, ParBnConfig, AutoAdapterModel

model = AutoAdapterModel.from_pretrained("microsoft/deberta-v3-base")

adapterconfig = ConfigUnion( PrefixTuningConfig(prefixlength=20), ParBnConfig(reductionfactor=4), ) model.addadapter("myadapter", config=adapterconfig, set_active=True) ```

Learn More

Easily compose adapters in a single model:

```python from adapters import AdapterSetup, AutoAdapterModel import adapters.composition as ac

model = AutoAdapterModel.from_pretrained("roberta-base")

qc = model.loadadapter("AdapterHub/roberta-base-pf-trec") sent = model.loadadapter("AdapterHub/roberta-base-pf-imdb")

with AdapterSetup(ac.Parallel(qc, sent)): print(model(**tokenizer("What is AdapterHub?", return_tensors="pt"))) ```

Learn More

Useful Resources

HuggingFace's great documentation on getting started with Transformers can be found here. adapters is fully compatible with Transformers.

To get started with adapters, refer to these locations:

• Colab notebook tutorials, a series notebooks providing an introduction to all the main concepts of (adapter-)transformers and AdapterHub
• https://docs.adapterhub.ml, our documentation on training and using adapters with adapters
• https://adapterhub.ml to explore available pre-trained adapter modules and share your own adapters
• Examples folder of this repository containing HuggingFace's example training scripts, many adapted for training adapters

Implemented Methods

Currently, adapters integrates all architectures and methods listed below:

| Method | Paper(s) | Quick Links | | --- | --- | --- | | Bottleneck adapters | Houlsby et al. (2019)
Bapna and Firat (2019) | Quickstart, Notebook | | AdapterFusion | Pfeiffer et al. (2021) | Docs: Training, Notebook | | MAD-X,
Invertible adapters | Pfeiffer et al. (2020) | Notebook | | AdapterDrop | Rücklé et al. (2021) | Notebook | | MAD-X 2.0,
Embedding training | Pfeiffer et al. (2021) | Docs: Embeddings, Notebook | | Prefix Tuning | Li and Liang (2021) | Docs | | Parallel adapters,
Mix-and-Match adapters | He et al. (2021) | Docs | | Compacter | Mahabadi et al. (2021) | Docs | | LoRA | Hu et al. (2021) | Docs | | (IA)^3 | Liu et al. (2022) | Docs | | UniPELT | Mao et al. (2022) | Docs |

Supported Models

We currently support the PyTorch versions of all models listed on the Model Overview page in our documentation.

Developing & Contributing

To get started with developing on Adapters yourself and learn more about ways to contribute, please see https://docs.adapterhub.ml/contributing.html.

Citation

If you use this library for your work, please consider citing our paper AdapterHub: A Framework for Adapting Transformers:

@inproceedings{pfeiffer2020AdapterHub, title={AdapterHub: A Framework for Adapting Transformers}, author={Pfeiffer, Jonas and R{\"u}ckl{\'e}, Andreas and Poth, Clifton and Kamath, Aishwarya and Vuli{\'c}, Ivan and Ruder, Sebastian and Cho, Kyunghyun and Gurevych, Iryna}, booktitle={Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations}, pages={46--54}, year={2020} }