adapters - Adapters v1.2.0

Blog post: https://adapterhub.ml/blog/2025/05/adapters-for-any-transformer

This version is built for Hugging Face Transformers v4.51.x.

New

Adapter Model Plugin Interface (@calpt via #738; @lenglaender via #797)

The new adapter model interface makes it easy to plug most adapter features into any new or custom Transformer model. Check out our release blog post for details. Also see https://docs.adapterhub.ml/plugin_interface.html.

Multi-Task composition with MTL-LoRA (@FrLdy via #792)

MTL-LoRA (Yang et al., 2024) is a new adapter composition method leveraging LoRA for multi-task learning. See https://docs.adapterhub.ml/multitaskmethods.html#mtl-lora.

VeRA - parameter-efficient LoRA variant (@julian-fong via #763)

VeRA (Kopiczko et al., 2024) is a LoRA adapter variant that requires even less trainable parameters. See https://docs.adapterhub.ml/methods.html#vera.

New Models (via new interface)

A couple of new models are supported out-of-the-box via the new adapter model plugin interface: - Gemma 2, Gemma 3 - ModernBERT - Phi 1, Phi 2 - Qwen 2, Qwen 2.5, Qwen 3

More

New init_weights_seed adapter config attribute to initialize adapters with identical weights (@TimoImhof via #786)
Support defining custom forward method args via ForwardContext (@calpt via #789)

Changed

Upgrade supported Transformers version (@calpt via #799; @TimoImhof via #805)

- Python
Published by calpt 9 months ago

adapters - Adapters v1.1.1

This version is built for Hugging Face Transformers v4.48.x.

Changes

Upgrade supported Transformers version to 4.48 (@TimoImhof via #782)

Fixed

Fix BatchSplit for PrefixTuning (@FrLdy via #750)

- Python
Published by calpt 11 months ago

adapters - Adapters v1.1.0

This version is built for Hugging Face Transformers v4.47.x.

New

Add AdapterPlus adapters (@julian-fong via #746, #775):

AdapterPlus (Steitz & Roth, 2024) is a new bottleneck adapter variant optimized for vision transformers. Check out our notebook for training AdapterPlus adapters for ViT models.

Easy saving, loading and pushing of full adapter compositions to the Hub (@calpt via #771):

The newly added save_adapter_setup(), load_adapter_setup() and push_adapter_setup_to_hub() methods allow saving, loading and uploading complex adapter compositions - including AdapterFusion setups - with one line of code. Read our documentation for more.

Enabling full gradient checkpointing support with adapters (@lenglaender via #759)

Gradient checkpointing is a technique for enabling fine-tuning in very memory-limited setttings that nicely complements efficient adapters. It is now supported across all integrated adapter methods. Check out our notebook for fine-tuning Llama with gradient checkpointing and adapters.

More

Custom names for AdapterFusion layers (@calpt via #774): Allows distinguishing multiple fusions on the same adapter by name. See details.
Allow specifying adapter dtype in AdapterConfig (@killershrimp via #767)

Changed

Major refactoring of library test suite (@TimoImhof via #740): See tests readme for details.
Upgrade supported Transformers version (@calpt via #757; @lenglaender via #776)

Fixed

Fix Mistral flash attention with adapters (@divyanshuaggarwal via #758)
Fix Bottleneck configs to work with lnbefore = True and initweights = "mam_adapter" (@julian-fong via #761)
Fix default Lora/ (IA)^3 scaling in forward (@calpt via #770) Note: This restores compatibility with adapter-transformers, but changes logic compared to previous Adapters versions. See #770 for details.
Fixes for sequence generation & orthogonal projection with ReFT (@calpt via #778)
Various minor compatibility and warning fixes (@calpt via #780, #787)

- Python
Published by calpt about 1 year ago

adapters - Adapters v1.0.1

This version is built for Hugging Face Transformers v4.45.x.

New

Add example notebook for ReFT training (@julian-fong via #741)

Changes

Upgrade supported Transformers version (@calpt via #751)

Fixed

Fix import error for Huggingface-hub version >=0.26.0 & Update Notebooks (@lenglaender via #750)
Fix output_embeddings get/ set for empty active head (@calpt via #754)
Fix LoRA parallel composition (@calpt via #752)

- Python
Published by calpt over 1 year ago

adapters - Adapters v1.0.0

Blog post: https://adapterhub.ml/blog/2024/08/adapters-update-reft-qlora-merging-models

This version is built for Hugging Face Transformers v4.43.x.

New Adapter Methods & Model Support

Add Representation Fine-Tuning (ReFT) implementation (LoReFT, NoReFT, DiReFT) (@calpt via #705)
Add LoRA weight merging with Task Arithmetics (@lenglaender via #698)
Add Whisper model support + notebook (@TimoImhof via #693; @julian-fong via #717)
Add Mistral model support (@KorventennFR via #609)
Add PLBart model support (@FahadEbrahim via #709)

Breaking Changes & Deprecations

Remove support for loading from archived Hub repository (@calpt via #724)
Remove deprecated addfusion() & trainfusion() methods (@calpt via #714)
Remove deprecated arguments in push_adapter_to_hub() method (@calpt via #724)
Deprecate support for passing Python lists to adapter activation (@calpt via #714)

Minor Fixes & Changes

Upgrade supported Transformers version (@calpt & @lenglaender via #712, #719, #727)
Fix SDPA/ Flash attention support for Llama (@calpt via #722)
Fix gradient checkpointing for Llama and for Bottleneck adapters (@calpt via #730)

- Python
Published by calpt over 1 year ago

adapters - Adapters v0.2.2

This version is built for Hugging Face Transformers v4.40.x.

New

Add example notebook for embeddings training & update docs (@hSterz via #706)

Changed

Upgrade supported Transformers version (@calpt via #697)
Add download redirect for AH adapters to HF (@calpt via #704)

Fixed

Fix saving adapter model with custom heads (@hSterz via #700)
Fix moving adapter head to device with adapter_to() (@calpt via #708)
Fix importing encoder-decoder adapter classes (@calpt via #711)

- Python
Published by calpt over 1 year ago

adapters - Adapters v0.2.1

This version is built for Hugging Face Transformers v4.39.x.

New

Support saving & loading via Safetensors and use_safetensors parameter (@calpt via #692)
Add adapter_to() method for moving & converting adapter weights (@calpt via https://github.com/adapter-hub/adapters/pull/699)

Fixed

Fix reading model info in get_adapter_info() for HF (@calpt via #695)
Fix load_best_model_at_end=True for AdapterTrainer with quantized training (@calpt via #699)

- Python
Published by calpt almost 2 years ago

adapters - Adapters v0.2.0

This version is built for Hugging Face Transformers v4.39.x.

New

Add support for QLoRA/ QAdapter training via bitsandbytes (@calpt via #663): Notebook Tutorial
Add dropout to bottleneck adapters (@calpt via #667)

Changed

Upgrade supported Transformers version (@lenglaender via #654; @calpt via #686)
Deprecate Hub repo in docs (@calpt via #668)
Switch resolving order if source not specified in load_adapter() (@calpt via #681)

Fixed

Fix DataParallel training with adapters (@calpt via #658)
Fix embedding Training Bug (@hSterz via #655)
Fix fp16/ bf16 for Prefix Tuning (@calpt via #659)
Fix Training Error with AdapterDrop and Prefix Tuning (@TimoImhof via #673)
Fix default cache path for adapters loaded from AH repo (@calpt via #676)
Fix skipping composition blocks in not applicable layers (@calpt via #665)
Fix Unipelt Lora default config (@calpt via #682)
Fix compatibility of adapters with HF Accelerate auto device-mapping (@calpt via #678)
Use default head dropout prob if not provided by model (@calpt via #685)

- Python
Published by calpt almost 2 years ago

adapters - Adapters v0.1.2

This version is built for Hugging Face Transformers v4.36.x.

New

Add MT5 support (@sotwi via #629)

Changed

Upgrade supported Transformers version (@calpt via #617)
Simplify XAdapterModel implementations (@calpt via #641)

Fixed

Fix prediction head loading for T5 (@calpt via #640)

- Python
Published by calpt almost 2 years ago

adapters - Adapters v0.1.1

This version is built for Hugging Face Transformers v4.35.x.

New

Add leave_out to LoRA and (IA)³ (@calpt via #608)

Fixed

Fix error in pushadapterto_hub() due to deprecated args (@calpt via https://github.com/adapter-hub/adapters/pull/613)
Fix Prefix-Tuning for T5 models where dkv != dmodel / num_heads (@calpt via https://github.com/adapter-hub/adapters/pull/621)
[Bart] Move CLS rep extraction from EOS tokens to head classes (@calpt via https://github.com/adapter-hub/adapters/pull/624)
Fix adapter activation with skip_layers/ AdapterDrop training (@calpt via #634)

Docs & Notebooks

Update notebooks & add new complex configuration demo notebook (@hSterz & @calpt via https://github.com/adapter-hub/adapters/pull/614)

- Python
Published by calpt about 2 years ago

adapters - Adapters 0.1.0

Blog post: https://adapterhub.ml/blog/2023/11/introducing-adapters/

With the new Adapters library, we fundamentally refactored the adapter-transformers library and added support for new models and adapter methods.

This version is compatible with Hugging Face Transformers version 4.35.2.

For a guide on how to migrate from adapter-transformers to Adapters have a look at https://docs.adapterhub.ml/transitioning.md. Changes are given compared to the latest adapters-transformers v3.2.1.

New Models & Adapter Methods

Add LLaMA model integration (@hSterz)
Add X-MOD model integration (@calpt via #581)
Add Electra model integration (@hSterz via #583, based on work of @amitkumarj441 and @pauli31 in #400)
Add adapter output & parameter averaging (@calpt)
Add Prompt Tuning (@lenglaender and @calpt via #595)
Add Composition Support to LoRA and (IA)³ (@calpt via #598)

Breaking Changes

Renamed bottleneck adapter configs and config strings. The new names can be found here: https://docs.adapterhub.ml/overview.html (@calpt)
Removed the XModelWithHeads classes (@lenglaender) (XModelWithHeads have been deprecated since adapter-transformers version 3.0.0)

Changes Due to the Refactoring

Refactored the implementation of all already supported models (@calpt, @lenglaender, @hSterz, @TimoImhof)
Separate the model config (PretrainedConfig) from the adapters config (ModelAdaptersConfig) (@calpt)
Updated the whole documentation, Jupyter Notebooks and example scripts (@hSterz, @lenglaender, @TimoImhof, @calpt)
Introduced the load_model function to load models containing adapters. This replaces the Hugging Face from_pretrained function used in the adapter-transformers library (@lenglaender)
Sharing more logic for adapter composition between different composition blocks (@calpt via #591)
Added Backwards Compatibility Tests which allow for testing if adaptations of the codebase, such as Refactoring, impair the functionality of the library (@TimoImhof via #596)
Refactored the EncoderDecoderModel by introducing a new mixin (ModelUsingSubmodelsAdaptersMixin) for models that contain other models (@lenglaender)
Rename the class AdapterConfigBase into AdapterConfig (@hSterz via #603)

Fixes and Minor Improvements

Fixed EncoderDecoderModel generate function (@lenglaender)
Fixed deletion of invertible adapters (@TimoImhof)
Automatically convert heads when loading with XAdapterModel (@calpt via #594)
Fix training T5 adapter models with Trainer (@calpt via #599)
Ensure output embeddings are frozen during adapter training (@calpt #537)

- Python
Published by lenglaender over 2 years ago

adapters - v.3.2.1

Based on transformers v4.26.1

Fixed

Fix compacter init weights (@hSterz via #516)
Restore compatibility of GPT-2 weight initialization with Transformers (@calpt via #525)
Restore Python 3.7 compatibility (@lenglaender via #510)
Fix LoRA & (IA)³ implementation for Bart & MBart (@calpt via #518)
Fix resume_from_checkpoint in AdapterTrainer class (@hSterz via #514)

- Python
Published by hSterz almost 3 years ago

adapters - v3.2.0

Based on transformers v4.26.1

New

New model integrations

Add BEiT integration (@jannik-brinkmann via #428, #439)
Add GPT-J integration (@ChiragBSavani via #426)
Add CLIP integration (@calpt via #483)
Add ALBERT integration (@lenglaender via #488)
Add BertGeneration (@hSterz via #480)

Misc

Add support for adapter configuration strings (@calpt via #465, #486) This enables you to easily configure adapter configs. To create a Pfeiffer adapter with reduction factor 16 you can know use pfeiffer[reduction_factor=16]. Especially for experiments using different hyperparameters or the example scripts, this can come in handy. Learn more
Add for Stack, Parallel & BatchSplit composition to prefix tuning (@calpt via #476) In previous adapter-transformers versions, you could combine multiple bottleneck adapters. You could use them in parallel or stack them. Now, this is also possible for prefix-tuning adapters. Add multiple prefixes to the same model to combine the functionality of multiple adapters (Stack) or perform several tasks simultaneously (Parallel, BatchSplit) Learn more
Enable parallel sequence generation with adapters (@calpt via #436)

Changed

Removal of the MultiLingAdapterArguments class. Use the AdapterArguments class and setup_adapter_training method instead. Learn more.
Upgrade of underlying transformers version to 4.26.1 (@calpt via #455, @hSterz via #503)

Fixed

Fixes for GLUE & dependency parsing example script (@calpt via #430, #454)
Fix access to shared parameters of compacter (e.g. during sequence generation) (@calpt via #440)
Fix reference to adapter configs in T5EncoderModel (@calpt via #437)
Fix DeBERTa prefix tuning with enabled relative attention (@calpt via #451)
Fix gating for prefix tuning layers (@calpt via #471)
Fix input to T5 adapter layers (@calpt via #479)
Fix AdapterTrainer hyperparameter tuning (@dtuit via #482)
Move loading best adapter to AdapterTrainer class (@MaBeHen via #487)
Make HuggingFace Hub Mixin work with newer utilities (@Helw150 via #473)
Only compute fusion reg loss if fusion layer is trained (@calpt via #505)

- Python
Published by calpt almost 3 years ago

adapters - v3.1.0

Based on transformers v4.21.3

New

New adapter methods

Add LoRA implementation (@calpt via #334, #399): Documentation
Add (IA)^3 implementation (@calpt via #396): Documentation
Add UniPELT implementation (@calpt via #407): Documentation

New model integrations

Add Deberta and DebertaV2 integration(@hSterz via #340)
Add Vision Transformer integration (@calpt via #363)

Misc

Add adapter_summary() method (@calpt via #371): More info
Return AdapterFusion attentions using output_adapter_fusion_attentions argument (@calpt via #417): Documentation

Changed

Upgrade of underlying transformers version (@calpt via #344, #368, #404)

Fixed

Infer label names for training for flex head models (@calpt via #367)
Ensure root dir exists when saving all adapters/heads/fusions (@calpt via #375)
Avoid attempting to set prediction head if non-existent (@calpt via #377)
Fix T5EncoderModel adapter integration (@calpt via #376)
Fix loading adapters together with full model (@calpt via #378)
Multi-gpu support for prefix-tuning (@alexanderhanboli via #359)
Fix issues with embedding training (@calpt via #386)
Fix initialization of added embeddings (@calpt via #402)
Fix model serialization using torch.save() & torch.load() (@calpt via #406)

- Python
Published by calpt over 3 years ago

adapters - v3.0.1

Based on transformers v4.17.0

New

Support float reduction factors in bottleneck adapter configs (@calpt via #339)

Fixed

[AdapterTrainer] add missing preprocess_logits_for_metrics argument (@stefan-it via #317)
Fix savealladapters such that with_head is not ignored (@hSterz via #325)
Fix inferring batch size for prefix tuning (@calpt via #335)
Fix bug when using compacters with AdapterSetup context (@calpt via #328)
[Trainer] Fix issue with AdapterFusion and load_best_model_at_end (@calpt via #341)
Fix generation with GPT-2, T5 and Prefix Tuning (@calpt via #343)

- Python
Published by calpt almost 4 years ago

adapters - v3.0.0

Based on transformers v4.17.0

New

Efficient Fine-Tuning Methods

Add Prefix Tuning (@calpt via #292)
Add Parallel adapters & Mix-and-Match adapter (@calpt via #292)
Add Compacter (@hSterz via #297)

Misc

Introduce XAdapterModel classes as central & recommended model classes (@calpt via #289)
Introduce ConfigUnion class for flexible combination of adapter configs (@calpt via #292)
Add AdapterSetup context manager to replace adapter_names parameter (@calpt via #257)
Add ForwardContext to wrap model forward pass with adapters (@calpt via #267, #295)
Search all remote sources when passing source=None (new default) to load_adapter() (@calpt via #309)

Changed

Deprecate XModelWithHeads in favor of XAdapterModel (@calpt via #289)
Refactored adapter integration into model classes and model configs (@calpt via #263, #304)
Rename activation functions to match Transformers' names (@hSterz via #298)
Upgrade of underlying transformers version (@calpt via #311)

Fixed

Fix seq2seq generation with flexible heads classes (@calpt via #275, @hSterz via #285)
Parallel composition for XLM-Roberta (@calpt via #305)

- Python
Published by calpt almost 4 years ago

adapters - v2.3.0

Based on transformers v4.12.5

New

Allow adding, loading & training of model embeddings (@hSterz via #245). See https://docs.adapterhub.ml/embeddings.html.

Changed

Unify built-in & custom head implementation (@hSterz via #252)
Upgrade of underlying transformers version (@calpt via #255)

Fixed

Fix documentation and consistency issues for AdapterFusion methods (@calpt via #259)
Fix serialization/ deserialization issues with custom adapter config classes (@calpt via #253)

- Python
Published by calpt about 4 years ago

adapters - v2.2.0

Based on transformers v4.11.3

New

Model support

T5 adapter implementation (@AmirAktify & @hSterz via #182)
EncoderDecoderModel adapter implementation (@calpt via #222)

Prediction heads

AutoModelWithHeads prediction heads for language modeling (@calpt via #210)
AutoModelWithHeads prediction head & training example for dependency parsing (@calpt via #208)

Training

Add a new AdapterTrainer for training adapters (@hSterz via #218, #241 )
Enable training of Parallel block (@hSterz via #226)

Misc

Add getadapterinfo() method (@calpt via #220)
Add set_active argument to add & load adapter/fusion/head methods (@calpt via #214)
Minor improvements for adapter card creation for HF Hub upload (@calpt via #225)

Changed

Upgrade of underlying transformers version (@calpt via #232, #234, #239 )
Allow multiple AdapterFusion configs per model; remove set_adapter_fusion_config() (@calpt via #216)

Fixed

Incorrect referencing between adapter layer and layer norm for DataParallel (@calpt via #228)

- Python
Published by calpt over 4 years ago

adapters - v2.1.0

Based on transformers v4.8.2

New

Integration into HuggingFace's Model Hub

Add support for loading adapters from HuggingFace Model Hub (@calpt via #162)
Add method to push adapters to HuggingFace Model Hub (@calpt via #197)
Learn more

`BatchSplit` adapter composition

BatchSplit composition block for adapters and heads (@hSterz via #177)
Learn more

Various new features

Add automatic conversion of static heads when loaded via XModelWithHeads (@calpt via #181) Learn more
Add list_adapters() method to search for adapters (@calpt via #193) Learn more
Add deleteadapter(), deleteadapterfusion() and deletehead() methods (@calpt via #189)
MAD-X 2.0 WikiAnn NER notebook (@hSterz via #187)
Upgrade of underlying transformers version (@hSterz via #183, @calpt via #194 & #200)

Changed

Deprecate addfusion() and trainfusion() in favor of addadapterfusion() and trainadapterfusion() (@calpt via #190)

Fixed

Suppress no-adapter warning when adapter_names is given (@calpt via #186)
leave_out in load_adapter() when loading language adapters from Hub (@hSterz via #177)

- Python
Published by calpt over 4 years ago

adapters - v2.0.1

Based on transformers v4.5.1

New

Allow different reduction factors for different adapter layers (@hSterz via #161)
Allow dynamic dropping of adapter layers in load_adapter() (@calpt via #172)
Add method get_adapter() to retrieve weights of an adapter (@hSterz via #169)

Changed

Re-add adapter_names argument to model forward() methods (@calpt via #176)

Fixed

Fix resolving of adapter from Hub when multiple options available (@Aaronsom via #164)
Fix & improve adapter saving/ loading using Trainer class (@calpt via #178)

- Python
Published by calpt almost 5 years ago

adapters - v2.0.0

Based on transformers v4.5.1

All major new features & changes are described at https://docs.adapterhub.ml/v2_transition.

all changes merged via #105

Additional changes & Fixes

Support loading adapters with loadbestmodelatend in Trainer (@calpt via #122)
Add setter for active_adapters property (@calpt via #132)
New notebooks for NER, text generation & AdapterDrop (@hSterz via #135)
Enable trainer to load adapters from checkpoints (@hSterz via #138)
Update & clean up example scripts (@hSterz via #154 & @calpt via #141, #155)
Add unfreezeadapters param to trainfusion() (@calpt via #156)
Ensure eval/ train mode is correct for AdapterFusion (@calpt via #157)

- Python
Published by calpt almost 5 years ago

adapters - v1.1.1

Based on transformers v3.5.1

New

Modular & custom prediction heads for flex head models (@hSterz via #88)

Fixed

Fixes for DistilBERT layer norm and AdapterFusion (@calpt via #102)
Fix for reloading full models with AdapterFusion (@calpt via #110)
Fix attention and logits output for flex head models (@calpt via #103 & #111)
Fix loss output of flex model with QA head (@hSterz via #88)

- Python
Published by calpt about 5 years ago

adapters - v1.1.0

Based on transformers v3.5.1

New

New model with adapter support: DistilBERT (@calpt via #67)
Save label->id mapping of the task together with the adapter prediction head (@hSterz via #75)
Automatically set matching label->id mapping together with active prediction head (@hSterz via #81)
Upgraded underlying transformers version (@calpt via #55, #72 and #85)
Colab notebook tutorials showcasing all AdapterHub concepts (@calpt via #89)

Fixed

Support for models with flexible heads in pipelines (@calpt via #80)
Adapt input to models with flexible heads to static prediction heads input (@calpt via #90)

- Python
Published by calpt about 5 years ago

adapters - v1.0.1

Based on transformers v2.11.0

New

Adds squad-style QA prediction head to flex-head models

Bug fixes

Fixes loading and saving of adapter config in model.save_pretrained()
Fixes parsing of adapter names in fusion setup

- Python
Published by calpt over 5 years ago

adapters - v1.0.0: First adapter-transformers release

Based on transformers v2.11.0

our first release 🎉

- Python
Published by calpt over 5 years ago

Recent Releases of adapters

adapters - Adapters v1.2.0

New

Adapter Model Plugin Interface (@calpt via #738; @lenglaender via #797)

Multi-Task composition with MTL-LoRA (@FrLdy via #792)

VeRA - parameter-efficient LoRA variant (@julian-fong via #763)

New Models (via new interface)

More

Changed

adapters - Adapters v1.1.1

Changes

Fixed

adapters - Adapters v1.1.0

New

Add AdapterPlus adapters (@julian-fong via #746, #775):

Easy saving, loading and pushing of full adapter compositions to the Hub (@calpt via #771):

Enabling full gradient checkpointing support with adapters (@lenglaender via #759)

More

Changed

Fixed

adapters - Adapters v1.0.1

New

Changes

Fixed

adapters - Adapters v1.0.0

New Adapter Methods & Model Support

Breaking Changes & Deprecations

Minor Fixes & Changes

adapters - Adapters v0.2.2

New

Changed

Fixed

adapters - Adapters v0.2.1

New

Fixed

adapters - Adapters v0.2.0

New

Changed

Fixed

adapters - Adapters v0.1.2

New

Changed

Fixed

adapters - Adapters v0.1.1

New

Fixed

Docs & Notebooks

adapters - Adapters 0.1.0

New Models & Adapter Methods

Breaking Changes

Changes Due to the Refactoring

Fixes and Minor Improvements

adapters - v.3.2.1

Fixed

adapters - v3.2.0

New

New model integrations

Misc

Changed

Fixed

adapters - v3.1.0

New

New adapter methods

New model integrations

Misc

Changed

Fixed

adapters - v3.0.1

New

Fixed

adapters - v3.0.0

New

Efficient Fine-Tuning Methods

Misc

Changed

Fixed

adapters - v2.3.0

New

Changed

Fixed

`BatchSplit` adapter composition