trl

Train transformer language models with reinforcement learning.

https://github.com/huggingface/trl

Science Score: 44.0%

This score indicates how likely this project is to be science-related based on various indicators:

✓
CITATION.cff file
Found CITATION.cff file
✓
codemeta.json file
Found codemeta.json file
✓
.zenodo.json file
Found .zenodo.json file
○
DOI references
○
Academic publication links
○
Academic email domains
○
Institutional organization owner
○
JOSS paper metadata
○
Scientific vocabulary similarity
Low similarity (13.9%) to scientific vocabulary

Last synced: 11 months ago · JSON representation ·

Repository

Train transformer language models with reinforcement learning.

Basic Info

Host: GitHub
Owner: huggingface
License: apache-2.0
Language: Python
Default Branch: main
Homepage: http://hf.co/docs/trl
Size: 11.8 MB

Statistics

Stars: 15,353
Watchers: 96
Forks: 2,163
Open Issues: 540
Releases: 63

Created over 6 years ago · Last pushed 11 months ago

Metadata Files

Readme Contributing License Code of conduct Citation

TRL - Transformer Reinforcement Learning

A comprehensive library to post-train foundation models

🎉 What's New

✨ OpenAI GPT OSS Support: TRL now fully supports fine-tuning the latest OpenAI GPT OSS models! Check out the:

OpenAI Cookbook

GPT OSS recipes

Our example script

Overview

TRL is a cutting-edge library designed for post-training foundation models using advanced techniques like Supervised Fine-Tuning (SFT), Proximal Policy Optimization (PPO), and Direct Preference Optimization (DPO). Built on top of the 🤗 Transformers ecosystem, TRL supports a variety of model architectures and modalities, and can be scaled-up across various hardware setups.

Highlights

Trainers: Various fine-tuning methods are easily accessible via trainers like SFTTrainer, GRPOTrainer, DPOTrainer, RewardTrainer and more.
Efficient and scalable:
- Leverages 🤗 Accelerate to scale from single GPU to multi-node clusters using methods like DDP and DeepSpeed.
- Full integration with 🤗 PEFT enables training on large models with modest hardware via quantization and LoRA/QLoRA.
- Integrates 🦥 Unsloth for accelerating training using optimized kernels.
Command Line Interface (CLI): A simple interface lets you fine-tune with models without needing to write code.

Installation

Python Package

Install the library using pip:

bash pip install trl

From source

If you want to use the latest features before an official release, you can install TRL from source:

bash pip install git+https://github.com/huggingface/trl.git

Repository

If you want to use the examples you can clone the repository with the following command:

bash git clone https://github.com/huggingface/trl.git

Quick Start

For more flexibility and control over training, TRL provides dedicated trainer classes to post-train language models or PEFT adapters on a custom dataset. Each trainer in TRL is a light wrapper around the 🤗 Transformers trainer and natively supports distributed training methods like DDP, DeepSpeed ZeRO, and FSDP.

`SFTTrainer`

Here is a basic example of how to use the SFTTrainer:

```python from trl import SFTTrainer from datasets import load_dataset

dataset = load_dataset("trl-lib/Capybara", split="train")

trainer = SFTTrainer( model="Qwen/Qwen2.5-0.5B", train_dataset=dataset, ) trainer.train() ```

`GRPOTrainer`

GRPOTrainer implements the Group Relative Policy Optimization (GRPO) algorithm that is more memory-efficient than PPO and was used to train Deepseek AI's R1.

```python from datasets import load_dataset from trl import GRPOTrainer

dataset = load_dataset("trl-lib/tldr", split="train")

Dummy reward function: count the number of unique characters in the completions

def rewardnumunique_chars(completions, **kwargs): return [len(set(c)) for c in completions]

trainer = GRPOTrainer( model="Qwen/Qwen2-0.5B-Instruct", rewardfuncs=rewardnumuniquechars, train_dataset=dataset, ) trainer.train() ```

`DPOTrainer`

DPOTrainer implements the popular Direct Preference Optimization (DPO) algorithm that was used to post-train Llama 3 and many other models. Here is a basic example of how to use the DPOTrainer:

```python from datasets import load_dataset from transformers import AutoModelForCausalLM, AutoTokenizer from trl import DPOConfig, DPOTrainer

model = AutoModelForCausalLM.frompretrained("Qwen/Qwen2.5-0.5B-Instruct") tokenizer = AutoTokenizer.frompretrained("Qwen/Qwen2.5-0.5B-Instruct") dataset = loaddataset("trl-lib/ultrafeedbackbinarized", split="train") trainingargs = DPOConfig(outputdir="Qwen2.5-0.5B-DPO") trainer = DPOTrainer( model=model, args=trainingargs, traindataset=dataset, processing_class=tokenizer ) trainer.train() ```

`RewardTrainer`

Here is a basic example of how to use the RewardTrainer:

```python from trl import RewardConfig, RewardTrainer from datasets import load_dataset from transformers import AutoModelForSequenceClassification, AutoTokenizer

tokenizer = AutoTokenizer.frompretrained("Qwen/Qwen2.5-0.5B-Instruct") model = AutoModelForSequenceClassification.frompretrained( "Qwen/Qwen2.5-0.5B-Instruct", numlabels=1 ) model.config.padtokenid = tokenizer.padtoken_id

dataset = loaddataset("trl-lib/ultrafeedbackbinarized", split="train")

trainingargs = RewardConfig(outputdir="Qwen2.5-0.5B-Reward", perdevicetrainbatchsize=2) trainer = RewardTrainer( args=trainingargs, model=model, processingclass=tokenizer, train_dataset=dataset, ) trainer.train() ```

Command Line Interface (CLI)

You can use the TRL Command Line Interface (CLI) to quickly get started with post-training methods like Supervised Fine-Tuning (SFT) or Direct Preference Optimization (DPO):

SFT:

bash trl sft --model_name_or_path Qwen/Qwen2.5-0.5B \ --dataset_name trl-lib/Capybara \ --output_dir Qwen2.5-0.5B-SFT

DPO:

bash trl dpo --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \ --dataset_name argilla/Capybara-Preferences \ --output_dir Qwen2.5-0.5B-DPO

Read more about CLI in the relevant documentation section or use --help for more details.

Development

If you want to contribute to trl or customize it to your needs make sure to read the contribution guide and make sure you make a dev install:

bash git clone https://github.com/huggingface/trl.git cd trl/ pip install -e .[dev]

Citation

bibtex @misc{vonwerra2022trl, author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallouédec}, title = {TRL: Transformer Reinforcement Learning}, year = {2020}, publisher = {GitHub}, journal = {GitHub repository}, howpublished = {\url{https://github.com/huggingface/trl}} }

License

This repository's source code is available under the Apache-2.0 License.

Owner

Name: Hugging Face
Login: huggingface
Kind: organization
Location: NYC + Paris

Website: https://huggingface.co/
Twitter: huggingface
Repositories: 344
Profile: https://github.com/huggingface

The AI community building the future.

Citation (CITATION.cff)

cff-version: 1.2.0
title: 'TRL: Transformer Reinforcement Learning'
message: >-
  If you use this software, please cite it using the
  metadata from this file.
type: software
authors:
  - given-names: Leandro
    family-names: von Werra
  - given-names: Younes
    family-names: Belkada
  - given-names: Lewis
    family-names: Tunstall
  - given-names: Edward
    family-names: Beeching
  - given-names: Tristan
    family-names: Thrush
  - given-names: Nathan
    family-names: Lambert
  - given-names: Shengyi
    family-names: Huang
  - given-names: Kashif
    family-names: Rasul
  - given-names: Quentin
    family-names: Gallouédec
repository-code: 'https://github.com/huggingface/trl'
abstract: "With trl you can train transformer language models with Proximal Policy Optimization (PPO). The library is built on top of the transformers library by \U0001F917 Hugging Face. Therefore, pre-trained language models can be directly loaded via transformers. At this point, most decoder and encoder-decoder architectures are supported."
keywords:
  - rlhf
  - deep-learning
  - pytorch
  - transformers
license: Apache-2.0
version: "0.22"

Issues and Pull Requests

Last synced: 11 months ago

All Time

Total issues: 1,207
Total pull requests: 1,331
Average time to close issues: 25 days
Average time to close pull requests: 9 days
Total issue authors: 836
Total pull request authors: 404
Average comments per issue: 1.79
Average comments per pull request: 1.94
Merged pull requests: 794
Bot issues: 0
Bot pull requests: 0

Past Year

Issues: 761
Pull requests: 887
Average time to close issues: 7 days
Average time to close pull requests: 6 days
Issue authors: 524
Pull request authors: 270
Average comments per issue: 1.07
Average comments per pull request: 1.82
Merged pull requests: 492
Bot issues: 0
Bot pull requests: 0

View more stats

Top Authors

Issue Authors

qgallouedec (42)
RylanSchaeffer (11)
Tuziking (10)
AIR-hl (9)
AMindToThink (9)
MohamedAliRashad (9)
August-murr (9)
YooSungHyun (8)
lewtun (8)
jiosephlee (6)
yananchen1989 (6)
QiyaoWei (6)
macheng6 (6)
fancyerii (6)
edbeeching (6)

Pull Request Authors

qgallouedec (351)
kashif (85)
younesbelkada (67)
lewtun (54)
vwxyzjn (27)
edbeeching (24)
shirinyamani (24)
LeonEricsson (19)
August-murr (18)
pramodith (17)
sergiopaniego (17)
mnoukhov (13)
winglian (12)
claralp (11)
gaetanlop (8)

Top Labels

Issue Labels

🐛 bug (143) 🏋 GRPO (101) ❓ question (66) 🏋 SFT (46) DPO (32) ✨ enhancement (32) ⚡ PEFT (30) ⚡accelerate (25) 🏋 DPO (24) bug (23) 🏋 PPO (22) ⏳ needs more info (17) 🚀 deepspeed (13) 🏋 Reward (11) 📚 documentation (9) enhancement (9) 🏋 Online DPO (6) 📱 cli (6) 👁️ VLM (5) KTO (5) vlm (5) 🏋 GKD (5) good first issue (3) 🏋 KTO (3) 👶 good first issue (3) 🏋 RLOO (3) 🗃️ data (3) 🦥 unsloth (2) 👥 duplicate (2) help wanted (2)

Pull Request Labels

🐛 bug (11) 😴 stale (8) ✨ enhancement (5) DPO (4) enhancement (3) 📚 documentation (3) 🏋 DPO (3) KTO (2) for patch (1) 👨‍⚖️ judge (1) 🚀 deepspeed (1) documentation (1) vlm (1) bug (1) 🏋 PPO (1) 🧒 good second issue (1) 🗃️ data (1) 🏋 SFT (1) ❓ question (1) 🏋 KTO (1)

Packages

Total packages: 4
Total downloads:
- pypi 1,666,962 last-month
Total docker downloads: 10,027

Total dependent packages: 47
(may contain duplicates)
Total dependent repositories: 52
(may contain duplicates)
Total versions: 147
Total maintainers: 5

pypi.org: trl

Train transformer language models with reinforcement learning.

Homepage: https://github.com/huggingface/trl
Documentation: https://trl.readthedocs.io/
License: apache-2.0
Latest release: 0.22.2
published 11 months ago

Versions: 71
Dependent Packages: 47
Dependent Repositories: 52
Downloads: 1,666,886 Last month
Docker Downloads: 10,027

Rankings

Stargazers count: 0.3%

Dependent packages count: 0.5%

Downloads: 0.8%

Forks count: 1.6%

Average: 1.6%

Dependent repos count: 2.0%

Docker downloads count: 4.3%

Maintainers (3)

lysandre lvwerra lewtun

Last synced: 11 months ago

proxy.golang.org: github.com/huggingface/trl

Documentation: https://pkg.go.dev/github.com/huggingface/trl#section-documentation
License: apache-2.0
Latest release: v0.22.2
published 11 months ago

Versions: 63
Dependent Packages: 0
Dependent Repositories: 0

Rankings

Stargazers count: 0.9%

Forks count: 1.1%

Average: 5.3%

Dependent packages count: 8.4%

Dependent repos count: 10.6%

Last synced: 11 months ago

pypi.org: mod-trl

Train transformer language models with reinforcement learning.

Homepage: https://github.com/huggingface/trl
Documentation: https://mod-trl.readthedocs.io/
License: apache-2.0
Latest release: 0.20.0.dev0
published about 1 year ago

Versions: 1
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 18 Last month

Rankings

Dependent packages count: 8.9%

Average: 29.4%

Dependent repos count: 50.0%

Maintainers (1)

stupidfish001

Last synced: 11 months ago

pypi.org: trl-fpo

Train transformer language models with reinforcement learning.

Homepage: https://github.com/huggingface/trl
Documentation: https://trl-fpo.readthedocs.io/
License: Apache 2.0
Latest release: 0.0.15
published over 1 year ago

Versions: 12
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 58 Last month

Rankings

Dependent packages count: 9.9%

Average: 32.8%

Dependent repos count: 55.7%

Maintainers (1)

rajarshi321456

Last synced: 11 months ago

Dependencies

requirements.txt pypi

datasets ==1.17.0
jupyterlab ==2.2.10
matplotlib ==3.5.1
nbdev ==0.2.16
pandas ==1.3.5
torch >=1.4.0
tqdm *
transformers ==4.15.0
wandb ==0.10.20

.github/workflows/build_documentation.yml actions

.github/workflows/build_pr_documentation.yml actions

.github/workflows/clear_cache.yml actions

actions/checkout v3 composite

.github/workflows/stale.yml actions

actions/checkout v3 composite
actions/setup-python v4 composite

.github/workflows/tests.yml actions

actions/checkout v2 composite
actions/checkout v3 composite
actions/setup-python v2 composite
actions/setup-python v4 composite
pre-commit/action v2.0.3 composite

.github/workflows/upload_pr_documentation.yml actions

examples/research_projects/stack_llama_2/scripts/requirements.txt pypi

accelerate *
bitsandbytes *
datasets *
peft *
transformers *
trl *
wandb *

setup.py pypi

.github/workflows/benchmark.yml actions

actions/checkout v3 composite
actions/github-script v6 composite
actions/setup-node v3 composite
actions/setup-python v4 composite
myrotvorets/set-commit-status-action master composite
xt0rted/pull-request-comment-branch v1 composite

pyproject.toml pypi

trl

Science Score: 44.0%

Repository

Basic Info

Statistics

Metadata Files

README.md

TRL - Transformer Reinforcement Learning

A comprehensive library to post-train foundation models

🎉 What's New

Overview

Highlights

Installation

Python Package

From source

Repository

Quick Start

SFTTrainer

GRPOTrainer

Dummy reward function: count the number of unique characters in the completions

DPOTrainer

RewardTrainer

Command Line Interface (CLI)

Development

Citation

License

Owner

Citation (CITATION.cff)

Issues and Pull Requests

All Time

Past Year

Top Authors

Issue Authors

Pull Request Authors

Top Labels

Issue Labels

Pull Request Labels

Packages

pypi.org: trl

Rankings

Maintainers (3)

proxy.golang.org: github.com/huggingface/trl

Rankings

pypi.org: mod-trl

Rankings

Maintainers (1)

pypi.org: trl-fpo

Rankings

Maintainers (1)

Dependencies

`SFTTrainer`

`GRPOTrainer`

`DPOTrainer`

`RewardTrainer`