huggingface /trl
Train transformer language models with reinforcement learning.
★ 19,741⑂ 3,148👁 104⑃ 5,283● 2,914▤ 104
+6
日增长
19,741
stars
3,148
forks
104
watchers
5,283
prs
2,914
issues
104
releases
README
TRL - Transformers Reinforcement Learning
A comprehensive library to post-train foundation models
🎉 What's New
⚡ AsyncGRPOTrainer is now stable: AsyncGRPOTrainer graduates to the stable API. It runs GRPO much faster by decoupling generation from training: a vLLM server streams rollouts while the trainer updates the model.
Overview
TRL is a cutting-edge library designed for post-training foundation models using advanced techniques like Supervised Fine-Tuning (SFT), Group Relative Policy Optimization (GRPO), and Direct Preference Optimization (DPO). Built on top of the 🤗 Transformers ecosystem, TRL supports a variety of model architectures and modalities, and can be scaled-up across various hardware setups.
Highlights
- Trainers: Various fine-tuning methods are easily accessible via trainers like SFTTrainer, GRPOTrainer, AsyncGRPOTrainer, DPOTrainer, KTOTrainer and more.
- Efficient and scalable:
- Leverages 🤗 Accelerate to scale from single GPU to multi-node clusters using methods like DDP and DeepSpeed.
- Full integration with 🤗 PEFT enables training on large models with modest hardware via quantization and LoRA/QLoRA.
- Integrates 🦥 Unsloth for accelerating training using optimized kernels.
- Command Line Interface (CLI): A simple interface lets you fine-tune with models without needing to write code.
Installation
Python Package
Install the library using pip:
bash
pip install trl
From source
If you want to use the latest features before an official release, you can install TRL from source:
bash
pip install git+https://github.com/huggingface/trl.git
Repository
If you want to use the examples you can clone the repository with the following command:
bash
git clone https://github.com/huggingface/trl.git
Quick Start
For more flexibility and control over training, TRL provides dedicated trainer classes to post-train language models or PEFT adapters on a custom dataset. Each trainer in TRL is a light wrapper around the 🤗 Transformers trainer and natively supports distributed training methods like DDP, DeepSpeed ZeRO, and FSDP.
SFTTrainer
Here is a basic example of how to use the SFTTrainer:
python
from trl import SFTTrainer
from datasets import load_dataset
dataset = load_dataset("trl-lib/Capybara", split="train")
trainer = SFTTrainer(
model="Qwen/Qwen2.5-0.5B",
train_dataset=dataset,
)
trainer.train()
GRPOTrainer
GRPOTrainer implements the Group Relative Policy Optimization (GRPO) algorithm that is more memory-efficient than PPO and was used to train Deepseek AI's R1. accuracy_reward needs math-verify, which you can install with pip install trl[math_verify].
python
from datasets import load_dataset
from trl import GRPOTrainer
from trl.rewards import accuracy_reward
dataset = load_dataset("trl-lib/DeepMath-103K", split="train")
trainer = GRPOTrainer(
model="Qwen/Qwen2.5-0.5B-Instruct",
reward_funcs=accuracy_reward,
trainTop issues
#7202Audit the trl-internal-testing repos that have no generation script14 comments · 2026-09-14#3399FP8 training14 comments · 2025-05-01#3549Packing sequences for memory efficiency in GRPO and other preference learning implementations12 comments · 2025-06-09#7137Align tiny test model configs with their reference models11 comments · 2026-09-09#6857Tiny test models are incompatible with Flash Attention (head_size=2)10 comments · 2026-08-21#4611SFTTrainer: Logging Experts Statistics9 comments · 2025-12-02#3717Add activation checkpointing to DPO and GRPO9 comments · 2025-07-10#2580GKDTrainer + FSDP results in RuntimeError: Expected all tensors to be on the same device, but found at least two devices9 comments · 2025-01-17
Discussions
#8257loraspectra: Zero-dependency Rust CLI to audit spectral rank, prune noise singular values, and orthogonally merge SFT/DPO/GRPO LoRA .safetensors0 comments · 2026-11-12#7534When should I choose SFT over GRPO for fine-tuning a small language model?1 comments · 2026-10-05#7231LoRA adapters trained on a 4-bit base degrade when served on a BF16 base — 100-sample comparison1 comments · 2026-09-16#7191Taking on ongoing maintenance responsibilities0 comments · 2026-09-12#7082You fine-tune on customer data with SFTTrainer. Can you prove it didn't memorize? memaudit runs that audit inside your training job.0 comments · 2026-09-06#5933Proposal: Add TrOPD-style outlier FKL masking to GOLDTrainer1 comments · 2026-06-03#5509Decouple rollout from trainer process — RolloutService protocol1 comments · 2026-04-10#5432Is this a common/reasonable recipe for full finetuning Qwen3.5-4B?1 comments · 2026-04-01
Latest release
v1.18.0 v1.18.0 2026-11-13Features [!WARNING] The submodules of trl.experimental.async_grpo are gone. from trl.experimental.async_grpo.async_grpo_config import AsyncGRPOConfig, and imports from async_rollout_worker, packing, vllm_client or weight_transfer in that package, no longer work. Import the trainer from trl, the rollout loop from trl.trainer.async_grpo_trainer, and the vLLM client from trl.generation.async_vllm_client (renamed AsyncVLLMClient). from trl.experimental.async_grpo import AsyncGRPOConfig, AsyncGRPOTrainer still works, with a FutureWarning, until v2.0.0. :mortar_board: AsyncGRPOTrainer is n