Index of every page on this site: llms.txt
All skills
huggingface avatar

/trl-training

@5291c2b official
by Hugging Facehuggingface/skills11k stars
797
Run it yourself

Post-train LLMs with TRL (Transformers Reinforcement Learning) — SFT, DPO, GRPO, KTO, and reward-model training. Use when writing or debugging training code with the TRL Python API or the trl CLI.

Use this Skill: https://skilld.dev/gh/huggingface/skills/trl-training

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈53 tokens for metadata: the name and description. ≈996 when used: this file.

Description uses 2.6% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈211 characters, or 2.6%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

TRL

Each method pairs a *Trainer class with a *Config dataclass. Configs extend transformers.TrainingArguments, so all of its arguments work in any trainer config.

Trainer Dataset type
SFTTrainer language modeling or prompt-completion
DPOTrainer preference (chosen/rejected pairs)
GRPOTrainer prompt-only + reward function(s)
DistillationTrainer prompt-only + a teacher model (on-policy distillation)
KTOTrainer unpaired preference (per-sample bool label)
RewardTrainer preference (chosen/rejected pairs); trains a scalar reward model, not a policy

Many more trainers (OnlineDPO, ORPO, CPO, GKD, …) live in trl.experimental with unstable APIs: https://huggingface.co/docs/trl/experimental_overview

from datasets import load_dataset
from trl import SFTConfig, SFTTrainer

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-0.5B",  # model ID or a PreTrainedModel instance
    args=SFTConfig(output_dir="Qwen2.5-0.5B-SFT"),
    train_dataset=load_dataset("trl-lib/Capybara", split="train"),
)
trainer.train()

Pass model as a string and route loading kwargs through model_init_kwargs (e.g. {"dtype": "bfloat16", "attn_implementation": "kernels-community/flash-attn2"}) instead of calling from_pretrained yourself. The tokenizer/processor is inferred from the model; pass processing_class only when it differs. For LoRA, pass peft_config=LoraConfig(...).

Dataset formats

Conversational: {"messages": [{"role": ..., "content": ...}]} (language modeling) or {"prompt": [...], "completion": [...]}. The chat template is applied automatically — never apply it yourself. Extra columns are allowed; GRPO forwards them to reward functions. Reference: https://huggingface.co/docs/trl/dataset_formats

SFT: the fields that matter

SFTConfig(
    max_length=1024,        # truncation length; None disables truncation
    packing=True,           # pack sequences into max_length blocks: fewer pad tokens, higher throughput
    padding_free=True,      # flatten batch, no padding; requires FlashAttention; implied by packing
    use_liger_kernel=True,  # fused Liger kernels, reduces peak memory
    assistant_only_loss=True,  # loss only on assistant turns (conversational datasets)
)

GRPO: online RL

def reward_len(completions, **kwargs):
    return [-abs(20 - len(c[0]["content"])) for c in completions]

trainer = GRPOTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    reward_funcs=reward_len,  # or a list; rewards are summed
    args=GRPOConfig(output_dir="Qwen2.5-0.5B-GRPO", max_completion_length=512),
    train_dataset=load_dataset("trl-lib/DeepMath-103K", split="train"),
)

Reward functions are called with keyword arguments prompts, completions, completion_ids, trainer_state, plus every extra dataset column — accept **kwargs for the ones you ignore. Return list[float], one reward per completion. With conversational data, completions is a list of message lists, not strings.

The generation batch is per_device_train_batch_size × num_processes × steps_per_generation (or set generation_batch_size directly) and must be divisible by num_generations (default 8). Generation is the usual bottleneck — enable vLLM with use_vllm=True: vllm_mode="colocate" shares the training GPUs (size with vllm_gpu_memory_utilization); vllm_mode="server" uses a separate trl vllm-serve --model <model_id>.

AsyncGRPOTrainer (trl.experimental.async_grpo) implements the same algorithm with generation decoupled from training: a background worker streams completions from a vLLM server while the training loop consumes them, so the two overlap instead of alternating.

CLI

Flags mirror the config fields: trl sft --model_name_or_path Qwen/Qwen2.5-0.5B --dataset_name trl-lib/Capybara. YAML via --config; distributed presets via --accelerate_config zero3 (Python scripts: accelerate launch train.py).

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts1mo3 checks · Risk SAFE
  • Gen Agent Trust Hub1mo

    This skill provides guidance and CLI commands for the Hugging Face TRL library. It appears safe for use and follows standard machine learning development practices.

  • Socket1mo

    No alerts

  • Snyk1mo

    Risk: LOW · No issues

Signed by skilld at 5291c2b. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 3 days ago.

Activeupdated 2 months ago
Other metadata
metadata
{
  "author": "huggingface",
  "documentation": "https://huggingface.co/docs/trl"
}
  • trl
  • llm-training
  • dpo
  • sft
  • rlhf
  • transformers
  • huggingface
  • fine-tuning
  • reinforcement-learning
  • lora

README badge

README badge for huggingface/skills/trl-training

Trains and fine-tunes transformer language models using TRL with CLI commands for SFT, DPO, GRPO, KTO, RLOO, and reward model training. Integrates with Hugging Face Transformers and Accelerate, supporting LoRA adapters, multi-GPU distributed training, and YAML configuration files.

Generated from the current SKILL.md.

What training methods does this skill support?
SFT (Supervised Fine-Tuning), DPO (Direct Preference Optimization), GRPO (Group Relative Policy Optimization), KTO, RLOO (Reinforce Leave One Out), and Reward Model training via dedicated CLI commands.
Can I use LoRA adapters with TRL training?
Yes. All training methods support LoRA via the `--use_peft` flag with configurable `--lora_r` and `--lora_alpha` parameters to reduce memory and training time.
Does this skill support multi-GPU and distributed training?
Yes. TRL integrates with Accelerate for multi-GPU and multi-node training, including FSDP and DeepSpeed ZeRO configurations via the `--accelerate_config` flag.
Can I use configuration files instead of command-line arguments?
Yes. TRL supports YAML configuration files where you specify all training parameters, and you can override individual values at launch time.

Generated from the current SKILL.md. These answers refresh after source changes.