pytorch /torchtitan

A PyTorch native platform for training generative AI models

★ 5,909⑂ 1,080👁 58⑃ 5,018● 896▤ 6
PythonBSD-3-ClauseCreated 2023-12-13
下载:v0.4.0v0.4.0 ↓0.4.02026-10-28
+3
日增长
5,909
stars
1,080
forks
58
watchers
5,018
prs
896
issues
6
releases

README

A PyTorch-native platform for pretraining and post-training generative AI models at scale








Quick start · Models · Features · RL · Docs · Contributing



torchtitan is a clean-room, PyTorch-native implementation of large-scale
training. One readable codebase takes a model from pretraining through
supervised fine-tuning to reinforcement learning, composing FSDP, tensor,
pipeline, context, and expert parallelism without rewriting the model.

- Composable parallelism. Models declare how they shard; FSDP2, TP, PP, CP,
  and EP stack into N-D layouts through DeviceMesh and DTensor.
- Built for research. Minimal, hackable code; every run is a Python
  recipe you can read, copy, and modify.
- Pretraining to RL in one stack. TitanRL runs the
  same model definitions and kernels in the trainer and in vLLM generation, with
  a bitwise-reproducible batch-invariant mode.
- Frontier models and techniques. DeepSeek, Qwen, Kimi, gpt-oss, Llama, and
  FLUX, with MXFP8/NVFP4 low precision, Dist-MoE, CUDA graphs, and more.

 [!NOTE]
 torchtitan is under active development. To use the latest features, use a
 recent PyTorch nightly.

 Latest news

- [2026/10] Zephon, a data loader by DatologyAI, released its TorchTitan integration. Zephon keeps the same training batches in the same order even when your GPU count changes (elastic determinism). Check out the blog post on pytorch.org.
- [2026/08] TitanRL is a hackable RL stack for scaling and debugging. It reuses TorchTitan model definitions and kernels across training and vLLM generation and supports batch-invariant mode.
- [2025/11] AMD released an optimized fork of torchtitan for AMD GPUs.
- [2025/10] We released torchtitan v0.2.0.
- [2025/10] SkyPilot now supports torchtitan! See the tutorial.


Older news

- [2025/07] We published instructions on how to add a model to torchtitan.
- [2025/04] Our paper was accepted by ICLR 2025.
- [2024/12] GPU MODE lecture on torchtitan.
- [2024/07] Presentation at PyTorch Conference 2024.



 Quick start

1. Install with a PyTorch nightly (Python 3.11+). Replace cu132 with your CUDA or ROCm build.

bash
git clone https://github.com/pytorch/torchtitan && cd torchtitan
pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu132
pip install -r requirements.txt


2. Train a debug model. It uses the tokenizer bundled with the tests, so no download is needed.

bash
NGPU=1 ./run_train.sh    Llama 3 debug model, 10 steps


3. Train a real model. Download the tokenizer (Llama weights require
access on Hugging Face), then
select a recipe by module and function name:

bash
python scripts/download_hf_assets.py --repo_id meta-llama/Llama-3.1-8B --assets tokenizer --hf_token=...
MODULE=torchtitan_recipes.models.llama3 CONFIG=llama3_8b ./run_train.sh    8 GPUs


4. Make it yours. A recipe is a Python function returning a complete config,
so changing a run means writing a function, not passing flags:

python
 my_recipes.py
from torchtitan_recipes.models.llama3 import

Top issues

Discussions

Latest release

v0.4.0 v0.4.0 2026-10-28
TorchTitan 0.4.0

TorchTitan 0.4.0 aligns with PyTorch 2.15, expands model coverage, improves large-scale MoE training, and unifies core training and TitanRL on a shared engine.

These highlights cover changes since 0.3.0. For broader updates and benchmarks, see TorchTitan Update: New SOTA Models, Performance at Scale, TitanRL.

 New model implementations

- Muse Spark Open: Adds the 816B-A42B model, including text and vision/audio configurations. Multimodal CUDA graphs remain unsupported. (5420)
- Kimi K3: Hybrid KDA/MLA attention, LatentMoE, and vision support, with tensor/sequence 

Snapshot: 2026-11-16T09:23:14Z · Data license Apache-2.0 · JSON