+0
日增长
5,817
stars
764
forks
40
watchers
1,868
prs
1,102
issues
10
releases
README
⚠️ Torchtune is no longer actively maintained: torchtune development wound down in 2025 — see The future of torchtune. Huge thanks to the 150+ contributors who made this library what it was. torchtune !Integration Tests Overview | Installation | Get Started | Documentation | Community | Citing torchtune | License 📣 Recent updates 📣 May 2025: torchtune has added support for Qwen3 models! Check out all the configs here April 2025: Llama4 is now available in torchtune! Try out our full and LoRA finetuning configs here February 2025: Multi-node training is officially open for business in torchtune! Full finetune on multiple nodes to take advantage of larger batch sizes and models. December 2024: torchtune now supports Llama 3.3 70B! Try it out by following our installation instructions here, then run any of the configs here. November 2024: torchtune has released v0.4.0 which includes stable support for exciting features like activation offloading and multimodal QLoRA November 2024: torchtune has added Gemma2 to its models! October 2024: torchtune added support for Qwen2.5 models - find the configs here September 2024: torchtune has support for Llama 3.2 11B Vision, Llama 3.2 3B, and Llama 3.2 1B models! Try them out by following our installation instructions here, then run any of the text configs here or vision configs here. Overview 📚 torchtune is a PyTorch library for easily authoring, post-training, and experimenting with LLMs. It provides: - Hackable training recipes for SFT, knowledge distillation, DPO, PPO, GRPO, and quantization-aware training - Simple PyTorch implementations of popular LLMs like Llama, Gemma, Mistral, Phi, Qwen, and more - Best-in-class memory efficiency, performance improvements, and scaling, utilizing the latest PyTorch APIs - YAML configs for easily configuring training, evaluation, quantization or inference recipes Post-training recipes torchtune supports the entire post-training lifecycle. A successful post-trained model will likely utilize several of the below methods. Supervised Finetuning (SFT) | Type of Weight Update | 1 Device | 1 Device | 1 Node | |-----------------------|:--------:|:---------:|:-------:| | Full | ✅ | ✅ | ✅ | | LoRA/QLoRA | ✅ | ✅ | ✅ | Example: tune run lora_finetune_single_device --config llama3_2/3B_lora_single_device You can also run e.g. tune ls lora_finetune_single_device for a full list of available configs. Knowledge Distillation (KD) | Type of Weight Update | 1 Device | 1 Device | 1 Node | |-----------------------|:--------:|:---------:|:-------:| | Full | ❌ | ❌ | ❌ | | LoRA/QLoRA | ✅ | ✅ | ❌ | Example: tune run knowledge_distillation_distributed --config qwen2/1.5B_to_0.5B_KD_lora_distributed You can also run e.g. tune ls knowledge_distillation_distributed for a full list of available configs. Reinforcement
Top issues
#2237Finetune meta-llama/Llama-Guard-3-1B32 comments · 2025-01-08#2093Multi GPU timeout on save checkpoint (WorkNCCL, Watchdog, timeout)28 comments · 2024-11-29#2402Does FSDP v2 have the best performance?27 comments · 2025-02-17#2180Add multiprocess dataset packing23 comments · 2024-12-19#1244[RFC] Long context fine tuning in torchtune22 comments · 2024-07-30#791Vision/Multimodal22 comments · 2024-04-18#1965OOM part way through epoch using llama3.2 vision finetuning distributed recipe21 comments · 2024-11-07#1249Does torchtune support for traditional text-generation tasks?21 comments · 2024-08-01
Discussions
#2999EvalPort: optional export for EleutherEvalRecipe results0 comments · 2026-09-27#2933Question on KVCache1 comments · 2025-10-15#2919Graph Neural Network Library for LibTorch - Seeking Community Input0 comments · 2025-08-27#2882How do you make torchtune deterministic?1 comments · 2025-07-15#2878LLaMa 3.2 vision - Clarification on User Input Masking During Training1 comments · 2025-07-12#2872How do you specify to use ZeRO-2, DDP, or ZeRO-3 (FSDP) in torchtune?1 comments · 2025-07-08#2870How do you do LoRA on torchtune with only lora applied to MLP?2 comments · 2025-07-07#2773[bike shed] `tune` binary name1 comments · 2025-05-30
Latest release
v0.6.1 v0.6.1 2025-04-07v0.6.1 Patch for 2561. Full Changelog: https://github.com/pytorch/torchtune/compare/v0.6.0...v0.6.1