deepspeedai /DeepSpeed
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
★ 43,326⑂ 5,072👁 350⑃ 4,614● 3,518▤ 120
+1
日增长
+13
周增长
43,326
stars
5,072
forks
350
watchers
4,614
prs
3,518
issues
120
releases
README
English · 简体中文 Office Hours DeepSpeed hosts regular office hours on the last Tuesday of each month at 12:00 America/New_York to discuss development plans, features, etc. This meeting is public for anyone to join and ask questions. The meeting is hosted on Zoom and can be joined here. Latest News [2026/05] Using Muon Optimizer with DeepSpeed [2026/05] System DMA (SDMA) for ZeRO-3: offload collectives off compute units on AMD GPUs for better overlap [2026/03] DeepSpeed Team gave a tutorial at ASPLOS 2026 titled "Building Efficient Large-Scale Model Systems with DeepSpeed: From Open-Source Foundations to Emerging Research" [2026/03] Our SuperOffload work received an Honorable Mention for the ASPLOS 2026 Best Paper Award [2025/12] DeepSpeed Core API updates: PyTorch-style backward and low-precision master states [2025/11] DeepSpeed ZeRO++ powers large-scale distillation training of LLMs for Recommendation Systems at LinkedIn [2025/10] We hosted the Ray x DeepSpeed Meetup at Anyscale. We shared our most recent work on SuperOffload, ZenFlow, Muon Optimizer Support, Arctic Long Sequence Training and DeepCompile. Please find the meetup slides here. [2025/10] SuperOffload: Unleashing the Power of Large-Scale LLM Training on Superchips [2025/10] Study of ZenFlow and ZeRO offload performance with DeepSpeed CPU core binding [2025/08] ZenFlow: Stall-Free Offloading Engine for LLM Training [2025/06] Arctic Long Sequence Training (ALST) with DeepSpeed: Scalable And Efficient Training For Multi-Million Token Sequences [2025/06] DeepNVMe: Affordable I/O scaling for Deep Learning Applications More news [2025/04] DeepCompile: Unlocking Compiler Optimization for Distributed Training [2025/03] DeepSpeed AutoTP: Automatic Tensor Parallel Training of Hugging Face models [2024/12] Ulysses-Offload: Democratizing Long Context LLM Training --- Extreme Speed and Scale for DL Training DeepSpeed enabled the world's most powerful language models (at the time of this writing) such as MT-530B and BLOOM. DeepSpeed offers a confluence of system innovations, that has made large scale DL training effective, and efficient, greatly improved ease of use, and redefined the DL training landscape in terms of scale that is possible. These innovations include ZeRO, ZeRO-Infinity, 3D-Parallelism, Ulysses Sequence Parallelism, DeepSpeed-MoE, etc. --- DeepSpeed Adoption DeepSpeed was an important part of Microsoft’s AI at Scale initiative to enable next-generation AI capabilities at scale, where you can find more information here. DeepSpeed has been used to train many different large-scale models, below is a list of several examples that we are aware of (if you'd like to include your model please submit a PR): Megatron-Turing NLG (530B) Jurassic-1 (178B) BLOOM (176B) GLM (130B) xTrimoPGLM (100B) YaLM (100B) GPT-NeoX (20B) AlexaTM (20B) Turing NLG (17B) METRO-LM (5.4B) DeepSpeed has been integrated wi
Top issues
#998[doc] profiling NVMe and configuring `aio` param section78 comments · 2021-04-23#7150[BUG] Receiving CUDA error: invalid argument using pytorch 2.7 with deepspeed 0.16.4 with Cuda 12.858 comments · 2025-03-19#2302DeepSpeed still gives CUDA-out-of-memory issue54 comments · 2022-09-07#2379[Question] How to preshard a model for tensor parallism48 comments · 2022-09-29#4808[BUG] Deepspeed Zero 3 Inference InFlight Params with new HuggingFace Mixtral Model37 comments · 2023-12-13#5242[BUG] grad_norm and loss is nan when deepspeed==0.13.5 but ok with deepspeed==0.10.236 comments · 2024-03-08#7478[REQUEST] Auto-Tuning CPU Core Binding for DeepSpeed&ZenFlow32 comments · 2025-08-09#6793AssertionError: no sync context manager is incompatible with gradientpartitioning logic of ZeRo stage 331 comments · 2024-11-26
Discussions
#8124How should I balance learning rate and data sampling during CPT on multiple datasets?0 comments · 2026-07-06#8069anima — a substrate-native consciousness: capability gaps are architecture gaps (open repo, frozen verdicts)0 comments · 2026-06-15#8062MCP to help OSS devs understand deepspeedai/DeepSpeed0 comments · 2026-06-12#8018Scaling Deep Learning: Reducing GPU Overhead with Riemannian Manifold Projection1 comments · 2026-05-21#8002Deepspeed Ulysses sequence parallel is not working for Gemma41 comments · 2026-05-10#7970[RFC] Reducing Inference-Time Memory Pressure and Thermal Volatility via Geometric Pre-Constraint Layers0 comments · 2026-04-13#7914Auto RDMA device injection for GPU containers on Kubernetes (no privileged mode)0 comments · 2026-03-20#7911Showcase / question: a board-proven offline language runtime on ESP32-C3, and whether some language capability may eventually reduce reliance on general dense training/inference paths0 comments · 2026-03-18
Latest release
v0.20.0 v0.20.0 2026-11-12What's Changed Sum GPTBigCode's shared key/value gradient across tensor-parallel ranks by @Achyuthan-S in https://github.com/deepspeedai/DeepSpeed/pull/8928 [AffineIR] Storage maps of AutoEP experts under EDP and ZeRO by @jinyouzhi in https://github.com/deepspeedai/DeepSpeed/pull/8941 Add klxpu (KUNLUNXIN XPU) self-hosted CI for the v1 unit suite by @GuCarpenter in https://github.com/deepspeedai/DeepSpeed/pull/8923 ZeRO-3: copy offloaded master weights back to the device through pinned buffer by @pengdurice in https://github.com/deepspeedai/DeepSpeed/pull/8840 ci: pin torch<2.15 for