kvcache-ai /ktransformers

A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations

★ 19,675⑂ 1,602👁 120⑃ 813● 1,317▤ 34
PythonApache-2.0Created 2024-07-26kvcache-ai.github.io/ktransformers/
下载:KTransformers v0.7.2KTransformers-0.7.2 ↓0.7.22026-10-15
+4
日增长
+17
周增长
+63
月增长
19,675
stars
1,602
forks
120
watchers
813
prs
1,317
issues
34
releases

README

A Flexible Framework for Experiencing Cutting-edge LLM Inference/Fine-tune Optimizations
  🎯 Overview | 🚀 Inference | 🎓 SFT | 🔥 Citation | 🚀 Roadmap(2026Q2)  


 🎯 Overview

KTransformers is a research project focused on efficient inference and fine-tuning of large language models through CPU-GPU heterogeneous computing. The project now exposes two user-facing capabilities from the kt-kernel source tree: Inference and SFT.

 🔥 Updates

 Oct 15, 2026: DeepSeek-V4-Flash LoRA fine-tuning supports native MXFP4 routed experts. quickstart.
 Oct 8, 2026: We release the author preprint of KTransformers-FineTune (KT-FT), presenting layout-aware and schedule-optimized CPU-GPU co-computation for MoE fine-tuning. Paper and citation · PDF.
 Sep 13, 2026: Kimi K2.5 / K2.6 LoRA fine-tuning is supported with native RAWINT4 routed experts.
 Aug 26, 2026: Added native support for GLM-5.3-flash, bringing 1M-token context and multimodal input to consumer GPUs. (Tutorial)
 Aug 25, 2026: Uploaded a new easy-to-use KTransformers × LlamaFactory MoE Fine-Tuning Cookbook, covering hardware checks, installation, BF16/FP8/INT8 recipes, LoRA and full fine-tuning, resource planning, and troubleshooting.
 Aug 17, 2026: LoRA fine-tuning now supports compatible AVX512 x86 CPUs, including AMD servers, without requiring AMX. (v0.7.0 Release Notes)
 Aug 16, 2026: DeepSeek-V4-Flash on a single Ascend NPU with CPU expert offload. (Tutorial)
 Aug 5, 2026: Introduced native block-FP8 LoRA fine-tuning (PR 2141)
 July 23, 2026: Added end-to-end BF16 full-parameter fine-tuning for MoE models, including complete checkpoint saving. (PR 2094)
 June 21, 2026: MiniMax-M3 Day0 Support! (Tutorial)
 June 17, 2026: GLM-5.2 Day0 Support! (Tutorial)
 May 6, 2026: KTransformers at GOSIM Paris 2026 — "Agentic AI on Edge" track. We'll present KT's inference performance on consumer hardware.
 May 02, 2026: DeepSeek-V4-Flash Support! (Tutorial)
 Apr 30, 2026: KTransformers v0.6.1 refreshes kt-kernel inference and SFT docs with separate Inference and SFT Quick Start entry points.
 Mar 26, 2026: Support AVX2-only CPU backend for KT-Kernel inference. (Tutorial)
 Feb 13, 2026: MiniMax-M2.5 Day0 Support! (Tutorial)
 Feb 12, 2026: GLM-5 Day0 Support! (Tutorial)
 Jan 27, 2026: Kimi-K2.5 Day0 Support! (Tutorial) (SFT Tutorial)
 Jan 22, 2026: Support CPU-GPU Expert Scheduling, Native BF16 and FP8 per channel Precision and AutoDL unified fine-tuning and inference
 Dec 24, 2025: Support Native MiniMax-M2.1 inference. (Tutorial)
 Dec 22, 2025: Support RL-DPO fine-tuning with LLaMA-Factory. (Tutorial)
 Dec 5, 2025: Support Native Kimi-K2-Thinking inference (Tutorial)
 Nov 6, 2025: Support Kimi-K2-Thinking inference (Tutorial) and fine-tune (Tutorial)
 Nov 4, 2025: KTransformers Fine-Tuning × LLaMA-Factory Integration. (Tutorial)
 Oct 27, 2025: Support Ascend NPU. (Tutorial)
 Oct 10, 2025: Integrating into SGLang. (Roadmap, Blog)
 Sept 11, 2025: Support Qwen3-Next. (Tutorial)
 Sept 05, 2025: Support Kimi-K2-0905. (Tut

Top issues

Latest release

KTransformers v0.7.2 v0.7.2 2026-10-15
KTransformers v0.7.2: DeepSeek-V4-Flash LoRA Fine-Tuning

KTransformers v0.7.2 adds LoRA fine-tuning for the April DeepSeek-V4-Flash checkpoint. Routed experts remain in native MXFP4 on CPU, while attention and other non-expert modules run on GPU.

 🚀 Highlights

- Native MXFP4 routed-expert LoRA forward and backward, without expanding the frozen expert weights to BF16.
- LLaMA-Factory SFT integration with checkpoint saving and SGLang adapter inference.
- Python 3.11 and 3.12 Linux x86-64 wheels.

 📦 Installation

For training:

bash
python -m pip install 'ktransformers[sft]==0.

Snapshot: 2026-11-16T09:23:14Z · Data license Apache-2.0 · JSON