FlashML-org /FreeToken
FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.
★ 17,219⑂ 1,706👁 104⑃ 491● 418▤ 6
+31
日增长
17,219
stars
1,706
forks
104
watchers
491
prs
418
issues
6
releases
README
| Download | Paper | Developer Slack | Community Discord | Community WeChat |
Unlock datacenter-class intelligence on the hardware you already own — Run 290B+ frontier MoE models locally on your gaming PC at blistering interactive speeds.
About
FreeToken is an edge-native Mixture-of-Experts (MoE) serving engine designed for running frontier-scale open-weight models on personal and consumer hardware. It treats heterogeneous edge resources—GPUs, CPUs, host memory, and interconnects—as a unified, elastic inference platform. Its core features include:
- Fast Edge-Native Runtime: Provides efficient MoE serving with bandwidth-adaptive CPU–GPU co-execution ($q^\star$ policy), full-layer double-buffered prefill streaming, global LRU expert caching, graph-compatible execution, and the FTW fast weight format.
- Semantic-Aware Caching: Features semantic anchor checkpoints for recurrent state and KV caches, allowing agentic context edits (e.g., tool calls, thinking blocks) to avoid redundant context recomputation.
- Elastic Memory Management: Supports dynamic, runtime VRAM re-allocation between expert caches and KV memory without engine restarts or weight reloading.
- Broad MoE & Ecosystem Support: Supports frontier open-weight MoE models (e.g., DeepSeek-V4-Flash, Qwen3.6-35B-A3B, GLM-5.2) across various parameter scales and quantization formats (e.g., MXFP4, NVFP4, FP8, BF16, GGUF), with Anthropic/OpenAI-compatible APIs for seamless integration with real-world coding and tool-calling agents (e.g., Codex, Claude Code, OpenCode, OpenClaw, DeepSeek Harness).
- Diverse Consumer Hardware: Scales across consumer laptops, gaming desktops, and workstation GPUs, with native support for NVIDIA RTX 30, RTX 40, and RTX 50 series GPUs.
Getting Started
Desktop app
Download FreeToken for Windows or Linux at flashml.ai. It sets the engine up for you and gives you a GUI for running models, chatting, and tuning the engine.
CLI
Install FreeToken with uv (recommended) or pip:
bash
uv pip install "freetoken[accel]"
Or build from source:
bash
git clone https://github.com/FlashML-org/FreeToken.git && cd FreeToken
uv venv && source .venv/bin/activate
uv pip install -e ".[accel]"
For More details:
- Install FreeToken
- Quick start
- Supported models
- CLI reference
- Repairing old FTW checkpoints
- Diagnosing a checkpoint that fails to load
Citation
If you use FreeToken for your research, please cite our paper:
bibtex
@article{yang2026freetoken,
title={FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution},
author={Yang, Shuo and Fan, Xiaoze and Pan, Melissa and Xi, Haocheng and Wang, Zhe and Sun, Shanlin and Keutzer, Kurt and Han, Song and Zaharia, Matei and Xu, Chenfeng and Stoica, Ion},
journal={arXiv preprint arXiv:2608.16157},
year={2026}
}
Acknowledgment
FreeToken was deeply inspired by mini-sglang, and
learned the design and reused code from the following projects:
SGLang,
vLLM,
FlashInfer,
flashTop issues
#74[Feature Request]: Support for Intel GPUs (Arc / Core Ultra) via oneAPI / SYCL or OpenVINO backend23 comments · 2026-08-23#60Feature request: AMD GPU support22 comments · 2026-08-22#82ROCm RDNA4 gfx1201 Ported. Windows only for now. Test please19 comments · 2026-08-23#79FreeToken Roadmap (2026)17 comments · 2026-08-23#9Feature Request: macOS (Apple Silicon) Support17 comments · 2026-08-21#214[Feature Request] Possible to support Qwen3.8-Flash-Next?15 comments · 2026-08-26#15Support for dual GPU15 comments · 2026-08-22#314Requesting support for SM 79 comments · 2026-08-31
Latest release
v0.2.2 v0.2.2 2026-11-09Highlights - Speculative decoding. --speculative-algorithm mtp drafts with the checkpoint's own MTP layer, and --speculative-algorithm dflash drafts with a separate DFlash 2 checkpoint (836, 844, 856, 857). - New model. Ripple-65B Preview, from BF16 safetensors, the ModelOpt NVFP4 export or GGUF (871). - Faster CPU experts in hybrid. The NVFP4 W4A8 CPU kernels are faster; rerun ft bench bw so hybrid uses the new CPU rate (851). Speculative Decoding Each decode step drafts k tokens and verifies them on the target in one forward, so one step can commit several tokens. With sampling,