sgl-project /mini-sglang
A compact implementation of SGLang, designed to demystify the complexities of modern LLM serving systems.
★ 5,582⑂ 987👁 19⑃ 152● 33▤ 0
+2
日增长
5,582
stars
987
forks
19
watchers
152
prs
33
issues
0
releases
README
Mini-SGLang
A lightweight yet high-performance inference framework for Large Language Models.
---
Mini-SGLang is a compact implementation of SGLang, designed to demystify the complexities of modern LLM serving systems. With a compact codebase of ~5,000 lines of Python, it serves as both a capable inference engine and a transparent reference for researchers and developers.
✨ Key Features
- High Performance: Achieves state-of-the-art throughput and latency with advanced optimizations.
- Lightweight & Readable: A clean, modular, and fully type-annotated codebase that is easy to understand and modify.
- Advanced Optimizations:
- Radix Cache: Reuses KV cache for shared prefixes across requests.
- Chunked Prefill: Reduces peak memory usage for long-context serving.
- Overlap Scheduling: Hides CPU scheduling overhead with GPU computation.
- Tensor Parallelism: Scales inference across multiple GPUs.
- Optimized Kernels: Integrates FlashAttention and FlashInfer for maximum efficiency.
- ...
🚀 Quick Start
⚠️ Platform Support: Mini-SGLang currently supports Linux only (x86_64 and aarch64). Windows and macOS are not supported due to dependencies on Linux-specific CUDA kernels (sgl-kernel, flashinfer). We recommend using WSL2 on Windows or Docker for cross-platform compatibility.
1. Environment Setup
We recommend using uv for a fast and reliable installation (note that uv does not conflict with conda).
bash
Create a virtual environment (Python 3.10+ recommended)
uv venv --python=3.12
source .venv/bin/activate
Prerequisites: Mini-SGLang relies on CUDA kernels that are JIT-compiled. Ensure you have the NVIDIA CUDA Toolkit installed and that its version matches your driver's version. You can check your driver's CUDA capability with nvidia-smi.
2. Installation
Install Mini-SGLang directly from the source:
bash
git clone https://github.com/sgl-project/mini-sglang.git
cd mini-sglang && uv venv --python=3.12 && source .venv/bin/activate
uv pip install -e .
💡 Installing on Windows (WSL2)
Since Mini-SGLang requires Linux-specific dependencies, Windows users should use WSL2:
1. Install WSL2 (if not already installed):
powershell
In PowerShell (as Administrator)
wsl --install
2. Install CUDA on WSL2:
- Follow NVIDIA's WSL2 CUDA guide
- Ensure your Windows GPU drivers support WSL2
3. Install Mini-SGLang in WSL2:
bash
Inside WSL2 terminal
git clone https://github.com/sgl-project/mini-sglang.git
cd mini-sglang && uv venv --python=3.12 && source .venv/bin/activate
uv pip install -e .
4. Access from Windows: The server will be accessible at http://localhost:8000 from Windows browsers and applications.
🐳 Running with Docker
Prerequisites:
- Docker
- NVIDIA Container Toolkit
1. Build the Docker image:
bash
docker build -t minisgl .
2. Run the server:
bash
docker run --gpus all -p 1919:1919 \
minisgl --model Qwen/Qwen3-0.6B --host 0.0.0.0
3. Run in interactive sheTop issues
#90Interest in adding AMD GPU (ROCm) support6 comments · 2026-03-01#58crash to run offline inference6 comments · 2025-12-30#128does mini-sglang support SM_753 comments · 2026-05-09#1295070 SM120 Unsupported architecture with `TllmGenFmhaRunner`2 comments · 2026-05-10#60torch.compile compatibility: significant performance overhead due to guard failures and recompilations2 comments · 2026-01-05#28any diffusion model support plan?2 comments · 2025-12-21#12CPU support2 comments · 2025-12-18#185[Bug] Missing SSE event delimiters in /generate1 comments · 2026-11-10