+12
日增长
16,439
stars
2,863
forks
86
watchers
182
prs
112
issues
0
releases
README
Nano-vLLM
A lightweight vLLM implementation built from scratch.
Key Features
🚀 Fast offline inference - Comparable inference speeds to vLLM
📖 Readable codebase - Clean implementation in ~ 1,200 lines of Python code
⚡ Optimization Suite - Prefix caching, Tensor Parallelism, Torch compilation, CUDA graph, etc.
Installation
bash
pip install git+https://github.com/GeeeekExplorer/nano-vllm.git
Model Download
To download the model weights manually, use the following command:
bash
huggingface-cli download --resume-download Qwen/Qwen3-0.6B \
--local-dir ~/huggingface/Qwen3-0.6B/ \
--local-dir-use-symlinks False
Quick Start
See example.py for usage. The API mirrors vLLM's interface with minor differences in the LLM.generate method:
python
from nanovllm import LLM, SamplingParams
llm = LLM("/YOUR/MODEL/PATH", enforce_eager=True, tensor_parallel_size=1)
sampling_params = SamplingParams(temperature=0.6, max_tokens=256)
prompts = ["Hello, Nano-vLLM."]
outputs = llm.generate(prompts, sampling_params)
outputs[0]["text"]
Benchmark
See bench.py for benchmark.
Test Configuration:
- Hardware: RTX 4070 Laptop (8GB)
- Model: Qwen3-0.6B
- Total Requests: 256 sequences
- Input Length: Randomly sampled between 100–1024 tokens
- Output Length: Randomly sampled between 100–1024 tokens
Performance Results:
| Inference Engine | Output Tokens | Time (s) | Throughput (tokens/s) |
|----------------|-------------|----------|-----------------------|
| vLLM | 133,966 | 98.37 | 1361.84 |
| Nano-vLLM | 133,966 | 93.41 | 1434.13 |
Star HistoryTop issues
#167rope argument inconsistent9 comments · 2026-01-28#25How did this job achieve such excellent performance?8 comments · 2025-06-18#220[Feature Request] Support weightless RMSNorm (for FlashNorm weight folding trick)5 comments · 2026-04-29#73Support for Chunked Prefill of new seqs while decoding old seqs5 comments · 2025-07-08#48Implement support for Qwen2.5vl5 comments · 2025-06-24#240[Bug] may_append allocates new block one token too late, causing KV cache write to unallocated block2 comments · 2026-06-08#189Qwen models initialization fails with TypeError: unhashable type: 'dict' and AssertionError due to rope_scaling2 comments · 2026-03-23#165🚀 Introducing Nano-vLLM-v1 – A Lightweight, Re-engineered Nano vLLM Implementation with vLLM v1 Scheduler and Chunked Prefill2 comments · 2026-01-25