GeeeekExplorer /nano-vllm

Nano vLLM

★ 16,439⑂ 2,863👁 86⑃ 182● 112▤ 0
PythonMITCreated 2025-06-09
+12
日增长
16,439
stars
2,863
forks
86
watchers
182
prs
112
issues
0
releases

README

Nano-vLLM

A lightweight vLLM implementation built from scratch.

 Key Features

 🚀 Fast offline inference - Comparable inference speeds to vLLM
 📖 Readable codebase - Clean implementation in ~ 1,200 lines of Python code
 ⚡ Optimization Suite - Prefix caching, Tensor Parallelism, Torch compilation, CUDA graph, etc.

 Installation

bash
pip install git+https://github.com/GeeeekExplorer/nano-vllm.git


 Model Download

To download the model weights manually, use the following command:
bash
huggingface-cli download --resume-download Qwen/Qwen3-0.6B \
  --local-dir ~/huggingface/Qwen3-0.6B/ \
  --local-dir-use-symlinks False


 Quick Start

See example.py for usage. The API mirrors vLLM's interface with minor differences in the LLM.generate method:
python
from nanovllm import LLM, SamplingParams
llm = LLM("/YOUR/MODEL/PATH", enforce_eager=True, tensor_parallel_size=1)
sampling_params = SamplingParams(temperature=0.6, max_tokens=256)
prompts = ["Hello, Nano-vLLM."]
outputs = llm.generate(prompts, sampling_params)
outputs[0]["text"]


 Benchmark

See bench.py for benchmark.

Test Configuration:
- Hardware: RTX 4070 Laptop (8GB)
- Model: Qwen3-0.6B
- Total Requests: 256 sequences
- Input Length: Randomly sampled between 100–1024 tokens
- Output Length: Randomly sampled between 100–1024 tokens

Performance Results:
| Inference Engine | Output Tokens | Time (s) | Throughput (tokens/s) |
|----------------|-------------|----------|-----------------------|
| vLLM           | 133,966     | 98.37    | 1361.84               |
| Nano-vLLM      | 133,966     | 93.41    | 1434.13               |


 Star History

Top issues

Snapshot: 2026-11-16T09:23:14Z · Data license Apache-2.0 · JSON