skyzh /tiny-llm

learn LLM inference system on Apple Silicon for systems engineers: build a tiny vLLM + Qwen

★ 5,742⑂ 532👁 34⑃ 308● 27▤ 0
PythonApache-2.0Created 2025-04-19tiny-llm.dev
+3
日增长
5,742
stars
532
forks
34
watchers
308
prs
27
issues
0
releases

README

tiny-llm



tiny-llm is a hands-on course for systems engineers who want to understand LLM
inference end to end. You can think of it as an LLM-serving counterpart to
CMU's Needle
project: build the path that loads a Qwen3 model, turns tokens into logits, and
generates text.

The course begins with array and matrix operations, then introduces kernels and
serving machinery as the running model needs them. Keeping the implementation
small enough to read end to end makes it possible to connect the equations to
memory traffic, kernel occupancy, KV-cache growth, batching, and request
scheduling.

The course is built on MLX arrays and the MLX extension runtime, without using
high-level neural-network layers. When a chapter teaches an operator, your
solution implements that operator in Python, C++, or Metal rather than calling
the corresponding optimized MLX operation. MLX remains the correctness oracle
and performance baseline.

 The Learning Path

The course follows a four-week learning path:

- Week 1: From Matmul to Text. Build a Qwen3 model directly from mlx.core
  array operations: attention, RoPE, GQA, RMSNorm, the MLP, sampling, and
  the autoregressive loop.
- Week 2: A Faster Single Request. Day 1 adds kv-cache and
  request-bounded capacity-cache; Day 2 keeps W4 projection weights packed
  through the quantized-matvec checkpoint. Day 3 adds SIMD matrix prefill at
  simd-matmul. Day 4 adds cumulative RMSNorm, RoPE, and SwiGLU checkpoints.
  Day 5 adds tiled-prefill and runs the completed selected model.
- Week 3: Build a Mini vLLM. Introduce continuous
  batching and chunked admission, then make paged KV the canonical serving
  layout. Decode attention and FlashAttention learn to read pages directly so
  the scheduler does not rebuild dense history on every step.
- Week 4: Build a Coding Agent. Start with a bounded, validated agent loop,
  then connect it to a small workspace. The course is publishing one reviewed
  checkpoint at a time; Days 1 through 9 now cover inspection, approved edits,
  one validation command, simple effect receipts, and one visible
  checkpoint-and-resume boundary, receipt-backed context compaction, and one
  visible inspect-and-steer pause, and deterministic evaluation of observable
  outcomes, then tokenizer/KV-prefix reuse for two isolated steered branches
  and one explicit evidence-backed selection, followed by bounded,
  range-retrievable evidence for oversized tool results.

 Why MLX and Qwen3?

Apple silicon provides a practical local environment with one shared memory
space and direct access to Metal kernels. Students can inspect the complete
path on one machine instead of depending on an expensive CUDA GPU setup.

Qwen3-4B is large enough to expose real weight-bandwidth, attention, and cache
costs, but small enough to iterate on locally. Its grouped-query attention,
QK normalization, BF16 activations, and 4-bit weights also keep the exercises
close to current model-serving work.

 Start Here

The book is publi

Top issues

Snapshot: 2026-11-16T09:23:14Z · Data license Apache-2.0 · JSON