+3
日增长
5,742
stars
532
forks
34
watchers
308
prs
27
issues
0
releases
README
tiny-llm tiny-llm is a hands-on course for systems engineers who want to understand LLM inference end to end. You can think of it as an LLM-serving counterpart to CMU's Needle project: build the path that loads a Qwen3 model, turns tokens into logits, and generates text. The course begins with array and matrix operations, then introduces kernels and serving machinery as the running model needs them. Keeping the implementation small enough to read end to end makes it possible to connect the equations to memory traffic, kernel occupancy, KV-cache growth, batching, and request scheduling. The course is built on MLX arrays and the MLX extension runtime, without using high-level neural-network layers. When a chapter teaches an operator, your solution implements that operator in Python, C++, or Metal rather than calling the corresponding optimized MLX operation. MLX remains the correctness oracle and performance baseline. The Learning Path The course follows a four-week learning path: - Week 1: From Matmul to Text. Build a Qwen3 model directly from mlx.core array operations: attention, RoPE, GQA, RMSNorm, the MLP, sampling, and the autoregressive loop. - Week 2: A Faster Single Request. Day 1 adds kv-cache and request-bounded capacity-cache; Day 2 keeps W4 projection weights packed through the quantized-matvec checkpoint. Day 3 adds SIMD matrix prefill at simd-matmul. Day 4 adds cumulative RMSNorm, RoPE, and SwiGLU checkpoints. Day 5 adds tiled-prefill and runs the completed selected model. - Week 3: Build a Mini vLLM. Introduce continuous batching and chunked admission, then make paged KV the canonical serving layout. Decode attention and FlashAttention learn to read pages directly so the scheduler does not rebuild dense history on every step. - Week 4: Build a Coding Agent. Start with a bounded, validated agent loop, then connect it to a small workspace. The course is publishing one reviewed checkpoint at a time; Days 1 through 9 now cover inspection, approved edits, one validation command, simple effect receipts, and one visible checkpoint-and-resume boundary, receipt-backed context compaction, and one visible inspect-and-steer pause, and deterministic evaluation of observable outcomes, then tokenizer/KV-prefix reuse for two isolated steered branches and one explicit evidence-backed selection, followed by bounded, range-retrievable evidence for oversized tool results. Why MLX and Qwen3? Apple silicon provides a practical local environment with one shared memory space and direct access to Metal kernels. Students can inspect the complete path on one machine instead of depending on an expensive CUDA GPU setup. Qwen3-4B is large enough to expose real weight-bandwidth, attention, and cache costs, but small enough to iterate on locally. Its grouped-query attention, QK normalization, BF16 activations, and 4-bit weights also keep the exercises close to current model-serving work. Start Here The book is publi
Top issues
#30Is there any timeline for week2+ tasks?5 comments · 2025-06-24#322Week 1 Day 1 reference attention tests fail on float32 GPU with MLX’s default precision2 comments · 2026-09-25#82Is there any timeline for week3?2 comments · 2025-11-30#116A simple CUDA adaptation of the tutorial0 comments · 2026-05-08#84Support for MLA (Multi-Head Latent Attention) Task0 comments · 2026-01-22#45precision issues in M10 comments · 2025-08-17#37feat: Distributed inference (TP/PP)0 comments · 2025-08-04