karpathy /build-nanogpt
Video+code lecture on building nanoGPT from scratch
★ 5,644⑂ 907👁 52⑃ 45● 27▤ 0
+1
日增长
5,644
stars
907
forks
52
watchers
45
prs
27
issues
0
releases
README
build nanoGPT This repo holds the from-scratch reproduction of nanoGPT. The git commits were specifically kept step by step and clean so that one can easily walk through the git commit history to see it built slowly. Additionally, there is an accompanying video lecture on YouTube where you can see me introduce each commit and explain the pieces along the way. We basically start from an empty file and work our way to a reproduction of the GPT-2 (124M) model. If you have more patience or money, the code can also reproduce the GPT-3 models. While the GPT-2 (124M) model probably trained for quite some time back in the day (2019, ~5 years ago), today, reproducing it is a matter of ~1hr and ~$10. You'll need a cloud GPU box if you don't have enough, for that I recommend Lambda. Note that GPT-2 and GPT-3 and both simple language models, trained on internet documents, and all they do is "dream" internet documents. So this repo/video this does not cover Chat finetuning, and you can't talk to it like you can talk to ChatGPT. The finetuning process (while quite simple conceptually - SFT is just about swapping out the dataset and continuing the training) comes after this part and will be covered at a later time. For now this is the kind of stuff that the 124M model says if you prompt it with "Hello, I'm a language model," after 10B tokens of training: Hello, I'm a language model, and my goal is to make English as easy and fun as possible for everyone, and to find out the different grammar rules Hello, I'm a language model, so the next time I go, I'll just say, I like this stuff. Hello, I'm a language model, and the question is, what should I do if I want to be a teacher? Hello, I'm a language model, and I'm an English person. In languages, "speak" is really speaking. Because for most people, there's And after 40B tokens of training: Hello, I'm a language model, a model of computer science, and it's a way (in mathematics) to program computer programs to do things like write Hello, I'm a language model, not a human. This means that I believe in my language model, as I have no experience with it yet. Hello, I'm a language model, but I'm talking about data. You've got to create an array of data: you've got to create that. Hello, I'm a language model, and all of this is about modeling and learning Python. I'm very good in syntax, however I struggle with Python due Lol. Anyway, once the video comes out, this will also be a place for FAQ, and a place for fixes and errata, of which I am sure there will be a number :) For discussions and questions, please use Discussions tab, and for faster communication, have a look at my Zero To Hero Discord, channel nanoGPT: Video Let's reproduce GPT-2 (124M) YouTube lecture Errata Minor cleanup, we forgot to delete register_buffer of the bias once we switched to flash attention, fixed with a recent PR. Earlier version of PyTorch may have difficulty converting from uint16 to long. Inside load_tokens, we added
Top issues
#6Consider using `torch.compile(model, fullgraph=True, mode="reduce-overhead")`11 comments · 2024-06-11#25Sharding the dataset not completing?7 comments · 2024-06-14#47Cannot get the log file "log124M_40B/log.txt"?5 comments · 2024-07-01#18Embeddings are initialized with std of 0.023 comments · 2024-06-12#92Model convergence differences between cpu and mps2 comments · 2025-03-09#89Why does weights sharing have to be in that direction?2 comments · 2025-01-16#49How to support padding in the train dataset for training ?2 comments · 2024-07-02#48Integrating GPT-2 with deepspeed Zero-1, Zero-2 and Zero-32 comments · 2024-07-01
Discussions
#114Modular Reimplementation0 comments · 2026-08-26#113Reproduced ChatGPT2 (124M): pretrained model + SFT + RLHF !!!0 comments · 2026-05-19#112Why is bias used in the Attention Layer ?0 comments · 2026-05-05#107Reproduced GPT-2 (124M) + RoPE + SwiGLU — trained from scratch on FineWeb-Edu (10BT) using RTX 50902 comments · 2025-11-29#106Re-building and improving GPT-20 comments · 2025-09-24#101Implemented rotary positional embedding and swiglu activation0 comments · 2025-07-02#98Will training be faster and efficient without Torch.cuda.synchronize()1 comments · 2025-04-22#97Not sure to understand well Lamda AI plz helppp1 comments · 2025-04-06