How to Build and Train Your Own GPT on a Regular Laptop

Most neural network tutorials today boil down to two scenarios. You're either asked to call the OpenAI API with a couple of lines of Python, or to download a ready-made model from Hugging Face and call the .generate() method. In both cases, all the inner workings stay behind the scenes. You get a result, but you barely understand how matrix multiplications turn raw text into meaningful phrases.

Recently I came across the llm-from-scratch repository by developer angelos-p. It's an interactive workshop where you write every component of a language model from scratch. No heavy abstractions: just pure PyTorch, clear math, and training a 10 million parameter model on your laptop in under an hour.

Входной текст
    │
    ▼
┌─────────────────┐
│   Tokenizer     │  "hello" → [20, 43, 50, 50, 53]
└────────┬────────┘
         ▼
┌─────────────────┐
│  Token Embed +  │  Векторы токенов + позиция
│  Position Embed │  (размерность n_embd)
└────────┬────────┘
         ▼
┌─────────────────┐
│  Transformer    │  × n_layer слоев
│  Block:         │
│  ┌────────────┐ │
│  │ LayerNorm  │ │
│  │ Self-Attn  │ │  n_head параллельных голов внимания
│  │ + Residual │ │
│  ├────────────┤ │
│  │ LayerNorm  │ │
│  │ MLP (FFN)  │ │  расширение 4x, GELU, проекция назад
│  │ + Residual │ │
│  └────────────┘ │
└────────┬────────┘
         ▼
┌─────────────────┐
│   LayerNorm     │
│   Linear → logits│  вероятности следующего символа
└─────────────────┘

Where It All Started

Many have heard of Andrej Karpathy's nanoGPT project. Karpathy showed that a full GPT-2 architecture with 124 million parameters can fit into just a couple hundred lines of code. However, reproducing the original GPT-2 still requires a decent amount of computation and time.

The author of llm-from-scratch went further. They removed all unnecessary cluster optimizations and adapted the code so it can be written by hand in a single evening. The final model trains on Apple Silicon (MPS), Nvidia GPUs via CUDA, or even a regular CPU. If you don't have local hardware, everything runs in free Google Colab.

What the Workshop Contains

The material is broken down into step-by-step modules in the docs/ folder. You sequentially create three working files: model.py, train.py, and generate.py.

1. Character-Level Tokenization

When working with massive text corpora, the standard solution is the BPE (Byte Pair Encoding) algorithm with a 50,000-token vocabulary. But if you're training a small network on a 1-megabyte dataset, BPE will break your training. Most token pairs will appear only a couple of times across the entire text, and the weights simply won't converge for them.

That's why the author uses a character-level tokenizer for training on Shakespeare's plays. The alphabet consists of just 65 unique characters. This vocabulary is compact, encodes quickly, and gives a tiny model a chance to capture grammar and dialogue structure.

2. Building the Transformer

Here you hand-assemble the GPT decoder architecture:

• Embedding tables for characters and positions in the text.
• Multi-Head Self-Attention block, where each character's vector interacts with context through queries, keys, and values (Q, K, V).
• Causal mask mechanism that prevents the model from peeking at future tokens during generation.
• Fully connected layers (MLP) with GELU activation and LayerNorm normalization.

The architecture is assembled without magic from third-party frameworks. You can clearly see where every tensor flows and at what stage residual connections are added.

3. Training Loop

In the third stage, you write the training pipeline. It includes:

• Calculating Cross-Entropy loss between predicted logits and actual next characters.
• AdamW optimizer with weight decay tuning.
• Gradient clipping to avoid weight explosion during training.
• Learning rate scheduler with warmup and cosine decay.

4. Text Generation and Sampling

The finished model needs to be made to speak. You implement an autoregressive loop: take a text seed, pass it through the transformer, extract the probability distribution for the next token, and select it using temperature with greedy search or top-k sampling.

Configurations for Experiments

The repository offers three model profiles. You can choose based on how much free time you have.

| Profile | Parameters | Layers (n_layer) | Heads (n_head) | Dimension (n_embd) | Training Time (M3 Pro) | |---|---|---|---|---|---| | Tiny | ~0.5M | 2 | 2 | 128 | ~5 minutes | | Small | ~4M | 4 | 4 | 256 | ~20 minutes | | Medium | ~10M | 6 | 6 | 384 | ~45 minutes |

All profiles work with a context length of 256 characters. The Tiny profile is great for quick debugging when you need to check that the code runs without errors shape mismatch. The Medium profile already produces quite readable Shakespearean prose with character dialogue divisions.

How to Run the Project

For environment management, the author recommends the fast package manager uv.

Installing dependencies and creating a working folder:

# Установка uv (если еще не установлен)
curl -LsSf https://astral.sh/uv/install.sh | sh

# Клонируем репозиторий и синхронизируем окружение
git clone https://github.com/angelos-p/llm-from-scratch
cd llm-from-scratch
uv sync
mkdir scratchpad && cd scratchpad

If you prefer Google Colab, installing a minimal set of libraries is enough:

!pip install torch numpy tqdm tiktoken

Then you take the data/shakespeare.txt file, open the first guide docs/01-tokenization.md, and start writing code step by step.

Who This Workshop Is For

The project will appeal to those tired of abstract articles with attention plots and wanting to see real code. No deep knowledge of machine learning is required to complete it. Being comfortable reading Python syntax and understanding basic array operations is enough.

After completing all six guides, you'll get rid of the feeling of magic around LLMs. You'll understand in practice why models need learning rate warmup, how temperature affects the randomness of responses, and why small models are so sensitive to vocabulary size. It's a great way to spend a weekend productively for your engineering knowledge.