Course Overview

• Build a (Tiny) LLM from Scratch

Fundamentals

• What is an LLM?
• The Transformer: LLMs' Core Architecture
• Inside the Transformer Block
• Training vs Inference: Two Modes
• Transformer Components: The Complete Map

Linear Algebra Basics

• Vectors and Arrays: Lists of Numbers
• Vector Operations: Addition, Scaling, and Dot Product
• Matrices and Shapes: Tables of Numbers
• Matrix Operations: Adding, Scaling, and Multiplying

Text to Numbers

• Module 1: From Text to Numbers
• Tokenization Basics
• Subword Tokenization (BPE)
• Embeddings - Words as Vectors
• Vector Similarity

Core Concepts

• Module 2: Predicting the Next Word
• Functions with Learnable Parameters
• Matrix Multiplication - The Core Operation
• Predicting the Next Word
• From Embeddings to Logits
• Softmax - Turning Scores into Probabilities

Context-Aware Predictor

• Building a Next-Word Predictor

Introduction

• Module 3: Attention Mechanism

Query, Key, Value

• We Need Context: From Last Word to Weighted Sum
• Queries and Keys: How One Word Finds Relevant Context
• Q and K Matrices: Computing All Attention Scores at Once
• Attention Dimensions
• From Scores to Attention Weights
• Values: What Attention Actually Mixes
• Multi-Head Attention: Many Patterns at Once

Positional Information

• Positional Encoding: Teaching the Model Word Order

Context-Aware Predictor

• Building a Context-Aware Next-Word Predictor

Introduction

• Module 4: The Transformer Architecture

Stabilizing the Network

• Layer Normalization
• Residual Connections

Assembling Our Model

• Assembling Our Model

Scaling to Real Transformers

• Why Attention Alone Isn't Enough
• How Activation Functions Enable Learning
• Feed-Forward Networks: What Real Transformers Add

Generation

• Sampling Strategies: Controlling Generation
• Generating Text: Autoregressive Prediction

The Complete Picture

• The Complete Picture: From Text to Prediction
• Attention Is All You Need: Paper to Practice