Let's build GPT: from scratch, in code, spelled out.

Andrej Karpathy builds a generative pretrained transformer from scratch in code, spelling out the math of self-attention, feed-forward computation, residual streams, and training dynamics.

Status:completed

Extracted Stories:1 stories

Intelligence Map:4 Chapters • 2 Evidence Units

Audience Intel:Pending Window

Published: Invalid Date

Intelligence Engine Map

📍Interactive Conversation Chapters (4)

Derived from full-length transcript segmentation with programmatic timestamp alignment

Introduction & Character-Level Language Modeling

Setting up the tiny Shakespeare dataset, tokenization, train/val splits, and building a simple baseline bigram language model in PyTorch.

Subtopics:TokenizationBigram BaselineLoss Functions & PyTorch Training

The Self-Attention Mechanism as Communication

Mathematical derivation of self-attention: using triangular masks for causal language modeling and query-key dot products as data-dependent communication weights.

Subtopics:Query Key ValueCausal MaskingMulti-Head Attention

⚡Disagreement / Intellectual Tension: Attention as Algebraic Weighting vs Graph Communication Protocol

Andrej Karpathy:"Self-attention is fundamentally a communication mechanism where graph nodes pool vectors via weighted sums"

Conventional Textbooks:"Attention is primarily treated as matrix multiplication optimization over embedding tables"

Feed-Forward Networks: Individual Token Computation

Why multi-head attention must alternate with multi-layer perceptrons: once tokens communicate context, feed-forward layers allow tokens to think individually.

Subtopics:Feed Forward MLPCommunication vs ComputationValidation Loss Dynamics

Residual Connections, LayerNorm & Scaling Deep Transformers

Introducing residual connections and pre-layer normalization to stabilize gradient backpropagation across deep transformer stacks.

Subtopics:Residual StreamsPre-LayerNormFull GPT Architecture

Mental Models & Frameworks

🧠Key Knowledge Cards (2)

Distilled explanations, technical frameworks, and mental models from this conversation

Andrej Karpathy

Attention as a Communication Protocol

Self-attention acts as a data-dependent communication network where nodes in a directed graph pool contextual vectors from previous tokens.

"A group meeting where tokens exchange notes and pool information from across the sequence."

Andrej Karpathy

Feed-Forward as Individual Computation

While self-attention handles cross-token communication, feed-forward multilayer perceptrons allow each token to individually process and compute on the aggregated data.

"Tokens returning to their individual desks after a group meeting to think privately on the gathered data."

Accountability Tracker

🔮Logged Predictions & Forecasts (1)

Specific, falsifiable predictions extracted and tracked over time

Andrej KarpathyTarget: Foundational

"Building and inspecting transformers from raw matrix multiplications in code is the only way to gain mechanical intuition about modern LLMs."

Status: verified

Extracted Editorial Stories (1)

Verified moments and deep-dive analyses derived from this episode

Andrej Karpathy: Self-Attention Is Communication, Feed-Forward Is Thinking

💡Golden Nuggets

4 min

Andrej Karpathy

Andrej Karpathy: Self-Attention Is Communication, Feed-Forward Is Thinking

In his breakdown of GPT from scratch, Andrej Karpathy explains the core mechanics of transformers: why attention lets tokens talk to each other while feed-forward layers do the actual thinking.

Andrej Karpathy▶ 71:41

"Number one, attention is a communication mechanism. You can really think about it as a communication mechanism where you have a number of nodes in a directed graph... every node has some vector of information and it gets to aggregate information via a weighted sum from all of the nodes that point to it."

Verified (98/100)Read →