Andrej Karpathy: Self-Attention Is Communication, Feed-Forward Is Thinking

In modern transformer architectures, understanding the interplay between attention mechanisms and multilayer perceptrons often gets lost in algebraic abstraction. During his line-by-line implementation of a generative pretrained transformer from scratch, former Tesla AI Director Andrej Karpathy offered one of the most intuitive architectural mental models in machine learning literature: the distinction between communication and computation.

Karpathy emphasizes that self-attention is fundamentally a communication mechanism rather than an isolated mathematical trick. Rather than treating embeddings as static representations, attention allows tokens to dynamically broadcast and aggregate contextual information across a directed graph structure.

> "Number one, attention is a communication mechanism. You can really think about it as a communication mechanism where you have a number of nodes in a directed graph... every node has some vector of information and it gets to aggregate information via a weighted sum from all of the nodes that point to it." > - Andrej Karpathy (71:41)

However, communication alone does not produce reasoning or synthesis. Once tokens have gathered information from their context window via attention heads, they require dedicated capacity to transform and synthesize that collected knowledge. This is precisely where feed-forward layers enter the pipeline.

> "Self-attention is the communication, and then once they've gathered all the data, now they need to think on that data individually. And so that's what feed forward is doing." > - Andrej Karpathy (86:09)

This explicit separation of roles - inter-token communication followed by intra-token computation - explains why alternating attention and feed-forward blocks with residual streams forms the backbone of modern large language models.