Andrej Karpathy
Andrej Karpathy: Self-Attention Is Communication, Feed-Forward Is Thinking
In his breakdown of GPT from scratch, Andrej Karpathy explains the core mechanics of transformers: why attention lets tokens talk to each other while feed-forward layers do the actual thinking.
Andrej Karpathy▶ 71:41
"Number one, attention is a communication mechanism. You can really think about it as a communication mechanism where you have a number of nodes in a directed graph... every node has some vector of information and it gets to aggregate information via a weighted sum from all of the nodes that point to it."
Verified (98/100)Read →