Enter a sentence and see a simplified self-attention heatmap — the mechanism transformer models use to relate words to each other.
Self-attention is the mechanism that lets a transformer model determine how much each word in a sequence should "attend to" (be influenced by) every other word, including itself, when building its representation of the sentence. The heatmap above visualizes this, click any word (row) and see how strongly it relates to every other word (column) in the sentence. Darker cells mean stronger attention. This is the core innovation behind the transformer architecture, introduced in the 2017 paper "Attention Is All You Need," and it underlies essentially every modern large language model, including GPT and Claude.
Before transformers, sequence models (like RNNs and LSTMs) processed text strictly in order, one word at a time, making it computationally difficult to directly relate a word to another word far earlier in a long sequence, information had to pass through every intermediate step, degrading over distance. Self-attention lets every word directly relate to every other word in a single computation, regardless of distance between them, dramatically improving the model's ability to capture long-range dependencies, one word at the very start of a paragraph can directly influence how a word at the end is interpreted, no degradation over distance.
Real transformer attention computes three vectors for every word, a Query (what this word is "looking for"), a Key (what this word "offers" to others looking), and a Value (the actual information this word contributes if attended to). Attention weights are computed by comparing each word's Query against every word's Key, this visualization simplifies this into a single similarity comparison for illustrative clarity, real attention layers learn separate, trained Query/Key/Value transformations during training, allowing the model to develop far more sophisticated, task-relevant notions of "relatedness" than the simple word-similarity heuristic used here.
Production transformer models don't compute just one attention pattern per layer, they compute several in parallel, called "attention heads," each potentially learning to focus on different types of relationships, one head might specialize in tracking grammatical subject-verb agreement, another in tracking coreference (which pronoun refers to which noun), and so on. These multiple attention patterns get combined together, giving the model several simultaneous "views" of how words in a sentence relate to each other, a real transformer layer typically uses 8, 16, or more attention heads working in parallel, not just the single simplified pattern shown here.
Beyond capturing long-range relationships better than earlier architectures, self-attention's computations across all word pairs can be executed in parallel on GPUs, unlike the strictly sequential, one-step-at-a-time processing that RNNs required. This parallelizability is a major practical reason transformers scaled to the enormous model and dataset sizes that produced modern large language models, the architecture wasn't just more capable, it was also dramatically more efficient to train at scale.