Attention mechanisms have become the central architectural component of the most powerful AI systems. GPT-4, Gemini, DALL-E 3, AlphaFold — all rely fundamentally on attention. Yet many practitioners use these systems without a clear picture of what attention is actually computing. This article provides a grounded mathematical and intuitive explanation of attention, from its origins in sequence-to-sequence models to its modern multi-head, multi-layer form.
Origins: Sequence-to-Sequence with Attention
Attention was first introduced in the context of neural machine translation by Bahdanau and colleagues in 2015. The challenge was a fundamental limitation of encoder-decoder RNNs: the entire source sentence had to be compressed into a fixed-size context vector, which the decoder used to generate each target word. For long sentences, this bottleneck caused information loss and poor translation quality.
The attention solution was elegant: instead of passing a single fixed context vector, allow the decoder to look back at all encoder hidden states at each generation step, computing a weighted average of them. The weights — the attention weights — are computed by a small neural network that takes the decoder's current hidden state and each encoder hidden state as inputs, producing a score that reflects how relevant each source position is to generating the current target word.
Scaled Dot-Product Attention
The Transformer's attention mechanism simplifies and generalises Bahdanau's approach. Given matrices Q (queries), K (keys), and V (values), scaled dot-product attention computes:
Attention(Q, K, V) = softmax(QK^T / √d_k)V
Here, d_k is the dimension of the key vectors. The scaling by √d_k prevents the dot products from growing large in magnitude for high-dimensional vectors, which would push the softmax into regions of very small gradients.
Intuitively: for each query (representing something we want to look up), we compute a similarity score with every key (representing the "index" of each value). The softmax converts these scores into a probability distribution, which is used to take a weighted sum of the values. This is precisely the continuous, differentiable analogue of a dictionary lookup: instead of exactly matching a key, we retrieve a blend of all values weighted by query-key similarity.
Multi-Head Attention
Single-head attention computes one set of attention weights — one perspective on the input. Multi-head attention runs h parallel attention operations with different learned linear projections of Q, K, and V, then concatenates the results and projects back to the original dimension:
MultiHead(Q, K, V) = Concat(head_1, ..., head_h)W_O
head_i = Attention(QW_i^Q, KW_i^K, VW_i^V)
Different heads learn to attend to different aspects of the input. In a language model, one head might learn syntactic dependencies (verbs and their subjects), another coreference (pronouns and their antecedents), another positional proximity. The diversity of attentional patterns captured across heads is a key source of Transformer expressiveness.
Positional Encoding
Attention is permutation-invariant: shuffling the input sequence produces the same attention weights, just in a different order. This is a problem for language, where order is semantically critical. Transformers inject positional information by adding positional encodings to the input embeddings. The original Transformer used sinusoidal encodings: for position p and dimension d, PE(p, 2i) = sin(p / 10000^(2i/d_model)) and PE(p, 2i+1) = cos(p / 10000^(2i/d_model)). These functions have the useful property that the encoding at position p+k can be represented as a linear function of the encoding at position p, allowing the model to attend to relative positions.
Modern models use learned positional embeddings or Rotary Position Encoding (RoPE), which encodes relative positions by rotating query and key vectors — providing better generalisation to sequence lengths beyond those seen during training.
Computational Complexity and Long Contexts
The standard self-attention mechanism scales quadratically with sequence length: computing QK^T for a sequence of length n produces an n×n attention matrix. For sequences of a few thousand tokens, this is manageable. For very long contexts — entire codebases, books, hour-long transcripts — the memory and compute requirements become prohibitive.
Efficient attention approximations address this: sparse attention (computing attention only over a subset of pairs), linear attention (approximating the softmax with a kernel function that decomposes the computation), and flash attention (an IO-aware implementation that avoids materialising the full attention matrix in GPU SRAM). These techniques have extended practical context windows from a few thousand to hundreds of thousands of tokens, enabling new applications in long-document analysis and extended reasoning.