Attention Is All You Need

The big idea

A machine-learning model cannot read a sentence as humans do. It receives numbers and must learn which numbers are related. Before Transformers, many sequence models read text one item at a time and carried a running memory forward. This worked, but made long-range relationships and parallel computation difficult.

A Transformer starts from a different idea: let each token directly inspect every other relevant token. The mechanism that decides what to inspect is called attention. Attention does not replace the rest of a neural network; it is the information-routing operation at the center of each Transformer block.

We will carry one sentence through the complete architecture: Ninja moved the plant near the window because it needed more sunlight. This example makes attention useful immediately: the model must connect it to plant, not to the closer noun window. The vectors are deliberately tiny so every calculation remains visible. Real models use the same operations with much larger vectors, more attention heads, and many repeated layers.

Experiment in Playground 1: move one stage at a time from tokenization to decoding. First learn the route; the next playgrounds will open each stage and show its numbers.

Source: Vaswani et al., Attention Is All You Need.

1. Text must become numbers

The model first uses a tokenizer. A token may be a word, part of a word, punctuation, or another vocabulary unit. The tokenizer turns each token into an integer ID. That ID is only an address; a larger ID does not mean a more important word.

The ID selects one row from a learned embedding table. If the table is named E, then x_i = E[token_id_i] means: “look up the starting vector for token i.” The vector has d_model numbers. During training, the model adjusts these numbers so tokens useful in similar contexts develop useful geometric relationships.

Our vector for plant has four dimensions, such as[0.80, 0.50, 0.20, 0.60]. Those four entries do not have simple human labels. Together they form a learned representation.

Experiment in Playground 2: select every token. Notice that each vocabulary ID retrieves a different vector, while every vector has the same width. Compare plant and window; focus on the full pattern, not one number.

2. The model also needs word order

Self-attention can compare a collection of vectors, but by itself it does not know which token was first. “Ninja moved the plant near the window” and “The plant moved Ninja near the window” contain similar tokens but express different ideas. A position signal must therefore be combined with each token embedding.

The original Transformer used sine and cosine waves. Each vector dimension oscillates at a different frequency. For an even dimension, the value is a sine; for an odd dimension, it is a cosine. You do not need to memorize the exponent. The important idea is that each position receives a distinctive, smoothly changing numerical signature.

The final input is simply token embedding + positional encoding. Because it is addition, the vector keeps token information while also carrying order information.

Experiment in Playground 3: move the position from 0 to 12. Watch fast-changing dimensions and slow-changing dimensions. Compare neighboring positions, then compare positions far apart.

3. Query, key, and value are three views of a token

Attention gives each token three jobs. A query says what information this token is looking for. A keysays what kind of information this token can match. A value carries the information that will actually be returned.

Think of a search system: the query is what you typed, keys are the searchable labels, and values are the documents returned. A token creates all three using learned matrices:Q = XW_Q, K = XW_K, andV = XW_V.

Multiplying by a matrix means each output number becomes a weighted combination of the input numbers. The three matrices learn different combinations, so Q, K, and V are different even though they start from the same token vector.

Experiment in Playground 4: select one token and compare its input, Q, K, and V vectors. Then change the token. Notice that the projection rule stays fixed while the resulting vectors change with the input.

4. Attention measures compatibility

To decide whether one token should read another, the model compares a query with a key using a dot product. Suppose the vectors are [2, 1] and [3, 4]. Their dot product is (2 x 3) + (1 x 4) = 10. Similar directions tend to produce larger scores.

The raw score is divided by sqrt(d_k), the square root of the key-vector width. With large vectors, many products are added together, so scores can grow too large. Dividing keeps them in a range where softmax can learn without becoming immediately extreme.

After scores become percentages, the model multiplies each value vector by its percentage and adds the vectors. This weighted sum is the token's new context vector. A 50% weight means that value contributes much more than a value with a 5% weight.

Experiment in Playground 5: choose a different query token. Read each scaled score, its percentage, and the final context vector. Confirm that all percentages total approximately 100%.

5. Softmax turns scores into usable weights

Attention scores may be negative, positive, tiny, or large. Softmax converts them into positive weights that add to one. In plain language: exponentiate every score, then divide each result by the total. A larger score gets a larger share, but every token can still receive some weight.

The formula p_i = exp(s_i) / sum(exp(s_j)) says thatp_i is the percentage for score s_i. The denominator is the same for all tokens, which is why the final percentages sum to 100%.

Temperature provides another way to control sharpness. A low temperature exaggerates differences and makes one choice dominate. A high temperature flattens the distribution. Temperature is common during output sampling; this playground also uses it to make the softmax behavior easy to see.

Experiment in Playground 6: first change onlyd_k, then change only temperature. Find settings that make one token dominate, and settings that make the weights more even. Compare raw scores with scaled scores.

6. Multiple heads can notice different relationships

One attention head produces one pattern of information flow. That can be restrictive: a sentence may simultaneously contain local grammar, a subject-action relationship, and a distant reference. Multi-head attention gives the model several independent sets of Q, K, and V projections.

Each head works in a smaller feature space and can learn a different pattern. Their output vectors are concatenated, meaning placed next to one another, and a final matrix W_O mixes them back into one d_model-wide vector.

Experiment in Playground 7: switch among all three heads. Find the local verb relationship, the link from it back to plant, and the need for sunlight. Then compare each pattern with the displayed mean. The note below the playground explains the real concatenation step.

7. Attention lives inside a repeated block

A Transformer is not one attention calculation. It stacks blocks. In an encoder block, multi-head attention first mixes information between token positions. A feed-forward network then transforms each position. Residual connections and normalization surround these operations.

Early layers may capture simple local patterns. Later layers receive already contextualized vectors and can compose richer behavior. Every layer preserves the sequence length and d_modelwidth, making blocks easy to repeat.

Experiment in Playground 8: select the six stages in order. For each stage, ask whether it mixes tokens with other positions, transforms one position, or protects the existing signal.

8. Residual paths preserve information

A sublayer should improve a representation without having to rebuild it from scratch. A residual connection computesy = x + Sublayer(x). The original vector xhas a direct route forward, while the sublayer contributes an update.

Layer normalization then looks across one token's features. It subtracts their mean and divides by their standard deviation. This produces a more controlled feature scale. Real LayerNorm also learns a scale and offset after normalization.

In (y_i - mean) / sqrt(variance + epsilon), epsilon is a tiny safety number that prevents division by zero. Mean describes the center; variance describes how spread out the values are.

Experiment in Playground 9: set the sublayer contribution to zero, then increase it gradually. Compare the original vector, residual sum, mean, variance, and normalized output.

9. The feed-forward network thinks per token

Attention moves information between token positions. The feed-forward network, or FFN, performs a nonlinear transformation on each position separately. The same FFN weights are reused for every position in the sequence.

The first matrix expands the vector into a larger hidden dimension. An activation function bends the numerical space, allowing behavior that stacked linear operations alone cannot express. A second matrix projects the hidden vector back to d_model.

ReLU replaces negative values with zero. GELU changes them smoothly instead of making a hard cut. The original Transformer used ReLU; GELU is common in later models.

Experiment in Playground 10: switch between ReLU and GELU. Follow the same vector through expansion, activation, and projection. Pay special attention to negative hidden values.

10. A decoder must not look at future answers

When generating the French translation, the decoder should predict the next French token from target tokens already available. During training the complete correct translation exists in memory, so an unrestricted attention operation could cheat by reading future target tokens.

A causal mask adds zero to allowed scores and negative infinity to future scores. Softmax turns exp(-infinity) into zero, so future positions receive no attention. Each target position can see itself and its past, never its future.

Experiment in Playground 11: move the decoder query through the shifted French decoder input. At position zero,<BOS> predicts Ninja. At later positions, watch future input tokens remain blocked while the label names the separate token being predicted.

11. Encoder and decoder have different jobs

The encoder reads the complete English source sentence, Ninja moved the plant near the window because it needed more sunlight, and builds contextual representations for all source tokens. Its self-attention is not causal because the whole input is already known.

The decoder first uses masked self-attention on the target prefix. It then performs cross-attention: decoder states provide queries, while encoder outputs provide keys and values. This lets the decoder ask the source sentence for information needed at the current output position.

When the decoder is ready to produce plante, its query can assign strong weight to the English representation for plant. Later, producing parce qu’elle avait besoin de plus de soleildepends on the source clause because it needed more sunlightand on resolving it to plant rather than the nearer window.

Experiment in Playground 12: step through source encoding, French target-prefix reading, cross-attention, and French vocabulary prediction. At each step, identify where Q, K, and V originate.

12. Decoding predicts one token at a time

The decoder's final hidden vector is projected to one number for every vocabulary token. These numbers are called logits. Softmax turns logits into next-token probabilities. In our running translation example, those candidates come from the French vocabulary.

Greedy decoding chooses the largest probability. Sampling draws from the distribution, allowing less likely choices. Whichever token is selected becomes part of the next input prefix, and the process repeats until an end token such as <EOS> is chosen.

Experiment in Playground 13: first choose the most likely French token at every step to produce Ninja a déplacé la plante près de la fenêtre parce qu’elle avait besoin de plus de soleil. Reset and choose alternatives to build a different toy prefix. The candidate lists are scripted for learning; a real decoder would recalculate later probabilities from the entire chosen prefix.

13. Training rewards probability on the correct token

During supervised translation training, the correct next French token is known. The model's probability for that token is converted into cross-entropy loss: L = -log(p_correct). If the correct probability is near one, the loss is near zero. If it is near zero, the loss becomes large.

Gradient descent measures how each parameter influenced this loss and nudges millions or billions of parameters in a direction that should reduce future loss. Teacher forcing supplies the real previous token at every training position, so masked target positions can be scored in parallel.

Experiment in Playground 14: move the correct-token probability from 1% toward 99%. Notice that loss falls quickly at first and approaches zero, rather than decreasing in a straight line.

14. Parallel computation still has a quadratic cost

Attention for many tokens can be expressed as matrix operations, which GPUs can perform in parallel. That is a major advantage over architectures that must finish one sequence step before starting the next during training.

Parallel does not mean free. With n tokens, there aren queries and n keys. Every query compares with every key, producing n x n = n squared scores. If length doubles from 8 to 16, cells increase from 64 to 256: four times as many.

This quadratic matrix is one reason long contexts consume substantial memory. Efficient-attention methods reduce the practical cost using locality, sparsity, compressed representations, or more memory-aware algorithms.

Experiment in Playground 15: try lengths 8, 16, 32, 64, and 128. Each time you double the length, verify that the exact score-cell count becomes four times larger.

Nuances worth remembering

• Attention weights are routing signals: they show how values are mixed in one head and layer, but are not a complete explanation of model reasoning.
• Tokenization affects the sequence: the model predicts vocabulary pieces, not necessarily whole words.
• Position is a design choice: the original paper used fixed sinusoids; modern systems may use learned, relative, or rotary position methods.
• Training and inference differ: masked training can score known target positions together, while autoregressive inference must select each next token before continuing.
• One playground is one layer-level view: real model behavior is composed across many heads, residual updates, nonlinear transformations, and repeated blocks.

The complete mental model

1. Tokenize text and look up embeddings.
2. Add position information.
3. Project each representation into Q, K, and V.
4. Use scaled dot products and softmax to route information.
5. Run several heads, then combine their outputs.
6. Use residuals, normalization, and an FFN inside each block.
7. Mask future target tokens inside the decoder.
8. Use cross-attention to read encoder memory.
9. Predict one next token and train using cross-entropy loss.
10. Append the selected token and repeat until generation stops.