Positional encoding
In short
Positional encoding is how a transformer model knows the order of words. Because a transformer reads every token at once, extra information about each token's position is mixed in, so "dog bites man" differs from "man bites dog".
- Also called
- position embedding, positional embedding, RoPE, rotary position embedding
- Topic
- How AI models work
- Level
- Technical
- First described
- Vaswani and colleagues at Google introduced sinusoidal positional encodings in "Attention Is All You Need", June 2017; the paper credits Noam Shazeer with the parameter-free position representation1.
Older language models read text one word at a time, so order came for free. The transformer has no recurrence and no convolution, so the 2017 paper that introduced it notes that information about each token's position has to be injected separately1.
The original transformer added a positional encoding to each token's embedding at the bottom of the network. It had the same size as the embedding, so the two could simply be summed1. The authors built it from sine and cosine waves of different frequencies, with wavelengths running from 2π to 10,000 × 2π1. They chose this because it should let the model attend by relative position: the encoding for a position a fixed distance ahead is a linear function of the current one1.
The same paper tried learned position embeddings, where the model learns a vector for each position, and found results were nearly identical. It kept the sine version because it might cope with sequences longer than any seen in training1.
A popular later method is rotary position embedding, or RoPE, proposed by Jianlin Su and colleagues in 2021. It encodes a token's absolute position by rotating its vectors, which builds relative position directly into the attention step2. Meta's LLaMA models dropped absolute position embeddings and used RoPE at every layer instead3.
The choice matters for long inputs. The original authors picked their method partly in the hope of handling longer sequences than seen in training1, and RoPE's authors list flexibility of sequence length as one of its properties2.
Example
"The cat chased the dog" and "The dog chased the cat" contain exactly the same words. Without positional encoding, a transformer would see the same set of tokens in both and could not tell who chased whom1.
Why it matters
Word order carries meaning, and transformers cannot see it without this. Choices here also shape how much text a model can read at once.
Questions people ask
Why do transformers need positional encoding?
A transformer has no recurrence or convolution, so it does not read tokens in order. Without position information added in, it cannot tell the order of the sequence1.
What is RoPE?
Rotary position embedding, proposed by Su and colleagues in 2021. It rotates each token's vectors by an amount set by its position, so attention scores naturally reflect how far apart two tokens are2. Meta's LLaMA models use it3.
Are positional encodings learned or fixed?
Either. The original transformer used fixed sine and cosine patterns, but its authors found learned position embeddings worked almost exactly as well1.