AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Understanding Positional Embeddings in Modern LLMs

Collected Nov 15, 2026

Ahead of AI has published a long technical walkthrough on how large language models represent token position, spanning learned absolute embeddings, the original transformer's fixed sinusoidal scheme, and rotary position embeddings (RoPE). The post is framed as a standalone building-block piece meant to feed a later survey of 2026 LLM architecture trends, and it also previews which positional choices appear in the author's gallery of 2026 open-weight architectures.

The starting point is where position enters the network. A tokenizer maps text to token IDs, an embedding layer turns each ID into a feature vector, transformer blocks refine those vectors through attention and feedforward layers, and an output layer produces next-token logits. Position information is injected at different points depending on the architecture: GPT-2 adds a position vector to each token embedding before the first transformer block, while MiniMax-M3, cited as a stand-in for recent models, applies positional information inside attention in every block via RoPE. The post notes as an aside that the first three of MiniMax-M3's 60 feedforward layers are dense rather than mixture-of-experts.

The motivating example is word order. In "The red fox jumps over the gray fox" versus "The gray fox jumps over the red fox," the two instances of "fox" share a token ID and therefore the same token embedding, so without positional signal the representations would be identical. In a plain self-attention layer with no causal mask, swapping any two words leaves the pairwise attention scores unchanged. GPT-style causal masks, which restrict each token to itself and earlier tokens, already supply some implicit ordering, since "jumps" attends to different earlier words in the two sentences. The author's argument is that this implicit signal is not enough, because a causal mask cannot distinguish a gap of one position from a gap of ten, which is why explicit position methods remain in use.

GPT-2's approach is the simplest explicit one. It learns a table of position vectors alongside the token embedding table, with matching output dimensions so the two can be summed element-wise before the first block: the hidden state at position m is the token embedding plus a position vector selected by m. Because the vector depends only on the token's own index, this is an absolute scheme. The two "fox" tokens start identical but diverge once different position vectors are added, and their query, key and value vectors inside attention carry that information forward. The cost is a hard cap: GPT-2 defines 1,024 position vectors, so inputs longer than that do not fit, and the table size must be fixed before training. The post puts numbers on the scaling problem: for GPT-2 Medium at 1,024 positions the positional table is only 0.30% of total weights, but stretching the context to a common 100k range would push it above 22%, which the author calls uneconomic.

The 2017 "Attention Is All You Need" transformer uses the same additive mechanism but computes position vectors from fixed sine and cosine functions rather than learning them. It also scales token embeddings by the square root of the model dimension before adding positions; the paper does not explain this factor, and the post offers a reasoned guess that it puts both components on comparable numerical scales at initialization. GPT-2 did not need it because its initializations were already close (standard deviation 0.02 for token embeddings, 0.01 for position embeddings). In the sinusoidal scheme, features are grouped into pairs, with the even entry taking the sine and the odd entry the cosine of the position multiplied by a frequency that decreases geometrically across pairs. For an eight-feature example the frequencies are 1, 0.1, 0.01 and 0.001 radians per position, so at position 2 the first pair is roughly (0.909, -0.416) and the second, slower pair roughly (0.199, 0.980). These vectors are still absolute and are fixed during training, though they can be evaluated beyond the trained context length, which does not by itself guarantee good performance there. The design rationale, quoted from the paper, is that for any fixed offset k the encoding at position pos + k is a linear function of the encoding at pos, which the authors expected would help attention learn relationships like "three tokens earlier." A shift of three positions always rotates a given pair by the same angle, so the difference between positions 2 and 5 matches the difference between 102 and 105.

RoPE is the third mechanism and the pivot of the piece. It abandons additive vectors in favor of encoding relative position through rotation applied to query and key vectors inside attention, which is why modern blocks like those in MiniMax-M3 carry position information at every layer rather than once at the input. This suggests a practical trade-off: relative schemes decouple the parameter budget from context length, avoiding the table growth that makes learned absolute embeddings expensive at long context, but they shift complexity into the attention computation itself, where the rotation must be applied consistently, and extending context still requires care because models do not automatically generalize to positions far beyond those seen in training. The author also points readers to a section on architectures that adapt or omit explicit positional encodings, and notes that the article assumes familiarity with attention internals, recommending a prior self-attention walkthrough as background.

Why it matters: Developers choosing or fine-tuning open-weight models are increasingly working with RoPE-based architectures rather than the learned absolute tables of the GPT-2 era, and that choice shapes long-context behavior, memory footprint and how far a model can be stretched at inference. In practice, teams evaluating context extension, positional interpolation or custom attention kernels need to know which scheme a checkpoint uses before assuming how it will behave past its trained window, and the post's gallery grouping of 2026 architectures by encoding type is a useful map for that check. This is inference from the described mechanisms, not a claim the author makes about any specific model's performance.

Read at Ahead of AI

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

The Methods Behind How LLMs Represent Token Positions