AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention

Collected Sep 30, 2026

A survey of recent open-weight model releases published by Ahead of AI identifies long-context efficiency as the dominant architectural theme across models shipped in April and May. As reasoning models and agent workflows hold more tokens for longer, key-value cache size, memory traffic and attention cost become the binding constraints, and model authors are responding with increasingly specific changes inside the transformer block. The article examines four cases: KV sharing and per-layer embeddings in Google's Gemma 4, layer-wise attention budgeting in Poolside's Laguna XS.2, compressed convolutional attention in Zyphra's ZAYA1-8B, and mHC plus compressed attention in DeepSeek V4.

Gemma 4 shipped in early April in three tiers: the E2B and E4B models aimed at phones and embedded or IoT devices, a 26B mixture-of-experts model tuned for efficient local inference, and a 31B dense model positioned for maximum quality and easier post-training. The E variants use cross-layer attention, a scheme in which later layers skip their own key and value projections and instead reuse KV tensors computed by an earlier non-shared layer of the same attention type. Sliding-window layers share with earlier sliding-window layers, and full-attention layers share with earlier full-attention layers. Each layer still computes its own queries, so it can form a distinct attention pattern while the memory-heavy cache is amortized across layers.

E2B has 35 layers but only the first 15 compute KV projections; the remaining 20 reuse earlier tensors. E4B has 42 layers, with 24 computing KV and 18 sharing. Because roughly half the KVs are shared, cache size drops by about half, which the article puts at a 2.7 GB saving for E2B and about 6 GB for E4B at 128K context in bfloat16. The approach derives from published cross-layer attention work from 2024, and Gemma 4 is described as the first widely used architecture applying it. The trade-off is a loss of model capacity, though the original research found the impact minimal for the small models it tested. In practice, this is the kind of change that mainly buys longer usable context on memory-limited hardware rather than higher quality.

The E models add a second mechanism, per-layer embeddings (PLE), aimed at parameter efficiency rather than cache size. The "E" stands for "effective": E2B is listed at 2.3B effective parameters, or 5.1B counting embeddings, and E4B at 4.5B effective, or 8B with embeddings. PLE vectors are built outside the repeated blocks: token IDs go through a per-layer embedding lookup, normal token embeddings are linearly projected into the same space, and the two are combined, scaled and reshaped into a packed tensor with one slice per layer. Each block receives only its own slice. Inside the block, attention and feed-forward residual updates run as usual, then the resulting hidden state gates that layer's PLE vector, which is projected back to hidden size, normalized and added as an extra residual update.

The intent is to keep the expensive transformer stack near the smaller effective size while parking extra capacity in embedding tables, which are cheap lookup-style parameters that can be cached. The alternative, shrinking layers or hidden width, would reduce memory and latency but strip capacity from the parts doing the main computation. The article notes that only Google can confirm the design pays off and that a comparison against plain dense models of 2.3B and 5.1B parameters would be informative. PLE is not inherently limited to small models, but larger models already have capacity and typically use mixture-of-experts routing for that purpose.

Poolside's Laguna XS.2, the company's first open-weight model, looks conventional at first glance but varies attention cost by layer. Of its 40 layers, 30 use sliding-window attention over a 512-token window and 10 use global attention. That hybrid pattern is common, including in Gemma 4. What is notable is per-layer query-head counts, exposed in the Hugging Face config as num_attention_heads_per_layer: global layers use 6 query heads per KV head and sliding-window layers use 8, with KV heads fixed at 8. The reasoning is that full-attention layers are expensive because they span the whole context, so they get fewer query heads while local layers get more. Varying capacity by layer traces back at least to Apple's 2024 OpenELM. Laguna also applies per-head attention-output gating, similar to Qwen3-Next.

Zyphra's ZAYA1-8B was trained on AMD GPUs rather than the more common NVIDIA or Google TPU setups. Its attention block is built around compressed convolutional attention (CCA) combined with a 4:1 grouped-query layout. CCA is related in spirit to DeepSeek's multi-head latent attention, since both introduce a compressed latent representation into attention, but they use it differently. MLA stores KV tensors compactly and projects them back into head space for the actual attention computation, chiefly to shrink the cache. CCA compresses queries, keys and values and performs the attention operation directly in the compressed latent space.

Across these designs a pattern emerges: instead of one global efficiency knob, authors are budgeting capacity unevenly, spending attention where it is expensive and storing information where it is cheap. The practical question for developers is whether these savings translate into quality at a given size, which the article leaves open. DeepSeek V4's mHC and compressed attention are covered in the same piece, and V4 had not been released at the time of writing.

Why it matters: Teams running long-context or agent workloads on open-weight models should expect these efficiency techniques to become standard, since they directly lower memory per context window and shape what fits on a single GPU or device. Anyone fine-tuning or building inference stacks will need to handle non-uniform layer configurations such as shared KV caches and per-layer head counts, which complicates tooling that assumes a uniform transformer block, an inference drawn from the reported designs rather than stated by the publisher.

Read at Ahead of AI

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

From Gemma 4 to DeepSeek V4, How New Open-Weight LLMs Are Reducing Long-Context Costs