AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Token Pooling-Aware Training for Multi-Vector Embedding Models

Collected Nov 11, 2026

A Hugging Face blog post by Tom Aarsen, published on the Hugging Face Blog, describes token pooling-aware training for multi-vector embedding models and ships a reproducible Sentence Transformers example. The core result: on NanoBEIR@13, a model trained with the new objective scores 51.23 mean nDCG@10 at a pool factor of 4 while storing 74.8% fewer document vectors than unpooled retrieval, against 49.03 for an otherwise identical model trained with the ordinary contrastive loss. Retention of each model's own unpooled score rises from 94.6% to 97.8%, and unpooled retrieval itself edges up from 51.84 to 52.39.

Multi-vector models such as ColBERT keep a contextual vector per document token rather than compressing a document into one vector, so a query can match different spans of a document separately. Scoring uses MaxSim: each query vector picks its best-matching document vector and the results are summed. The storage cost is the problem. Even with the 2-bit or 4-bit residual quantization Fast-PLAID supports, 200 vectors of 128 dimensions at 4 bits per coordinate come to 12.8 KB per document, or 12.8 GB over a million documents. Pooling to 50 vectors drops that to 3.2 GB at the same dimensionality and precision.

HierarchicalTokenPooling merges similar token embeddings with Ward hierarchical clustering. It starts with one group per token, repeatedly merging the pair whose combination increases within-group spread least, where spread is the sum of squared distances to the group mean. The merges form a hierarchy, so any compression level can be selected afterwards; each group is represented by its mean, renormalized for MaxSim. The default protects the first token from merging and applies pooling only to documents, leaving queries intact. Three controls are available: pool_factor (proportional, so long documents keep more vectors), a fixed token cap, and a distance threshold. Adaptive pooling can be used on any multi-vector model, including one never trained for it: calibrate_distance_threshold takes document embeddings from a sample corpus plus a target_tokens average and returns a threshold reusable at indexing time. That target is an average, so individual documents vary, and the achieved average can shift on a different corpus.

The training change mirrors Matryoshka embeddings, which apply a retrieval objective at several widths so short prefixes stay useful. Here each batch produces two losses, one on full document embeddings and one on documents compressed to about a quarter of their vectors, averaged with equal weights. Because Ward clustering makes discrete merge decisions, a straight-through estimator passes gradients from pooled vectors back to their member tokens without differentiating the assignments. In Sentence Transformers, MultiVectorTokenPoolingLoss wraps an existing retrieval loss, here CachedMultiVectorMultipleNegativesRankingLoss, and takes a list of poolers with weights; None denotes the unpooled variant. Extra poolers reuse the already-computed token embeddings, adding clustering and scoring work but no second encoder pass. Gradient caching lets mini-batches of 64 supply in-batch negatives for a training batch of 256.

The two evaluated models were trained from answerdotai/ModernBERT-base with a fresh 128-dimensional token projection, one epoch over the same 120,000 MS MARCO triplets, seed 42, identical initialization and optimization. Training used a 300-token cap, and short queries were expanded to at least 32 token positions for more contextual query vectors. Only the loss differed, and no hyperparameter search was run. Gains carried over to fixed caps and adaptive thresholds: at an average of 29.58 vectors per document, adaptive pooling keeps 97.3% of the pooling-aware model's unpooled score with 84.7% fewer document vectors, against 93.3% for the baseline at the same budget.

Kernel acceleration is optional. Installing sentence-transformers with the train and kernels extras and setting use_kernels=True on the model enables a MaxSim kernel that scores without materializing the full token similarity matrix, though it needs CUDA and Triton, and a Ward clustering kernel reported at roughly 30-150x faster than SciPy depending on settings. Hierarchical poolers inherit the model's kernel settings. Kernel settings are not saved with a model and must be re-enabled at load time. The released model tomaarsen/mve-ModernBERT-base-token-pooling compresses a three-document toy example from 58 vectors to 16 while keeping Mars ranked above Jupiter and Venus. Pool factor can be changed at inference, and passing token_pooling at initialization makes pooling the runtime default. Two commands run both variants locally with uv; a Hugging Face Jobs command runs the pooling-aware script on an a10g-small GPU with a four-hour timeout and uploads the result to the Hub.

The author's ordering for shrinking an index: quantization first (4, 2 or 1 bit via Fast-PLAID), then token pooling, optionally with pooling-aware fine-tuning, then Matryoshka token embeddings for shorter vectors; if default dimensionality exceeds 128, Matryoshka truncation with fine-tuning moves earlier. MultiVectorMatryoshkaLoss can wrap MultiVectorTokenPoolingLoss to train for both at once. An evaluation script measures unpooled retrieval, caps of 64, 32, 16, 8 and 4, and pool factors 2 through 6.

This suggests the practical appeal is that compression becomes part of the objective rather than a lossy afterthought, with the numbers holding across pool factors of 2, 3, 4 and 6 and at adaptive budgets. A likely trade-off is extra training cost, since clustered variants add work per batch, and the results come from a single epoch, one dataset and no tuning, so teams should reproduce the comparison on their own corpus. The gain is also measured in retrieval quality relative to each model's unpooled baseline, not against a single fixed index.

Why it matters: teams running multi-vector search pay for vectors at index scale, and a change confined to the loss function buys back roughly three points of retained nDCG at the same compression level in this experiment. Developers using Sentence Transformers can adopt it through a loss wrapper and existing pooler classes, then apply the same trained model at different pool factors for different storage budgets, so the choice of index size becomes adjustable after training. How far that holds on other corpora and larger models is an open question that the published evaluation script is designed to answer.

Read at Hugging Face Blog

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt