Multi-Vector (Late Interaction) Embedding Models with Sentence Transformers

Sentence Transformers v6.0 adds a fourth model type, MultiVectorEncoder, for ColBERT-style late interaction retrieval, according to a Hugging Face blog post. The Python library previously covered dense and sparse embedding and reranker models. The post says any PyLate checkpoint and any Stanford-NLP ColBERT checkpoint loads straight into the new type, and that colpali-engine models for visual document retrieval can also be used through the same API.

The post explains the distinction from regular embedding models: instead of compressing a whole text into one vector, a multi-vector model keeps one vector per token and scores queries against documents with the MaxSim operator, preserving token-level matching information. It states this usually means stronger retrieval at the cost of a bigger index, and describes multi-vector models as state of the art for visual document retrieval, where a text query is matched against page images without an OCR step.

The post covers loading checkpoint formats, inspecting checkpoint configuration, encoding and scoring, search stacks, page images, and index affordability. It says ColPali-family checkpoints ship in colpali-engine's own format, which carries no information Sentence Transformers can use, so each needs a small configuration added before it loads, and that most of that work is done and waiting to be merged.

Installation is described as a plain pip install -U sentence-transformers, with an image extra for ColPali-style retrieval. The post states Sentence Transformers v6.0 requires transformers v5.x, torch 2.2+, and huggingface-hub v1.x, and refers readers to a Migration Guide for breaking changes. It also says LightOn built PyLate on top of Sentence Transformers to handle late interaction, and that with v6.0 those capabilities live in Sentence Transformers itself. A companion post covers training and finetuning multi-vector embedding models.