AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Pixels to Tokens: Token Space Efficient Active Learning for Low-Budget Semantic Segmentation

Collected Nov 10, 2026

Researchers at the University of Waterloo, including Utku Cicek, Mahip Singh, Ziyao Shang and Sirisha Rambhatla among the listed authors, have published a paper titled "Pixels to Tokens: Token Space Efficient Active Learning for Low-Budget Semantic Segmentation." The work introduces TEAL (Token-space Efficient Active Learning), which the authors describe as the first pixel-level active learning framework built around vision transformer representations. The central claim is that TEAL stays well ahead of prior methods when labels are extremely scarce, improving on the strongest baseline by as much as +24.36 mIoU when only a single labeled pixel per image is available.

Active learning is the practice of letting a model choose which unlabeled examples a human should annotate next, so that a fixed annotation budget buys the most accuracy. Semantic segmentation assigns a class to every pixel, which makes annotation expensive, and most active learning methods for vision transformers work at the patch or region level, asking for whole blocks of labels. The paper frames the pixel-level, low-budget setting as a distinct problem: with as little as one pixel query per image, a method has to both find the most informative locations and train a reliable decoder head from a very sparse signal.

TEAL's design has three parts. First, the ViT backbone stays frozen. Second, selection happens through diversity-based sampling conducted directly on the backbone's native token lattice, which the authors say avoids artifacts introduced when token embeddings are interpolated up to dense pixel resolution. Third, the framework applies a two-level MaxHerding strategy over multi-layer token descriptors to pick representative candidates, followed by margin-based uncertainty refinement on a finer decoder grid. In plain terms, MaxHerding is a selection rule that favors a subset whose average feature resembles the average over the whole candidate pool, so chosen points represent the spread of the data rather than clustering in one area; doing it in two levels and across several layers means the choice reflects both coarse and fine structure in the token features. The uncertainty pass then sharpens the picks at decoder resolution, where the model's confidence is finer-grained.

Evaluation covers CamVid, Cityscapes, ADE20K and Pascal Context, four standard segmentation benchmarks spanning driving scenes and general indoor and outdoor imagery. TEAL is reported to outperform previous baselines consistently under extreme label scarcity, gaining up to +24.36 mIoU at one labeled pixel per image, and to reach up to 97% of fully supervised performance with just 10 pixels per image. Mean intersection-over-union is the usual segmentation score, averaging per-class overlap between predicted and true masks, and fully supervised performance means training with complete dense annotations.

The authors also address whether the gains come from the backbone rather than the selection method. They report that the improvements persist when all baselines are given the same DINOv3 backbone, and argue that pairing token-space selection with uncertainty refinement on the decoder grid is what makes ViT token geometry an effective basis for extremely low-budget active segmentation. That control matters because a stronger feature extractor alone can lift every method; holding it fixed isolates the sampling strategy as the source of the difference.

For practitioners, the appeal is annotation cost: reaching near-supervised results with roughly ten labeled pixels per image instead of dense masks could change the economics of building segmentation models for new domains. A likely trade-off is that the protocol keeps the ViT frozen, so the method's value depends on the quality of the pretrained representation, and the headline numbers describe averages over the listed benchmarks rather than guarantees on an arbitrary dataset. Whether the selected pixels remain informative across rounds of annotation, and how much decoder retraining each round requires, are the practical questions a team adopting this would want to check. The paper's affiliation marks place it in academic rather than product-release territory, so treat the results as research findings.

Why it matters: Teams that need segmentation masks but cannot afford dense labeling are the direct audience, especially those already using pretrained ViTs. If the reported gains hold outside these benchmarks, as few as ten labels per image could make custom segmentation viable in low-budget settings, which in my reading would shift active learning from region-level heuristics toward selection done directly in the token space these models already use.

Read at Apple Machine Learning Research

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

While foundation models and vision transformers (ViTs) have redefined visual representations, current active learning (AL) strategies for these backbones largely operate at the patch or region level, where annotation requirements remain high due to extensive pixel labeling. Extending ViT-based AL to the pixel-level, low-budget regime is uniquely challenging: with as little as one pixel query per image, a method must simultaneously identify the most informative locations and train a reliable decoder head from an ext