Faster Retrieval for Better, More Efficient RAG

Apple machine learning researchers presented a paper at ICML 2026 titled Amortizing Maximum Inner Product Search with Learned Support Functions. The work targets maximum inner product search (MIPS), the subroutine underneath recommendation systems, search and retrieval-augmented generation (RAG), where a query vector is matched against millions of candidate key vectors in a database. The authors' core claim is that standard approximate methods ignore what queries actually look like, and that training small networks on a realistic query distribution yields faster and more accurate retrieval for production systems whose query patterns are stable.
MIPS matters in RAG because the best-matching key corresponds to a document that is then prepended to the user's question. As key counts reach millions and embedding dimensions reach thousands, exhaustively comparing a query to every key becomes prohibitively expensive. The established shortcut splits keys into clusters, keeps a centroid per cluster, compares the query against those centroids and then scans only the most promising clusters, which cuts work by roughly the number of clusters. The Apple team points out that centroid routing is approximate: a cluster can contain a key far from its own centroid but close to the query, so a centroid comparison scores that cluster low and skips it, missing the true best match.
The theoretical hook is that the function mapping a query to its largest inner product with any key in a set is a support function, a well-studied object in convex analysis. It has two useful properties: it is convex, so it can be approximated by input convex neural networks (ICNNs), and its gradient at a query returns the best-matching key. The researchers train networks by simple regression: sample queries from the real distribution, compute their exact best matches once by exhaustive search, and fit the networks to reproduce those outputs. The up-front cost is paid once per fixed key database, and inference then becomes cheap.
Two variants come out of this. SupportNet learns the support function directly with a loosely constrained ICNN and can produce one score per cluster, effectively answering how good the best key inside a cluster is for this query. That enables routing based on exact match quality rather than centroid distance. KeyNet instead learns to output the best-matching key vector itself; a consistency loss derived from Euler's theorem for homogeneous functions keeps its predictions geometrically tied to the underlying support function, though convexity is not enforced. Experiments found that specialized networks trained separately for each task outperformed a single network evaluated for both its value and its gradient.
On Natural Questions, with roughly 2.6 million unique passages split into 128 clusters, SupportNets of four sizes (XS, S, M and L, corresponding to 1%, 5%, 10% and 20% of database size, all 16 layers deep) were compared against centroid routing. Counting the cost of running the networks, the small SupportNet reached 95% routing recall about 1.9x faster (1.74 ms versus 3.36 ms), and the large model hit 97% in 1.74 ms. Looking purely at routing quality with a fixed scan budget, scanning 5% of the database, the learned routers found the true cluster roughly 90 to 93% of the time, peaking at 92.6% for the mid-size model, against 87.5% for centroids.
KeyNet was evaluated as a query rewriter in front of a standard FAISS inverted file (IVF) index, leaving the index itself untouched. The network maps the incoming query to a predicted region of embedding space, and the index is asked to search there. Reaching 99% recall took up to about 8x less wall-clock time with the largest model (0.23 ms versus 1.8 ms for the raw query). On quality alone, scanning just 0.001% of the database, KeyNet-mapped queries retrieved the true key 89 to 97% of the time (96.7% for the largest) versus 86% for raw queries, and 99% recall took about 11x fewer keys scanned (0.009% versus 0.10%). The authors report the trend held across several datasets.
The guidance for practitioners is a practical split. KeyNet suits latency-bound settings where an approximate best key is enough. SupportNet suits cases needing a calibrated score for re-ranking or thresholding, or where compute is tight at small model sizes. Both are reported to be reasonably robust to mild shifts in query patterns. For very large key sets, the paper suggests combining them: a SupportNet router over all clusters, plus a KeyNet per cluster to accelerate the regular index. My reading is that the approach's main dependency is having a representative sample of real queries before deployment, which fits steady production traffic but is harder for systems facing cold start or rapidly changing query behavior.
Why it matters: teams running RAG, search or recommendation at scale can treat retrieval latency and recall as a tuning problem tied to their own traffic rather than a fixed index trade-off, since the learned routers plug into existing IVF infrastructure. The reported latency and scan-budget gains, if they hold on a team's own data, could make larger or more accurate retrieval affordable at the same cost. The trade-off is an offline training and refresh pipeline matched to a query distribution, which adds operational work that query-agnostic indexes avoid.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
Maximum inner product search (MIPS) sits at the core of recommendation systems, search, and retrieval-augmented generation (RAG). Given a query vector, MIPS systems search within a database of millions of candidate key vectors for the one that best matches the query. The original data point associated with that best-matching key is then used to handle the query: in RAG, the document that is best suited to answer the question is typically prefixed to the query. As the number of keys grows to millions and vector dime