AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Making open models work for long-horizon agentic workloads

Collected Nov 16, 2026

Together AI, which builds inference infrastructure, and Ollama, which builds harness integrations, published a joint technical write-up arguing that long-running coding agents live or die on prefix caching, and that both the client side and the serving side have to cooperate to preserve it. The two companies describe themselves as working from opposite ends of the same handoff: Ollama decides what goes into each request, Together AI decides what happens when that request arrives. The stated motivation is that much of each team's work involves protecting the cache from things the other side can do to it.

The traffic pattern is the starting point. Each time an agent takes a step, the harness resends the model the entire conversation so far: system prompt, every tool definition, the user request, every file read and every command run. What comes back is typically a tool call a few dozen tokens long. This large input, small output shape is characteristic of agentic workloads, and if the GPU reprocessed the full transcript each turn, sessions would get slower and pricier as they grow.

Requests move through two phases. Prefill reads the whole prompt in one parallel pass and produces the first token; it is compute-heavy and determines how long you wait before anything appears. Decode writes the answer one token at a time, each token attending back over everything before it; it is limited by memory bandwidth and determines streaming speed. An agent turn is mostly prefill, and reasoning models add thinking tokens on top while the input still dominates as sessions reach tens of thousands of tokens.

The cache economics are simple to state. Prefix caching keeps the intermediate key-value pairs the engine computed for earlier tokens so that turn 7, which is turn 6 plus a little, only costs the new part. The authors illustrate with idealized arithmetic: ten equal-sized turns process 55 units of prompt without caching and 10 with it; at a hundred turns it is 5,050 versus 100. Every new token still has to attend over cached ones, so a hit is cheap rather than free, but the redo work otherwise grows with the square of session length.

The governing rule is that a prefix cache matches from the start of the prompt up to the first differing token and stops there. Appending to the end is free; changing even one character earlier forces reprocessing of everything after it, and a change near the top effectively discards the session. Everything else in the write-up follows from that constraint.

On the harness side, Ollama points to its ollama launch command, which connects tools such as Claude Code, Codex and Pi to open models like DeepSeek-V4.1-Flash and GLM-5.3 while keeping existing workflows. The authors note that connecting a harness to an API involves many translation layers and mismatched expectations, which they identify as a common cause of cache breaks, unexplained agent behavior and outright errors.

Late tool loading is one trap. Tool search lets an agent pull in a definition mid-task, but placing it in the tool list near the top of the prompt invalidates everything below, potentially hundreds of thousands of tokens on a long session. Appending the definition at the end is better, though some chat templates do not expect a tool definition mid-conversation and will error, so whether a harness can do this cheaply depends on the model.

Volatile content at the top is the most direct way to wreck a cache: a timestamp, git status or moving working directory re-rendered into the system prompt each turn forces a full re-read. Session-fixed values like a user name or start date do not hurt the current session but prevent new sessions from reusing a cached setup. The recommended fix is to put such information once into the task message and keep the system prompt identical.

The authors describe hitting this themselves with reasoning effort. In GLM-5.3-Flash, effort is not a side setting handled by the engine; the chat template writes it into the first system block. Raising effort from low to high mid-session therefore changes the prompt almost at the first token and reprocesses the whole session. Their advice is to choose an effort level up front, and if more thinking is needed, start fresh at the higher setting or compact first.

Oversized tool results are another cost they flag. A 3,000-line file read when forty lines would do is carried, and paid for in context, on every remaining turn. Truncation is the right call, but silently returning the first 100 lines of a 640-line file can leave a model editing code it never saw; a one-line notice of the range and total lets the model search or request another slice.

Subagents have a cache trade-off too. Inheriting the parent's full prompt, tools and history, as side chats do, means the whole prefix is already cached, but cached tokens are still not free: each generated token attends over them and they hold GPU memory. A reviewer that needs two files and a checklist may not need hundreds of turns of history, and a subagent with its own system prompt shares nothing at all.

On the engine side, the cache must still exist and the next request must reach the machine holding it. Together AI frames an inference engine as a machine for keeping GPUs busy, citing roughly $50k per GPU per year in rental cost and hundreds of concurrent users at different points in their requests. Routing first picks a data center that runs the model, has capacity and has hosted the session, then a node inside it holding the prior turns. One method is a client-supplied key such as prompt_cache_key pinned to a node; the other hashes the prefix and targets the node with the longest matching cached prefix, even from another session, which is why identical system prompts and tool definitions across sessions can give new sessions a warm start.

Always choosing the warm node is not automatic: if it is overloaded, load-aware routing may prefer an idle node and a recompute. Keeping caches alive between turns is the second half. Agent tool calls can idle for minutes, so caches can be offloaded to CPU memory and restored later. Memory is handed out in small blocks as needed via PagedAttention, and the running batch is rebuilt each pass via continuous batching so parallel subagents slot in as capacity frees. The authors' takeaway for harness builders is that cache memory headroom, not compute, usually caps concurrent requests per GPU. They also run prefill on compute-tuned clusters and decode on memory-bandwidth-tuned clusters, shipping the cache between them over RDMA.

The closing summary walks one session across both sides: the calendar tool is appended at the end, the truncated file read announces its range, effort stays put, a reviewer subagent inherits the session, each turn returns to the cached node, and the cache waits in CPU memory during test runs. The user simply sees the agent stay fast at turn ninety.

Why it matters: teams building coding agents with open models now have a concrete checklist spanning prompt construction, tool handling, subagent design and request routing. My reading is that many slowdowns and cost spikes attributed to model quality may be cache misses instead, so instrumentation for hit rate is likely to become as routine as latency and token accounting. The split responsibility also means harness and serving teams need shared expectations, since either side alone can quietly negate the other's work.

Read at Together AI

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

Every agent turn resends the full session. Together AI and Ollama break down how to protect the prefix cache, from the harness to the GPU.