AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Building Long-Running Agents with ScaleDown and Deep Agents

Collected Oct 13, 2026

LangChain has published a langchain-scaledown integration that plugs ScaleDown's constrained decoding models (CDMs) into LangChain and Deep Agents, and ScaleDown is now a direct model access provider in the LangSmith LLM Gateway at the /scaledown path. The integration ships with middlewares for summarization, tool result compression, and structured extraction. The announcement is framed around context management for long-running agents, where deciding what stays in the context window happens on nearly every turn and is slow and expensive when done with a frontier LLM.

The core idea behind a CDM is narrowness. A general-purpose LLM can emit any string token by token, while a CDM is trained for a single task and returns only answers that task permits, such as a label from a supplied list or a span copied from the input. That constraint lets the model be much smaller, faster, and cheaper. ScaleDown reports its models are 200x faster and 467x cheaper than frontier models on these narrow tasks.

ScaleDown offers four models, each trained for one task. Summarize writes a short summary of long text. Compress reduces a block of context to the parts relevant to a question. Extract pulls out fields described in plain language, returning each with a confidence score and its position in the source. Classify scores text against user-defined labels and returns a probability per label, with an option to return a natural-language reason for the decision. All four accept up to 1M tokens of input, so a full contract, a long policy page, or an hour of agent history fits in a single call without chunking. Extract, Summarize, and Classify are also multimodal and accept images and scanned PDFs alongside unstructured text.

On ScaleDown's own benchmarks, measured against the average of GPT-5.5, GPT-5.4 Mini, and GPT-5.4 Nano on public datasets, Summarize is 41x cheaper and 2.7x faster while scoring higher on accuracy, and Compress is 138x cheaper and 1.8x faster, also with higher accuracy. Readers should treat vendor-run benchmarks as directional rather than independent evidence, since the comparison models, datasets, and task framing are chosen by the vendor.

The summarization middleware, ScaledownSummarizationMiddleware, replaces Deep Agents' built-in summarization middleware and reuses its trigger and keep options to decide when to summarize and which recent messages to retain. The agent keeps its own frontier model for planning and writing code; only the summary moves to ScaleDown, which by default returns the same sections as LangChain's summary prompt: intent, summary, artifacts, and next steps. The trigger and keep options accept fraction-based values, so an agent can be configured to summarize when history crosses 80 percent of the context window and keep the most recent 10 percent. If a ScaleDown call fails, the middleware leaves history unchanged and the agent keeps running. The same middleware also works with LangChain's create_agent.

Tool result compression addresses a familiar cost problem: a fetched web page can run to tens of thousands of characters when the agent needs one paragraph, and every later step pays for those tokens again while the context fills up. ScaledownToolCompressionMiddleware adds an optional compression_query parameter to tools, which the model fills in on the tool call when it only needs part of a result, for example a refund policy question scoped to EU orders. The middleware strips that parameter before the tool executes, then compresses the output to the relevant parts before the model sees it. Tools themselves don't change, so this works with any tool including MCP tools. In one trace, a long docs page was cut by almost 90 percent before reaching the model, from 18,402 to 3,110 characters. The compressed result carries a note telling the model it can call the tool again without compression_query to get the full output, which preserves an escape hatch when the compression was too aggressive.

ScaledownExtractionMiddleware handles values an agent must copy exactly rather than paraphrase, such as an order number, an error code, or an email address. Fields are described in plain language, and Extract copies values from the text. A field can also be defined as a set of labels, which ScaleDown answers with Classify. Before the first model call of a run, the middleware sends the user's messages to Extract and stores fields in agent state under scaledown_extraction, where tools can read them. The example parses a message containing an order ID, a payment error code, and a retry note into order_id A-1042, error_message ERR_PAYMENT_DECLINED, customer_email None, and issue_type billing. Extract only reads user messages and never the agent's replies, so every value is something the customer actually said. With a checkpointer, extraction re-runs each turn, so the record fills in as the customer adds detail. A context_chars setting can be set to zero to skip evidence snippets and keep state small.

There is no prebuilt middleware for drift detection, but the post shows a custom circuit breaker built with the after_model hook. It sends the message history to ScaleDown Classify with labels for progressing, looping, and drifting, each with a rubric, and stops the run when the top label is not progressing and its score exceeds a chosen threshold such as 0.8. The breaker returns the classifier's one-sentence reason in the final message so the stop is explainable. A gentler variant adds the reason to the conversation as a nudge and only halts if the agent drifts again. This pattern is a good illustration of where CDMs fit: high-frequency, low-ambiguity decisions that gate agent behavior.

For observability, an admin adds a ScaleDown key to the workspace's provider secrets as SCALEDOWN_API_KEY once, then points langchain-scaledown at the gateway base URL and authenticates with a LangSmith key. Each ScaleDown call then appears as a LangSmith trace and counts toward the same spend and rate limits as other model calls. Installation is a pip-style update of langchain-scaledown plus setting the API key, and new ScaleDown accounts get 50M free tokens according to the post.

The guidance on when not to use these models is explicit: keep a general-purpose LLM for open-ended generation such as writing code, drafting replies, and planning, and use specialized models for the high-volume steps around it. That mirrors the cheap-by-default, frontier-on-exception pattern LangChain has promoted in its coverage of decision models like Jev. Compared with Jev, ScaleDown Classify takes text only in Jev's case up to 32K tokens of state per question, while Classify accepts up to 1M tokens including images and adds a one-sentence reason per decision. The stated advantage of the reason is debuggability: when an agent routes a ticket or halts a run on a label, the reason explains the choice.

A likely trade-off is operational: adopting these middlewares introduces another provider, API key, and failure mode into the agent loop, though the graceful fallback on summarization failures mitigates the worst case. Teams should also verify that narrow-task accuracy holds on their own data, since a compression or extraction miss can silently remove context the agent later needs. The escape hatch on compression and the source-position metadata on extraction are the main guardrails against that.

Why it matters: teams building long-running agents are usually bottlenecked on context management cost and latency rather than on raw model capability, so moving per-turn summarization, compression, extraction, and classification to specialized models can cut spend meaningfully while leaving planning and code generation on a frontier model. Developers can adopt the middlewares incrementally, starting with tool compression or extraction, and get tracing and budget enforcement through the LangSmith gateway without building separate instrumentation. The main work for product teams is deciding which decisions are narrow enough to delegate and validating accuracy on their own traffic.

Read at LangChain

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

Use ScaleDown's constrained decoding models with LangChain and Deep Agents to summarize history, compress tool results, and extract data faster and cheaper.