AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Open Models have crossed a threshold

Collected Oct 13, 2026

LangChain has published findings from a multi-week evaluation of open-weight large language models run through its Deep Agents harness, concluding that GLM-5 from z.ai and MiniMax M2.7 now perform comparably to closed frontier models on the core tasks that gate agent usability: file operations, tool use, instruction following, retrieval, conversation, memory and summarization. The company says tool calling is reliable and instruction following consistent, and that open models are now a viable option to run instead of, or alongside, closed frontier models in production agent workflows.

The evaluation suite spans seven categories and is run across 52 models periodically in CI, using hosted inference providers, with each eval case split into success assertions (hard-fail correctness checks) and efficiency assertions (soft checks on how the model reached the answer). Four metrics are reported: correctness, the fraction of test cases solved; solve rate, a combined accuracy-and-speed measure computed as expected steps divided by wall-clock seconds, with failed tests scoring zero; step ratio, actual agent steps divided by expected steps; and tool call ratio, the same comparison counting individual tool calls rather than steps. A step ratio of 1.0 means the model used exactly the expected number of steps, while a tool call ratio of 1.0 is on-budget. The efficiency metrics do not affect whether a test passes but indicate how economically a model reaches an answer.

Cost is the headline driver. LangChain cites an example application producing 10 million tokens per day: roughly $250 per day on Opus 4.6 versus about $12 per day on MiniMax M2.7, a gap of about $87,000 annually. It also notes that closed frontier models can run 8–10x more expensive for high-throughput workloads and are often too slow for interactive response times. On latency, OpenRouter data cited in the post shows GLM-5 on Baseten averaging 0.65 seconds and 70 tokens per second, compared with 2.56 seconds and 34 tokens per second for Claude Opus 4.6. LangChain points to Groq, Fireworks and Baseten as providers optimizing inference for latency and throughput.

For developers, switching is a one-line change. With the Deep Agents SDK, a GLM-5 agent is created via create_deep_agent(model="baseten:zai-org/GLM-5") after installing langchain-baseten, and a MiniMax M2.7 agent via create_deep_agent(model="openrouter:minimax/minimax-m2.7") after installing langchain-openrouter. The harness detects the model's context window, disables unsupported modalities, and injects the model's identity into the system prompt. The same model can be reached through multiple providers — GLM-5 as baseten:zai-org/GLM-5, fireworks:fireworks/glm-5, or ollama:glm-5 for self-hosted — with the same harness. Providers tested for this release are Baseten, Fireworks, Groq, OpenRouter and Ollama (cloud).

The harness absorbs differences between open and closed models that would otherwise fall to the developer. Model identity injection patches the system prompt at runtime with the model's name, provider, context limit and supported modalities. Context management adapts compression, offloading and summarization thresholds to the model's actual context window rather than a hardcoded default — LangChain's example is that a model with a 4K context gets more aggressive compaction than Opus with 1M. These adaptations matter because open models arrive with different context sizes, tool-calling formats and failure modes.

The same models are available in the open-source Deep Agents CLI, which LangChain describes as a coding agent and alternative to Claude Code. A new middleware, ConfigurableModelMiddleware, enables runtime model swapping via a /model slash command without restarting the agent. LangChain highlights the pattern of starting a task with a frontier model for planning and switching to a cheaper open model for execution, and says it plans to document harness tuning patterns for specific open model families and test multi-model subagent configurations such as a closed frontier orchestrator with open model subagents.

LangChain is careful to call these early results, noting it is actively maintaining and expanding the eval set, with recent runs viewable in a GitHub repo and a shared LangSmith project. Evals run against model groups, so pytest tests/evals --model-group open targets all open models and pytest tests/evals --model baseten:zai-org/GLM-5 targets a single one. For the frontier comparison, LangChain uses each provider's default thinking level: high for Gemini 3+, medium for OpenAI, and no extended thinking for Claude. That detail matters, since reasoning effort can shift results on agentic benchmarks; a like-for-like comparison would need matched reasoning budgets, and treating a default-thinking frontier model as the ceiling is one reason these numbers should be read as directional rather than definitive. Deep Agents can also run fully local and private models via Ollama or vLLM.

A practical consideration for teams weighing the switch: open models vary by provider, so throughput, quantization and context limits can differ even for the same weights, and latency/price figures depend on the serving stack chosen. The harness-level adjustments LangChain describes — identity injection, adaptive context management — are the kind of plumbing teams would otherwise build themselves, which is part of why the one-line model swap is the more interesting claim for product teams than any single benchmark number.

Why it matters: teams building production agents now have a documented path to run open-weight models on core tool-use and file-operation tasks through a mainstream harness, which could materially cut inference spend and improve interactive latency where the cited cost and speed gaps hold. The one-line model swap and runtime swapping mean model choice becomes a configuration decision rather than an architectural one, and the ability to mix a frontier planner with open-model executors is a pattern worth testing. As with any early eval set, practitioners should validate these results against their own tasks before treating them as a procurement decision.

Read at LangChain

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

Open models like GLM-5 and MiniMax M2.7 now match closed frontier models on core agent tasks — file operations, tool use, and instruction following — at a fraction of the cost and latency. Here's what our evals show and how to start using them in Deep Agents.