LangSmith LLM Gateway: Runtime Controls for Agents

LangSmith LLM Gateway is now available in public beta for Plus and Enterprise plans, according to an announcement from LangChain. The gateway is a central governance layer between agents and model providers, designed to enforce runtime controls for production agents.

The gateway provides capabilities to set time-bound spend caps at organization, workspace, API key, and user levels. When a cap is reached, the agent receives a 402 response with a clear error. Rate limits can be applied at the same levels to prevent traffic spikes from overwhelming provider limits. Fallback rules can be defined across models and hosts so that when a provider is down, rate-limited, or unavailable, agents can route to another model without custom logic in each application.

Sensitive data protection includes detecting, redacting, and replacing PII and secrets in requests before they reach model providers. The gateway also redacts sensitive data from traces it records in LangSmith. This feature is currently available only to Enterprise plan users.

The gateway supports popular providers such as OpenAI, Anthropic, and Fireworks, as well as custom models with OpenAI- or Anthropic-compatible endpoints. It is BYOK-first, allowing teams to bring their own provider keys for routing, governance, and policy controls without adding instrumentation to every LLM call. For simpler procurement, hosted inference for open models powered by Fireworks is available directly through LangSmith using Gateway Credits.

Gateway events are logged as metadata in LangSmith traces, so teams can monitor how controls behave with production traffic and inspect blocked requests in context.

Teams can call LLM Gateway directly via API or use it with coding harnesses like Claude Code, Codex, and Deep Agents Code (dcode). Setup involves pointing agents at the LangSmith Gateway endpoint, authenticating with a LangSmith API key, adding provider API keys to workspace secrets, and configuring controls for fallbacks, rate limits, spend limits, and sensitive data handling.

Future plans include broader guardrails for risks from agents calling external models and interacting with production systems, as well as CI/CD controls for deploying prompts and context, running A/B tests, managing blue-green and shadow deployments, and defining controls with Infrastructure-as-Code.