Building a self-service LLM gateway with Amazon Bedrock and LiteLLM

AWS published a walkthrough on its Machine Learning Blog for building a self-service LLM gateway that fronts Amazon Bedrock with LiteLLM, so developers point coding assistants at one OpenAI-compatible endpoint instead of touching Bedrock directly. The design runs LiteLLM as a proxy on AWS Fargate behind Route 53, CloudFront and an Application Load Balancer, with each developer authenticating using a personal API key. The stated goals are per-user cost control, consistent access policy and visibility into model behaviour — concerns the post attributes to platform, finance and security teams respectively.
In the described topology, the gateway keeps virtual-key and spend state in Amazon RDS for PostgreSQL, uses ElastiCache for Valkey as a response cache and for coordinating rate limits across Fargate tasks, and pulls secrets such as the master key and database URL from AWS Secrets Manager. CloudWatch captures requests per minute and tokens per minute plus a quota alarm. Langfuse, an open source LLM observability tool, is added through LiteLLM logging callbacks for prompt-level traces, latency and debugging. The examples use Claude Sonnet and Claude Haiku models; the post notes it omits Bedrock Guardrails such as content filtering and grounding, which it says would normally be part of a production deployment.
A key detail for anyone copying the pattern: LiteLLM maps a friendly model name to a Bedrock Application Inference Profile ARN, created with AWS CDK. The post warns that when the model field is a raw inference-profile ARN, LiteLLM cannot find it in its built-in cost map and treats the cost as zero — so per-key spend stays at zero and budgets never enforce. Explicit input and output per-token costs must therefore be set in each model's configuration. Example values are given for Sonnet and Haiku, along with sample limits of a $250 budget over a rolling seven-day window and a maximum of ten parallel requests per key.
Cross-Region inference is handled by the inference profiles. Global cross-Region inference is recommended where model support and organisational policy allow processing across supported AWS commercial Regions, because it reaches a wider capacity pool and may carry pricing benefits versus Geographic profiles, which confine processing to a defined geography such as the US. The CDK example must copy from the system-defined profile ARN carrying the global. prefix; pointing at a foundation-model ARN instead produces a single-Region profile. The IAM policy needs the application profile ARN plus the regional and Region-wildcarded foundation-model ARNs, otherwise invocations fail with a not-authorized error when Bedrock routes elsewhere. That keeps permissions scoped to the specific models rather than bedrock:* or *.
Licensing is a practical constraint. LiteLLM documents the AI Gateway as free for up to five users, with a licence required beyond that. The Enterprise tier adds single sign-on, audit logs, delegated admin roles and support. Virtual keys exist in the free tier, but the self-service key issuance the post describes requires Enterprise. A reader planning a pilot should treat SSO-bound key generation as a paid capability.
Cost enforcement surfaces as an HTTP 429 with a message saying the budget was exceeded rather than silent throttling or an unexpected bill. Teams can retrieve daily spend, prompt and completion tokens and request counts broken down by model, provider and key via GET /user/daily/activity with a date range, or per-user totals via GET /user/info, or browse the Admin UI's Usage tab. A scheduled job against those endpoints is enough to feed a chargeback dashboard. Developers generate their own SSO-bound key from the Virtual Keys page and can regenerate it after exposure or on a rotation schedule.
Monitoring advice departs from conventional practice: the post argues latency percentiles mislead for extended-thinking models whose requests can take minutes, so RPM and TPM are the primary signals. A CloudWatch alarm fires at roughly 60 percent of the account's TPM quota over three one-minute periods — the example threshold is 600,000 tokens — giving lead time to request a quota increase before throttling. LiteLLM has no built-in publisher for such a metric, so a small custom logger using PutMetricData supplies it; it must be registered through success_callback rather than callbacks, because the latter replaces LiteLLM's internal logger list including spend tracking and would silently disable budget enforcement. Timeouts in the example are 900 seconds for Sonnet, to accommodate extended thinking, and 120 seconds for Haiku.
Why it matters: teams standardising on coding assistants usually hit the same three problems — uncontrolled spend, no per-team attribution and keys scattered across tools — and this pattern centralises all of them behind one endpoint that Claude Code, Continue, Cursor and similar clients can consume unchanged. The main trade-offs are an Enterprise licence for the self-service experience and the need to maintain per-model prices so budgets actually bind; in practice, that manual cost table is the part most likely to drift as Bedrock pricing or model versions change.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
Build a self-service LLM gateway with LiteLLM on AWS Fargate that puts Amazon Bedrock foundation models behind an OpenAI-compatible API. Developers get instant access through a single API key, while platform teams gain per-user cost controls, quota isolation, and observability.