Deploy decision models with llama.cpp on Amazon SageMaker AI

AWS published a walkthrough on its Machine Learning Blog for deploying decision models with llama.cpp on Amazon SageMaker AI, continuing a series on specialized AWS Deep Learning Containers (DLCs) for generative AI inference. The post describes two deployment paths from one packaged model: a standard text-generation endpoint using the AWS DLC for llama.cpp, and a structured-decision endpoint that swaps the default server for a SemIf adapter, SemIf having formerly been called OpenJev. A companion code sample, covering CPU and GPU options, lives in the aws-samples/sagemaker-genai-hosting-examples repository under 03-features/semif-decision-model.
Decision models target a different job than chat completions. Applications that route requests, select agent actions, or assign policy outcomes often need scores over a fixed set of choices rather than free-form text. In the standard path, the model is packaged as model.tar.gz in Amazon S3 and referenced against a published DLC image directly. The container finds the first .gguf file under /opt/ml/model, maps POST /invocations to llama-server at /v1/chat/completions, and passes SM_LLAMA_CPP_* environment variables through to llama.cpp arguments. Requests follow the OpenAI Chat Completions schema, with a messages array of role and content plus controls such as max_tokens and stream, and generated text is read from choices[0].message.content. No Dockerfile or ECR repository is needed for this path.
The structured-decision path exists because the llama-server HTTP API does not expose the final-position logits and state operations that SemIf uses. That path therefore builds a child image with AWS CodeBuild and stores it in Amazon Elastic Container Registry. The child image keeps the DLC's compiled CPU or CUDA libraries while replacing its entry point with a SageMaker-compatible adapter. Its request schema is constrained rather than conversational: a caller-defined ID, unstructured state, a question, and an ordered list of allowed options. The response returns option_ids, probabilities, and raw option_logits in the same order, plus prompt, timing, and pinned model metadata. Because option IDs are stable, an application can apply thresholds, fallbacks, and policy checks without parsing generated prose. AWS groups SemIf with other decision models that score application-defined outcomes from unstructured state, naming Jev (TypeSafe's decision model and /v1/systemone API), Kev (a model server built for TypeSafe API compatibility), and Strands Decider 2B (a purpose-built model for agent workflows with calibrated outputs).
SageMaker AI images implement the /ping and /invocations container contract, which is why AWS advises selecting a server-sagemaker-cpu-v1 or server-sagemaker-cuda-v1 image rather than the corresponding EC2 image. As an alternative to the S3 model artifact, SM_LLAMA_CPP_HF_REPO and SM_LLAMA_CPP_HF_FILE can pull a GGUF file from Hugging Face when the endpoint has network egress. A CodeBuild step packages and pins the GGUF model and tokenizer, recording source revisions and checksums in model-manifest.json, and halting if the checksum differs from the validated value. The SemIf deployment script rejects mutable image tags, so the image URI must combine the ECR repository URI with an immutable digest. Prerequisites include Python 3.10 or later, Git, the AWS CLI, an S3 bucket in the deployment Region, a purpose-specific CodeBuild service role, an IAM role SageMaker AI can assume, permissions spanning SageMaker AI, ECR, S3, CodeBuild, IAM pass-role actions, and CloudWatch Logs, and quota for a compatible x86 CPU or NVIDIA GPU real-time endpoint.
The post's benchmark applies to the SemIf extension, not the text-generation endpoint, and was run by sending the same 231 public JevBench tasks serially to one CPU endpoint and one GPU endpoint. On CPU with Qwen3.5-4B Q4_K_M on an ml.c6i.8xlarge with 32 threads, using the llama-cpp:server-sagemaker-cpu-v1 image, all 231 tasks returned valid responses, 179 (77.5 percent) were accurate, and client latency was 1.715 seconds at p50 and 19.291 seconds at p95. On GPU with the same model and quantization on an ml.g6e.xlarge with full offload, using llama-cpp:server-sagemaker-cuda-v1, again 231 of 231 were valid, 180 (77.9 percent) were accurate, and latency was 0.352 seconds at p50 and 0.599 seconds at p95. That works out to roughly 4.9 times faster at p50 and 32.2 times faster at p95 for the GPU configuration.
The GPU deployment uses the SageMaker AI instance pools feature to list compatible instance types in priority order so SageMaker can pick an alternative when the first preference is unavailable; the pool ordered g6e, g6, g5, then g4dn. SageMaker selected ml.g6e.xlarge, and startup logs confirmed that llama.cpp offloaded all 34 model layers to an NVIDIA L40S. AWS notes the CPU and GPU images used different llama.cpp builds and diverged on six hard tasks, and advises treating the single-answer accuracy difference as runtime variation rather than evidence that GPU offload improves model quality. Smaller models cut median latency but scored lower: the 2B and 0.8B candidates reached 66.7 percent and 58.4 percent accuracy, so the sample keeps the 4B Q4_K_M model as its quality-first default. The measurements include network latency and are not service guarantees.
Operational guidance in the post separates CPU from GPU by workload shape. CPU suits background, offline, or low-throughput work that is not latency-sensitive and can tolerate seconds of response time; GPU suits interactive decisions, long prompts, or tight tail-latency targets. Keeping state and option descriptions concise, grouping decisions that share the same state into one request, and scaling endpoint instances for concurrency are the listed levers. Shared-state scoring reuses the common prompt prefix, and the adapter returns retryable HTTP 503 responses rather than allowing an unbounded queue around its stateful llama.cpp context. For the CPU SemIf configuration, n_gpu_layers is set to 0 and the adapter serializes scoring around one stateful llama.cpp context; the GPU configuration instead requests full offload with gpu-layers=-1 and a 1200-second provisioning timeout. Instance pools are ordered placement fallbacks, not inference load balancing, and endpoint creation can take longer while SageMaker works through the pool. The example response for a password-reset routing decision returns probabilities of 0.99996, 0.00003, and 0.00001 across account_access, billing, and sales, with raw logits of 28.47, 17.92, and 17.04.
Two cautions run through the guidance. Removing answer-token generation does not remove model inference, since the model still processes the complete state, question, and option descriptions before SemIf reads the final logits; prompts should stay focused and the selected model should be evaluated against labeled decisions from the application before production use. Authorization and deterministic business rules should stay outside the model, and model probabilities should be validated before being treated as operational confidence. Cleanup scripts delete only explicitly named SageMaker resources, leaving CodeBuild projects, the ECR repository, S3 model and source objects, logs, and purpose-specific IAM roles for manual removal.
Why it matters: Teams building routers, triage queues, or agent action selection get a managed serving option for quantized GGUF models with either an OpenAI-compatible chat interface or a probability-returning decision interface, without writing a custom container for the standard path. My reading is that the benchmark makes the CPU-versus-GPU choice largely a tail-latency decision rather than an accuracy one, since the reported accuracy gap is one task and AWS attributes it to differing builds; the serialized single-context adapter also implies throughput scaling comes from adding endpoint instances rather than parallel scoring inside a container, so capacity planning should follow measurement. Anyone adopting the SemIf path should expect the extra CodeBuild and ECR step, digest-pinned images, and a stricter request schema than chat completions.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
Learn how to deploy decision models with llama.cpp on Amazon SageMaker AI. This technical how-to serves OpenAI-compatible chat completions with the AWS Deep Learning Container for llama.cpp, then extends the runtime with SemIf to return probabilities over declared options on CPU or GPU.