Geek at Work Blog

The Inference Gateway: Routing, Caching, and Cost Control for Production LLM Traffic

Wed, 28 Oct 2026 00:00:00 Z

An LLM endpoint looks like a URL. In production, it is a policy surface. The first version of an AI feature calls a provider SDK directly, with an API key in an environment variable. That works until a second model, provider, team, fallback requirement, or finance question arrives. Then every application reimplements the same routing, retries, quotas, caching, and logging — each one differently.

The missing component is not another model server. It is the gateway in front of the models. This article covers that layer: routing, cache correctness, failure handling, and token-level cost control. The GPU and serving layers below it are covered in the previous article on inference infrastructure; here we stay above the model endpoint.

# Two Gateways, One Traffic Path

The industry uses two names for adjacent things, and conflating them causes most of the confusion:

An AI gateway is the entry point for LLM traffic. It owns identity, keys, virtual model names, provider routing, quotas, caching, guardrails, and cost attribution. It fronts managed model APIs and, often, self-hosted models.
An infrastructure inference gateway is a Kubernetes data-plane component. It routes requests to pods in an InferencePool using inference-aware signals: queue depth, KV-cache utilization, prefix-cache state, and active LoRA adapters. It does not manage provider keys or budgets.

They solve different halves of the same problem:

Concern	AI gateway	Infrastructure inference gateway
Identity, API keys, team budgets	Yes	No
Virtual model names and provider routing	Yes	Model-aware routing inside a pool
Cross-provider fallback	Yes	Endpoint fallback within a pool
Token accounting and cost attribution	Yes	Raw serving metrics only
Queue-depth and KV-cache-aware routing	Delegates or passes through	Yes

The split is logical, not necessarily physical. Agent Router (formerly Envoy AI Gateway) spans both: it routes across providers and supports InferencePool backends. LiteLLM, Kong AI Gateway, and similar products occupy the AI gateway layer; the Kubernetes Gateway API Inference Extension defines the inference layer.

The AI gateway fronts provider APIs and the inference gateway for your own pools.

If you have one model, one team, and no cost pressure, skip the gateway; it adds a hop and a failure domain. Once you need centralized credentials, per-team quotas, provider fallback, or a defensible answer to “who spent this money?”, it earns its place.

# Routing That Understands Inference

The first job of the gateway is to stop applications from naming models. A client asks for chat-default, not provider-x/model-y-2026-09-14; the gateway maps the alias to a concrete deployment without a client release. Version the alias (for example, chat-default-2026-10) so rollback is a routing change, not an incident.

Routing policy should use attributes the application actually knows:

Tenant and tier. Free users can get a smaller model; paid users the primary one.
Workload class. Interactive chat is latency-sensitive; batch summarization is not.
Context size. A 200k-token request should not land on a 32k-window model or on a replica with a full KV cache.
Region and data residency. An EU tenant should not silently route to a US deployment.
Capabilities. Tool calling, JSON mode, vision, and structured output are not interchangeable across models.
Cost ceiling. A team budget can exclude frontier models.

Inside a self-hosted pool, round-robin is blind to request cost: ten long-context requests can load a replica more than forty short ones. Least-connections is better but still treats requests as equal. The Kubernetes inference extension adds an Endpoint Picker (EPP) that reads model-server metrics and chooses an endpoint per request. vLLM exposes most of these signals:

vllm:num_requests_running — current load per replica
vllm:kv_cache_usage_perc — how close the replica is to memory pressure
vllm:prefix_cache_queries and vllm:prefix_cache_hits — whether the replica has a warm prefix cache
active LoRA adapters — whether the replica can serve the requested fine-tune

Prefix-cache affinity is the highest-leverage example. A multi-turn conversation resends the same system prompt and history every turn. If the next request lands on a replica that already holds that prefix in KV cache, prefill is largely skipped and time to first token drops. Strict session affinity has a failure mode, though: one popular conversation can pin itself to one replica while others idle. The useful policy is affinity with a load guard — prefer the warm replica until queue or KV-cache pressure crosses a threshold, then move.

An InferencePool can declare what happens when the endpoint picker is unavailable. With failureMode: FailOpen, the gateway picks an endpoint itself rather than failing the request. That favors availability, but requests may land on a cold or overloaded replica. Fail-closed protects latency SLOs and hurts uptime; make the trade-off explicit.

# Caching Without Lying to Users

Caching saves the most money and causes the most subtle incidents. Three caches get mixed together in vendor decks:

Cache type	What is stored	Good for	Breaks when
Exact response cache	Hash of request, model version, parameters, tenant	Identical prompts, classification, evals	Chat and agent loops, where every request differs
Provider prompt cache	Provider-side prefix/KV state	Long system prompts, RAG context, multi-turn chat	Prefix changes, or traffic moves provider/region
Semantic cache	Prompt embedding plus a stored response	Single-shot FAQ and support traffic	Multi-turn or agentic traffic, personalized answers, cross-tenant reuse

Exact response caching is the safest. The key must include the model version, every sampling parameter that affects output, tool definitions, response format, and tenant. Miss one and you will serve an answer generated under different conditions. A TTL and an invalidation path for model or prompt releases are mandatory. Hit rates are low for chat and high for support macros, classification, and test suites; cache the workloads where it works.

Provider prompt caching is not a response cache. The gateway passes the request through, and the provider reuses prefill state for a stable prefix. Savings show up as cache-read tokens and lower time to first token. The rule: keep the prefix byte-stable. Do not inject a timestamp, request ID, or per-user greeting into the system prompt if you want a hit. Keep sessions on the same provider and region when the cache is regional. Account for cache-read and cache-write tokens separately; they are priced differently from ordinary input.

Semantic caching is the dangerous one. It embeds the prompt, finds the nearest cached prompt above a threshold, and returns the stored response. For single-shot FAQ it can work. For multi-turn or agentic traffic it is a correctness bug: each turn is the previous request plus a small delta, so consecutive turns are nearly identical in embedding space. LiteLLM’s documentation warns that semantic caching “will replay stale responses” on multi-turn or agentic traffic and recommends exact-match caching instead. The embedding call also runs inline, so a slow embedding endpoint becomes request latency.

Tenant isolation is the other trap. Kong’s AI Semantic Cache documentation warns that the policy does not use request headers to separate cached responses and recommends configuring it per AI consumer rather than globally. LiteLLM’s semantic_cache_scope: end_user setting exists for the same reason: by default, every end user behind one virtual key shares a semantic bucket. A cached answer for one customer can be served to another who asks something similar. A global semantic cache is a data leak with a hit-rate dashboard.

The rule: cache hit rate is a cost metric, not a quality metric. A 40% hit rate with wrong answers is worse than no cache. Gate semantic caching behind an allowlist of intents, scope it by tenant and end user, keep TTLs short, support a bypass header, and evaluate cached answers like generated ones. When in doubt, use exact-match caching and leave semantic caching off.

# Failure Handling and Quotas

A gateway makes reliability policy explicit. SDK defaults are optimistic: retry a few times, wait, hope the provider recovers. At scale, those defaults compose badly.

Fallback is a product decision. A chain usually goes: same model in another region or provider, then a smaller model from the same family, then a static degraded response. If a paid user silently gets the small model, that is a product change. Return the actual model in response metadata (for example, an x-model-used header) and decide per feature whether a downgrade is acceptable. Test fallbacks for tool-call shape, JSON validity, context window, and refusal behavior.

Retries need one owner. If the SDK makes three attempts and the gateway makes three, one request can become nine upstream calls. Pick one layer — usually the gateway — and set the other to zero. Retry only transient failures: 429, 5xx, connection errors, and timeouts before any output. Never retry 400, 401, 403, context-length errors, or content-policy rejections. Add jitter and a retry budget; a retry storm during a provider incident turns a partial outage into a full one.

Timeouts should be split by phase. Connect, time to first token, idle between streamed chunks, and total are different controls. A single 30-second timeout kills legitimate long generations; a missing idle timeout holds connections open after a client disappears. Set total timeout above the upstream p99 but below client patience, and make sure ingress and load balancers do not time out streams earlier than the gateway.

Rate limits should use tokens and concurrency, not just requests. A 200k-token prompt is not equivalent to a 200-token prompt, but both count as one request per minute. Reserve estimated input plus max_tokens before dispatch, reconcile against actual usage, and release the difference. A 32k-token prompt with max_tokens: 8000 reserves 40k tokens; at a 200k-token-per-minute team quota, only five such requests can be in flight. Pair this with concurrency limits so a burst cannot exhaust provider quotas or gateway memory, and return 429 with Retry-After instead of queueing without bound.

Gateway quotas do not replace provider quotas. Accepting traffic you cannot deliver moves the failure downstream; keep both aligned and leave retry headroom.

# Observability and Unit Economics

The gateway sees every request, including quota rejections, so emit one structured event per request:

request ID, tenant, application, and virtual model
resolved provider, model, and region
input, output, cached-read, cached-write, and reasoning tokens
time to first token, time per output token, and total latency
status, error type, retry count, and fallback flag
cache status and estimated cost

Use OpenTelemetry GenAI conventions. Spans carry gen_ai.provider.name, gen_ai.request.model, gen_ai.response.model, gen_ai.operation.name, and usage attributes such as gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and gen_ai.usage.cache_read.input_tokens (a subset of input; do not double count). Server-side latency uses gen_ai.server.time_to_first_token and gen_ai.server.time_per_output_token; token counters live under gen_ai.client.inference.usage.*. They are still Development, so pin a version and map them to your schema.

Do not log prompts and responses by default. The GenAI content attributes (gen_ai.input.messages, gen_ai.output.messages) are opt-in and can contain personal data. If you must capture content, sample it, redact it, set a short retention, and make access auditable.

Cost accounting starts with a versioned price table: input, output, cache read, cache write, and reasoning prices per provider and model. Calculate cost from actual provider usage, not token estimates. Reconcile monthly against the invoice; a drifting price table is worse than none. Show cache hits as avoided upstream cost, not zero cost — the gateway still did work, and semantic-cache embeddings are not free.

Track p95 TTFT, p95 TPOT, error rate by provider, fallback rate, 429 rate, queue depth, tokens per request, and cost per successful task. “Cost per million tokens” is an input metric; “cost per successful task” is the business metric, because it captures retries, fallbacks, and wasted generations. Break both down by tenant and model; alert on SLO burn and fallback spikes, not raw spend.

# A Reference Architecture

A workable production topology separates policy from serving:

 +----------------------+
 apps / agents -----> | AI gateway |
 | auth, aliases, |
 | quotas, cache, |
 | fallback, telemetry |
 +----------+-----------+
 |
 +------------------+------------------+
 | |
 v v
 managed model APIs inference gateway
 (provider A, provider B) (Gateway API + EPP)
 |
 +--------+--------+
 | |
 v v
 vLLM pool A vLLM pool B
 base model LoRA adapters

A minimal LiteLLM configuration (model IDs are placeholders):

model_list:
 - model_name: chat-default
 litellm_params:
 model: openai/<primary-model>
 api_key: os.environ/PRIMARY_API_KEY
 - model_name: chat-default
 litellm_params:
 model: anthropic/<secondary-model>
 api_key: os.environ/SECONDARY_API_KEY
 - model_name: chat-default
 litellm_params:
 model: openai/llama-3.3-70b-instruct # served by vLLM
 api_base: http://inference-gateway.inference.svc/v1
 api_key: os.environ/INTERNAL_GATEWAY_KEY
 - model_name: chat-fallback
 litellm_params:
 model: openai/<smaller-model>
 api_key: os.environ/PRIMARY_API_KEY

router_settings:
 routing_strategy: simple-shuffle
 num_retries: 2
 timeout: 30
 allowed_fails: 3
 cooldown_time: 30

litellm_settings:
 fallbacks: [{"chat-default": ["chat-fallback"]}]
 cache: true
 cache_params:
 type: redis

On the Kubernetes side, the inference gateway routes to an InferencePool rather than a Service. An abridged example:

apiVersion: inference.networking.k8s.io/v1
kind: InferencePool
metadata:
 name: llama-70b
spec:
 selector:
 matchLabels:
 app: vllm-llama-70b
 targetPorts:
 - number: 8000
 endpointPickerRef:
 name: llama-70b-epp
 port:
 number: 9002
 failureMode: FailOpen
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: llama-route
spec:
 parentRefs:
 - group: gateway.networking.k8s.io
 kind: Gateway
 name: inference-gateway
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /v1/chat/completions
 backendRefs:
 - group: inference.networking.k8s.io
 kind: InferencePool
 name: llama-70b

The AI gateway treats the inference gateway as another OpenAI-compatible backend. You can change serving runtimes, GPU pools, or model versions without changing application code, and add a managed provider without changing Kubernetes manifests.

Treat the gateway like production infrastructure: at least two replicas across zones, a pod disruption budget, no synchronous calls to external services in the policy chain, Redis local to the region, and no response buffering so streaming actually streams. Test at twice your expected peak; a gateway that is cheap at idle can become the first-launch bottleneck. Decide whether a break-glass path direct to a provider exists. It bypasses quotas and audit, so document it as an emergency procedure, not a habit.

# Anti-Patterns and a Checklist

Predictable failures:

Gateway as a business-logic layer. Prompt construction, RAG, and agent orchestration belong in the application; the gateway routes, enforces, and observes.
Retry multiplication. SDK retries plus gateway retries plus provider retries turn one failure into a load spike. One owner, one budget.
Silent downgrade. Falling back to a smaller model without telling the caller changes quality. Surface the actual model.
Semantic cache by default. It is a correctness risk on multi-turn and agentic traffic, and a tenant-isolation risk when scoped globally.
Prompt logging everywhere. It creates a PII archive with no owner. Log metadata; opt in to content.
Routing on stale metrics. If the scrape interval outlasts the burst, the endpoint picker steers with yesterday’s traffic.
Unmonitored single point of failure. The gateway sees all traffic, so its own availability, latency, and error budget need dashboards too.

A short checklist before calling the layer production-ready:

Apps use virtual model names; aliases are versioned and reversible.
Per-team quotas exist in tokens and concurrency, not just requests per minute.
The fallback chain is tested for tool calling, JSON, context length, and refusal behavior.
Exactly one layer owns retries, with jitter and a budget.
Timeouts are split into connect, first-token, idle, and total.
Caches are scoped by tenant, model version, and parameters; semantic caching is opt-in.
Every request emits tokens, TTFT, TPOT, status, fallback flag, and cost.
Content is not logged by default; the gateway has its own SLO and capacity test.

# Conclusion

The model is the product; the gateway is how you keep it running. It turns a model name into policy, a provider incident into a fallback, a cache hit into either a saving or a correctness bug, and a token into a number finance can reconcile.

You do not need every feature on day one. Start with one virtual model, one fallback, one token-based quota, and one dashboard for cost per successful task. Add inference-aware routing when you run your own pools, and caching only where you can prove it is safe. It is the difference between a demo and a feature that survives its first real month.

# Further Reading

Agent Router (formerly Envoy AI Gateway) — provider routing and InferencePool support
Gateway API Inference Extension — InferencePool and Endpoint Picker
LiteLLM Router and caching — aliases, fallbacks, cooldowns, caches
Kong AI Gateway — policies, semantic caching, consumer scoping
OpenTelemetry GenAI conventions — spans, token metrics, content capture
vLLM metrics — queue, KV cache, prefix cache, TTFT, TPOT
AI inference infrastructure from a DevOps perspective — the GPU and serving layer below the gateway

AI Inference Infrastructure from a DevOps Perspective: Self-Managed Kubernetes vs Managed Cloud

Mon, 26 Oct 2026 00:00:00 Z

Serving machine learning models is no longer a research problem — it is an infrastructure problem. Model inference workloads have their own shape: bursty traffic, expensive GPU-bound nodes, multi-gigabyte images and model weights, and autoscaling that reacts to tokens per second rather than CPU. In this guide we walk through what a production inference stack actually looks like — first as self-managed infrastructure on Kubernetes, then as managed cloud services from AWS, Google Cloud, Azure, and the GPU-native clouds — and compare the two paths on operations, scaling, latency, and cost from the perspective of the person who has to keep both running at 3 a.m.

# Why Inference Infrastructure Is Different

A model inference service looks like a normal web API but behaves very differently under the hood:

Hardware constraints. GPUs are expensive, scarce, and can’t be live-migrated. A pod that pins a GPU shapes how you schedule, drain, upgrade, and preempt. Driver/CUDA/kernel version alignment is a real compatibility matrix you must manage.
Big, cold artifacts. Images with CUDA runtimes and downloaded model weights easily reach 5–50 GB. Cold starts are measured in minutes, not seconds, which breaks naive scale-to-zero designs.
Custom scaling signals. HPA on CPU is nearly useless: vLLM saturates GPU KV-cache utilization way before CPU looks busy. You need scaling on model-specific metrics (queue depth, requests per replica, time to first token).
Different SLOs. For LLM serving you don’t just care about p99 latency of a response; you care about time to first token (TTFT) and time per output token (TPOT), because streaming changes how users perceive latency.
Cost concentration. A single A100/H100 node can cost several thousand dollars a month. Inference is where most of your ML cloud bill lives, and idle-GPU waste is the default state unless you engineer against it.

These constraints drive the real architectural decision: where does the GPU fleet and the serving logic live — in your cluster, or in a provider’s?

# The Anatomy of an Inference Request

Before choosing infrastructure, it helps to understand what a single request actually does, because every infra decision downstream is reacting to this behavior.

For a typical LLM request on vLLM:

Request admission. The request arrives over HTTP/gRPC and enters a waiting queue.
Weight residency check. The model weights must already be in GPU memory — loading 16 GB from disk takes tens of seconds to minutes, so this must happen at startup, per replica.
Batching. The runtime waits a few milliseconds to coalesce concurrent requests into a batch. Modern serving engines do continuous batching: new requests join a running batch at each decode step rather than waiting for the whole batch to finish, keeping GPU utilization flat under load.
Prefill and decode. The prompt is processed (prefill, compute-bound), then tokens are generated one at a time (decode, memory-bandwidth-bound). The KV cache — the stored attention state of the context — grows with prompt + output length and is the real memory consumer, not the weights.
Streaming response. Tokens stream out over SSE/WebSocket as they’re generated.

Why this matters for DevOps:

KV cache means memory, not compute, is often the bottleneck → you tune max-model-len, quantization, and PagedAttention-style cache management. OOM at the KV cache is the most common production failure of a self-managed LLM deployment.
Continuous batching means throughput per GPU is highly dependent on concurrency → autoscaling to more replicas can decrease cost efficiency if it breaks batching. You often want fewer, busier replicas rather than many idle ones.
Weights at startup means every scale-up event pays a multi-minute loading cost → your autoscaler needs long cooldowns and pre-warmed node pools to handle bursts.

Keep this lifecycle in mind; it explains the Kubernetes specifics below.

# Option 1: Self-Managed Inference on Kubernetes

The typical self-managed stack looks like this:

Layer	Common choice
Serving runtime	vLLM (LLMs), Triton with TensorRT/TensorRT-LLM, SGLang, KServe, BentoML
GPU enablement	NVIDIA GPU Operator (drivers, device plugin, DCGM monitoring, MIG)
Cluster/fleet	EKS / GKE / AKS with GPU node groups, or bare metal
Request routing	Ingress or service mesh, load balancing aware of in-flight requests
Autoscaling	HPA + KEDA on custom metrics; Ray Autoscaler for LLM serving
Storage/weights	Model artifacts in object storage, PVC or init-container download, local NVMe cache
Observability	DCGM exporter, Prometheus, Grafana, LLM-specific dashboards

## Serving Runtimes: Know What You’re Deploying

Choosing a runtime is the first real decision, and it’s a DevOps-relevant one because each has different operational characteristics:

vLLM — the current default for serving open-weight LLMs via an OpenAI-compatible API. Best-in-class continuous batching with PagedAttention. Operationally, you manage GPU memory limits carefully (the gpu_memory_utilization flag) and version-bump aggressively: the project moves fast and performance improves release to release.
Triton Inference Server — NVIDIA’s general-purpose server, targeting traditional ML (XGBoost, ONNX, vision models) and TensorRT-LLM for NVIDIA-tuned LLM serving. Best when you have a mixed zoo of model types behind one server, or you want NVIDIA’s engines. More config-heavy (model repository structure, config.pbtxt), but predictable.
SGLang — newer engine with advanced scheduling (RadixAttention, prefix caching), often outperforms vLLM on multi-turn conversation traffic where prompts share prefixes. Watch this space if chat workloads dominate.
KServe — not a runtime but a serving framework on Kubernetes: gives you InferenceService CRDs, canary traffic splits among model revisions, scale-from-zero via queue-proxy, and pluggable backends (it can run vLLM or Triton underneath).

The practical pattern: vLLM (or SGLang) for OpenAI-compatible LLM endpoints, Triton for everything else, KServe if you want model-level reconciliation and canarying as Kubernetes-native objects rather than rolling your own.

## GPU Operator and Node Management

Running GPUs in Kubernetes without the operator means hand-installing drivers on node images, which breaks under upgrades. The NVIDIA GPU Operator packages the whole stack — drivers, container toolkit, device plugin, DCGM exporter for metrics, and MIG manager — as controllable Kubernetes objects. Node pool definition typically looks like:

apiVersion: eks.amazonaws.com/v1alpha2
kind: NodeGroup # EKS managed node group; GKE/AKS equivalents apply
metadata:
 name: gpu-inference
spec:
 instanceType: g6e.12xlarge # L40S-class cards
 labels:
 node-type: gpu-inference
 taints:
 - key: nvidia.com/gpu
 value: "dedicated"
 effect: NoSchedule
 scaling:
 minSize: 2
 maxSize: 10

Things to get right here:

Taint GPU nodes. Non-GPU pods will happily schedule onto your most expensive nodes unless you taint. Every real cluster has horror stories of a misconfigured logging DaemonSet costing thousands per month.
Set up Cluster Autoscaler / Karpenter for GPU instance types. GPU capacity is scarce in many regions. Configure least-waste priority (e.g., prefer 8-GPU nodes only for 8-GPU jobs), consolidation carefully — consolidations that reschedule GPU pods cause visible service blips from weight reloading.
MIG (Multi-Instance GPU) for small models. On A100/H100 you can partition a GPU into slices (e.g., 7×10GB slices). If your models fit in 10 GB, MIG turns one node into seven serving replicas and slashes cost. The GPU Operator’s MIG manager configures this; the device plugin then exposes nvidia.com/mig-1g.10gb as a schedulable resource.

## Weights: The Storage Side of Inference

Model loading is a first-class ops concern. The naive path —Pod starts, downloads 15 GB from Hugging Face on the fly — gives you multi-minute readiness, wasted bandwidth (every replica downloads), and flaky readiness on network hiccups.

Better patterns, in increasing order of investment:

Local NVMe cache on the node. An init container (or hostPath-backed downloader) checks for the model in a node-local directory and downloads only if absent. After the first scale-up on a node, subsequent pods start in seconds.
PVC with ReadOnlyMany. A pre-populated PVC (or a Filestore/EFS/GCFileSystem mount) shared across replicas avoids redundant downloads. Watch network filesystem throughput — for big models, it can become a bottleneck slower than downloading from object storage.
Pre-baked serving images or snapshots. Bake weights into the image (big images, ~15–40 GB pull) or AMI/disk snapshot (fast). This is the “warm node pool” approach: autoscaler launches nodes from a snapshot with weights and even model server config already in place.

The vLLM deployment example from earlier, with the cache volume pattern:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: llm-inference
spec:
 replicas: 2
 selector:
 matchLabels: { app: llm-inference }
 template:
 metadata:
 labels: { app: llm-inference }
 spec:
 nodeSelector:
 node-type: gpu-inference
 tolerations:
 - key: nvidia.com/gpu
 operator: Exists
 effect: NoSchedule
 initContainers:
 - name: fetch-model
 image: amazon/aws-cli:latest
 command: ["sh", "-c", "aws s3 sync s3://our-models/llama-8b /models/llama-8b --no-progress"]
 volumeMounts:
 - name: model-cache
 mountPath: /models
 containers:
 - name: vllm
 image: vllm/vllm-openai:v0.6.4
 args:
 - --model
 - /models/llama-8b
 - --max-model-len
 - "8192"
 - --gpu-memory-utilization
 - "0.92"
 resources:
 limits:
 nvidia.com/gpu: "1"
 memory: 64Gi
 requests:
 cpu: "8"
 readinessProbe:
 httpGet: { path: /health, port: 8000 }
 initialDelaySeconds: 10
 periodSeconds: 5
 failureThreshold: 60 # allow minutes for first load
 volumeMounts:
 - name: model-cache
 mountPath: /models
 volumes:
 - name: model-cache
 emptyDir:
 sizeLimit: 40Gi

Notice the long failureThreshold on the readiness probe: cold starts are legitimately minutes. If you set a normal 30-second threshold, every scale-up triggers a crash loop of your new replica and the autoscaler fights itself.

## Autoscaling on the Right Signal

Standard HPA on CPU is the wrong tool, for two reasons: GPU inference barely uses host CPU, and scale-up latency (node launch + image pull + weight load) is minutes, so you need to scale early and hold, not react and thrash.

The standard approach is KEDA with Prometheus metrics from the serving runtime. vLLM exposes:

vllm:num_requests_waiting — queue depth
vllm:num_requests_running — in-flight per replica
vllm:time_to_first_token_seconds / vllm:e2e_request_latency_seconds — latency quantiles
vllm:gpu_cache_usage_perc — KV cache saturation

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
 name: llm-scaledobject
spec:
 scaleTargetRef:
 name: llm-inference
 minReplicaCount: 2 # warm floor: never below your p-cold-start tolerance
 maxReplicaCount: 12
 cooldownPeriod: 600 # long enough to avoid thrash — readiness is slow
 triggers:
 - type: prometheus
 metadata:
 address: http://prometheus.monitoring:9090
 query: sum(vllm:num_requests_waiting)
 threshold: "10" # target: max ~10 queued requests per replica

Key engineering points:

Warm floor over scale-to-zero. With minute-scale cold starts, scale-to-zero only works for workloads where users tolerate minutes of latency (background jobs, internal batch tools). Otherwise keep a warm floor and treat scale-down as a cost lever, not a default.
Long cooldowns. Because readiness is slow, aggressive scale-down after a spike makes you pay the full cold-start cost repeatedly. Ten minutes of cooldown is more normal than datacenter default.
Node-level pre-scaling. Pod autoscaling doesn’t help if the autoscaler needs to launch a GPU node to satisfy it. Pre-provision GPU capacity or over-provision GPU nodes with fake low-priority “placeholder” pods (the classic cluster-autoscaler trick) if burst latency is contractual.
Batching-aware reasoning. A single replica can often handle 20–50 concurrent LLM requests with good TPOT. If you scale on requests-per-replica with a target like 30 or 50, you keep the batching advantage instead of shattering load across underfilled replicas.

## Load Balancing and Traffic Shape

Standard round-robin across pods is not ideal for LLM serving: replicas have different queue lengths already. Consider a routing layer aware of in-flight requests (e.g., a gateway or a serving framework’s own router that balances by num_requests_running), and sticky routing by conversation/session for multi-turn traffic so prefix caches stay warm. If you’re doing streaming (SSE/WebSocket), confirm your ingress layer has long/idle timeouts tuned — common sources of mysterious 502s mid-stream.

## Spot GPUs and Interruption Engineering

Spot/preemptible GPU capacity is frequently 50–80% cheaper and, because inference is stateless (unlike training checkpoints), somewhat friendlier to interruptions. Engineering it well means:

Disruption-tolerant deployments: reasonable PodDisruptionBudgets, including spot-node taints and tolerating instance termination notices (via 2-minute termination handlers draining pods).
Fast weight loading (the storage patterns above) so a rescheduled replica is serving reasonably quickly after a handoff.
Multiple capacity shapes: don’t demand only p5.48xlarge; deploy with fallback smaller instance types and run several smaller replicas.
Some teams run spot for the bulk of capacity and on-demand as a warm internal minimum, using cluster autoscaler “prefer-spot” strategies.

## Observability for Inference

The metrics stack is standard Prometheus/Grafana, but the content is GPU- and LLM-specific:

DCGM exporter — GPU utilization, memory usage, temperature, power, ECC errors. High temperature and throttling directly affect latency and hardware life.
Serving engine metrics — queue depths (waiting requests), running requests, TTFT and end-to-end latency quantiles, KV cache usage.
Gateway metrics — request rate, streaming connection counts and durations (long-lived streaming requires rounding and thinking about connection limits and timeouts differently than normal HTTP).
Cost instrumentation — tokens served per hour per GPU, dollars per million tokens. This is your north-star SLO for inference economics: track it per model version, because quantization and version changes move it substantially.

Alerting on GPU utilization is misleading: a well-batched GPU should run at ~100% memory bandwidth/utilization under load. Alert instead on queue depth and latency SLOs, treating GPU utilization as a capacity-planning graph.

# Option 2: Managed Cloud Inference Services

Every major provider has a managed inference product — you bring a model artifact, config, and get an endpoint:

Provider	Service	Notes
AWS	SageMaker Endpoints (incl. async & serverless), Bedrock, EC2 with Deep Learning AMIs	SageMaker handles serving infra; Bedrock offers hosted foundation models; asynchronous endpoints handle large payloads
Google Cloud	Vertex AI (Endpoints, Model Garden), GKE Autopilot with GPUs	Tight integration with BigQuery/Colab and Google’s managed Nvidia/TPU fleet
Azure	Azure ML online endpoints, Azure OpenAI	Azure OpenAI is often the shortest path for enterprise LLM use; Azure ML for custom models
GPU-native clouds	CoreWeave, RunPod, Lambda, Modal, Anyscale	Kubernetes-native (CoreWeave) or simpler per-GPU rental; strong price-performance and quick cold-start UX

The mental model: you give up governing the whole cluster below the model. The platform provides the serving runtime (or a container you supply), GPU instance selection, autoscaling targets, network isolation (VPC endpoints), encryption at rest/in transit, audit trails, and per-invocation pricing modes.

A typical managed setup is declarative, roughly conceptually:

# Conceptually — each provider uses its own DSL/SDK
endpoint:
 name: llama-8b-prod
 model: s3://our-models/llama-8b-finetuned.tar.gz
 container: 763104351884.dkr.ecr.us-east-1.amazonaws.com/djl-inference:vllm
 instance_type: ml.g6e.4xlarge
 autoscaling:
 target_metric: ModelInvocationsPerInstance
 target_value: 50
 min: 1
 max: 8
 data_capture: enabled # for drift monitoring / replay

What you gain

Time to first production call measured in hours: no GPU operator, no driver pain, no cluster provisioning, no MIG experiments.
Built-in autoscaling on provided inference metrics, handled by the provider’s control plane, with sensible scale-in defaults and warm pool semantics engineered upstream.
Compliance and network posture mostly handled: private VPC endpoints, KMS-encrypted model artifacts, IAM auth, audit logs — what would take weeks of K8s policy engineering.
Pricing discipline tools: serverless variants with per-invocation billing and scale-to-zero handled for you (SageMaker serverless endpoints, Vertex AI predictive); async endpoints for long-running jobs with S3 payloads.
Traffic management and monitoring included: canary weights on endpoint versions, dashboards, and drift/capture hooks.

What you pay for

Per-instance markup — typically 1.3–2× raw EC2-equivalent GPU-hour cost, in exchange for the managed control plane.
Limited runtime knobs. Batching behavior, continuous batching parameters, KV cache policy, and GPU partitioning are whatever the platform exposes; if the platform’s defaults aren’t optimal for your traffic shape, you absorb it, or you use custom containers and hope your tuning isn’t broken by an endpoint update.
Platform-shaped architectures. Your request path routes through the provider’s frontend, so p50 latency can add tens of milliseconds; some platforms add queue-before-GPU even for warm endpoints.
Lock-in to control plane conventions. IAM roles, autoscaling metric names, SDK/DSL, endpoint state model — nontrivial to port to another provider later (though most lock-in is at this config layer, not the model itself).
Cost concentration at sustained load. Running one managed endpoint 24/7 is close to always paying the premium. Managed serverless endpoints scale smoothly but pass through latency; if your traffic is sustained rather than bursty, the raw-GPU time-based pricing of EC2 (plus your own ops) often wins on cost per million tokens.

# Comparing the Two Paths

Dimension	Self-managed K8s	Managed cloud
Time to first production call	Days to weeks	Hours
Cost at sustained 24/7 load	Lower (spot + reserved + fractioning + MIG)	Higher per-GPU-hour
Cost at spiky/bursty traffic	Harder (idle GPUs, warm floors)	Easier (serverless variants)
Cold-start behavior	You engineer it (snapshots, caches, warm pools)	Platform-managed warm pools
Ops burden	High — GPU operator, drivers, serving tuning, spot interrupts	Low — endpoint ops only
Model-level optimization	Full (quantization, batching, CUDA graphs, speculative decoding)	Limited to what the platform exposes
Scaling signal	Custom metrics, high tuning effort	Platform-provided targets
Vendor lock-in	Low — cluster and serving code are portable	Significant — endpoint, IAM, SDK layer
Latency overhead	Direct pod networking	Provider frontend hop
Observability	DIY (Prometheus, DCGM, Grafana)	Included dashboards
Token-economics visibility	Complete — you measure everything	Provider dashboards only

# A Pragmatic Decision Framework

Most teams don’t need a binary answer — they need a routing rule. Patterns that work in practice:

Start managed, migrate what deserves it. Deploy on SageMaker/Vertex/Azure ML endpoints first — validate the model has product fit before investing in its infrastructure. When a model’s sustained traffic, special GPU configuration, or per-token economics justify it, lift the serving code to your cluster.
Keep self-managed for tuning-sensitive workloads. Continuous batching behavior, quantization (AWQ, GPTQ, FP8/FP4), speculative decoding, prefix caching, and KV-cache offloads matter enormously for cost per million tokens. If cost per token is your north-star metric, you want a runtime you control: your own vLLM (or SGLang) on GPUs you own is usually the frontier.
Use serverless managed endpoints for hard-to-forecast, low-QPS workloads. Internal tools, demos, admin features, geographically rare traffic — scale-to-zero in the cloud avoids keeping a GPU warm for three requests a day, and the latency is acceptable when users expect internal tools to be slower.
Match the traffic shape to the pricing model. Sustained 24/7 load → self-managed with spot/reserved and MIG fractioning. Bursty, spiky, unpredictable → managed serverless. Long-running batch inference jobs → async endpoints or spot-backed batch jobs on your own clusters.
Don’t fight the GPU supply shortage alone. If you can’t get capacity in-region from your primary cloud, the GPU-native clouds (CoreWeave is literally K8s under the hood; RunPod/Lambda prioritize availability) are often a supplement — treat them in your multi-cloud strategy as capacity, with the same serving images and IaC.
Beware common pitfalls either way: under-provisioning GPU memory for KV cache, forgetting to request/raise GPU service quotas in the target region, model weight download times dominating autoscaling behavior, GPU utilization metrics firing “capacity” alarms when the GPU is simply efficiently batched, and long-lived streaming connections leaking through ingress misconfigurations.

# Conclusion

Inference infrastructure is infrastructure engineering, more than ML engineering. Self-managed Kubernetes gives you control, best unit economics at scale, and deep optimization headroom (quantization, batching, fractioning) — at the price of being responsible for some of the hardest workloads to keep running: GPUs, huge deterministic artifacts, custom autoscaling, interruption engineering. Managed cloud services trade cost per GPU-hour and flexibility for operational simplicity, compliance posture, and speed to production.

Most mature ML platform teams end up with both: managed endpoints for early-stage and spiky models, and a tuned self-managed GPU fleet on Kubernetes for the workloads where cost per token and tail latency are competitive differentiators. The DevOps answer isn’t “Kubernetes or managed” — it’s knowing what your traffic looks like at the token level and using both paths where the tradeoffs serve you.

# Further Reading

vLLM documentation — serving runtime, continuous batching, and Prometheus metrics
NVIDIA GPU Operator — drivers, device plugin, DCGM, and MIG management on Kubernetes
KEDA documentation — event-driven autoscaling on custom metrics
KServe — model serving framework with traffic splitting on Kubernetes
SGLang — RadixAttention-based serving engine for prefix-heavy workloads
SageMaker Inference — endpoint types, autoscaling, and serverless inference
Vertex AI Endpoints and Azure ML online endpoints — managed inference reference