GKE Inference Gateway + llm-d practical guide: Why AI inference teams must now design the routing layer first, rather than the model server

In 2026, the inference bottleneck is not simply a matter of GPU quantity, but has moved to a matter of what routing rules are used to handle long prompts, KV cache, and rapid traffic fluctuations. This article focuses on the combination of GKE Inference Gateway and llm-d released at Google Cloud Next ’26, and summarizes what teams planning to operate a Kubernetes-based inference stack should design first. The target audience is platform engineers and MLOps teams reviewing their own model serving. Conversely, if you are a team that only uses fully managed APIs like Bedrock, the design complexity of this article may be excessive.

1. One-line problem definition

Summary: Inference costs leak faster from incorrect decisions about which instance to send a request to, where to place the KV cache, and when to scale it, than from the model itself.

Existing LLM serving articles often focused on whether to use vLLM or SGLang as the model server. However, in actual operation, the routing layer collapses first. Even for the same model, if long system prompts are repeated, the KV cache must be sent to the reused instance, and if short response-oriented traffic and long response-oriented traffic are processed with the same rule, TTFT and throughput will fluctuate at the same time.

In other words, the question now is not “Can the model be launched?” but “Can the actual requests be distributed in an operational way?” This article explains how the GKE Inference Gateway and llm-d attempt to solve this problem, and where to trust and where to start verifying yourself.

2. First, conclusion

Summary: Highly worth considering for teams with their own GPU/TPU operations and Kubernetes capabilities, but a fully managed API is more realistic for small teams or early products.

• A team that fits well: Teams that have many identical prompt patterns, need to manually optimize long contexts or multi-tenant inference, and want to closely control cost and latency
• Overworked team:Team where daily traffic is still small and feature release speed is more important than GPU operation
• My judgment: The core value of GKE Inference Gateway lies in “elevating inference routing to an operational concept on top of Kubernetes.” However, this is not immediately a universal answer. If you don't have an operations team to understand routing rules, cache hierarchies, and autoscaling, Bedrock·Vertex's managed inference API is safer.

3. Decomposition of core structure

Summary: This stack is not a single model server, but is a structure that coordinates inference by layering a router, scheduler, and cache layer on top of the model server.

Simplifying the structure, there are four layers below.

1. Model server layer: An engine such as vLLM is responsible for generating the actual token.
2. Inference Routing Layer: GKE Inference Gateway determines which backend to send the request to.
3. Orchestration layer: llm-d provides advanced serving patterns such as prefix-cache-aware routing, predicted latency scheduling, and prefill/decode separation.
4. Infrastructure layer: GKE, GPU/TPU, Local SSD, GCS/Lustre, HPA are responsible for actual resources and scaling.

The important thing to note is that Google is not pushing llm-d as a “model server replacement.” llm-d is more of a higher control layer on top of vLLM. So, the competitor in this combination is not a simple model server, but a homemade load balancer + cache policy + autoscaling script bundle.

4. Description of design intent

Summary:The design goal is to reduce “time to SOTA performance” rather than peak performance itself.

The phrase that Google and llm-d repeatedly emphasize is time to SOTA. This means that the inference tuning work that each team had to do for months will be condensed into a proven guide and scheduler. This reflects two realities:

• First, modern LLM serving has more difficult operating parameters than the model itself. The perceived performance will vary greatly depending on where the KV cache is offloaded, how long prompts are handled, and whether prefill/decode is separated.
• Second, the Kubernetes team already has the Gateway API, HPA, and observability tools, so they prefer to layer on top of existing operating systems rather than a new, closed inference platform.

There are also things I gave up instead. The simplicity of a few clicks like a fully managed service has been abandoned. In exchange for operational freedom, more responsibility for interpreting routing policies and benchmarks falls on your team.

5. Evidence and Comparison

Summary: The comparison criteria are control, initial speed, room for cost optimization, and operational difficulty.

Option	Advantages	Weakness	Suitable situation
GKE Inference Gateway + llm-d	Routing, KV cache, and autoscaling can be finely adjusted, fits well with Kubernetes	Many components and high operational difficulty	Teams where large-scale/iterative inference, multi-tenant, and long context optimization are important
vLLM exclusive operation	Simple structure and quick startup	Advanced routing/distributed optimization must be attached manually	Single model, single tenant, initial PoC
Managed API, such as Amazon Bedrock	Fast launch, low infrastructure operation burden, simple capacity management	Fine-grained routing/cache optimization control is limited	Early stage of product, team with few GPU operation staff

Google stated that TTFT can be reduced by more than 70% with Predictive Latency Boost, offloading the KV cache to RAM reduces TTFT by more than 40% at 10K system prompts, and offloading Local SSD suggests throughput improvement of about 70% at 50K system prompts. However, since this figure is derived from specific benchmark conditions, it is dangerous to put it in the budget table as is. Conversely, with the release of Claude Opus 4.7 in April 2026, AWS Bedrock touted managed strengths such as a ‘new inference engine’, out-of-the-box availability of up to 10,000 RPM, and queue-based processing for spikes in demand. In other words, Google's advantage is room for optimization, while AWS's advantage is operational simplicity.

6. Actual operation flow / step-by-step execution method

Summary: To work properly, this stack requires defining latency targets and prompt length distributions prior to model selection.

1. Gather requirements: Enter the target TTFT, output tokens/sec, representative prompt length, and multi-tenant status in numerical values. Example: TTFT 1200ms or less, average input 8K tokens, P95 input 40K tokens.
2. Explore candidate combinations with Inference Quickstart: Compare model, accelerator, and serving stack combinations with GKE Inference Quickstart to choose a cost/performance profile.
3. Default stack deployment: The simplest baseline is vLLM + GKE Inference Gateway. At this time, all advanced features of llm-d are not turned on yet.
4. Check traffic characteristics: If there is a lot of long prompt reuse, review prefix-cache-aware routing, and if it is very long input, add prefill/decode separation or KV offload.
5. Define autoscaling threshold: Capture HPA based on inferred metrics such as queue, TTFT, and token throughput instead of CPU or memory.
6. True load verification: Reproduce actual prompt distribution with inference-perf or self-replay. Google's default benchmark has a fixed input/output distribution, so if the service pattern is different from ours, the results will be different.

#Example thought flow
1) gcloud container ai profiles list
2) Check the benchmark of the candidate profile
3) Deploy optimized manifest with kubectl apply
4) Re-measure TTFT/TPOT/P95 cost with real traffic replay
5) If necessary, apply llm-d’s cache-aware routing and tiered KV cache.

The part that beginner teams miss the most is step 2. If you decide on hardware first, you usually overinvest. First, you need to quantify your input length and latency goals to determine which cache tier you need.

7. Pitfalls

Summary: This stack is not a “the more features you turn on, the better it is” structure.

• Mistake 1: Using benchmark numbers as is in the introduction budget
  Prevention: Use the numbers released by Google and llm-d only as a reference point, and be sure to use your own prompt distribution. Remeasure.
  Recovery: Create a real log-based sample set and recalculate TTFT·TPOT·$/token.
• Mistake 2: Mistaking KV cache offload as a panacea
  Prevention: For short-request-oriented services, adding an offload layer may only increase operational complexity.
  Recovery: Prompt length Separate the cache layer by section and perform A/B testing.
• Mistake 3: Failing to distinguish between model server issues and routing issues
  Prevention: When p95 delay rises, isolate the indicator whether the vLLM itself is a bottleneck or a scheduler selection failure.
  Recovery: Gateway decision log, queue depth, and cache hit rate are collected together.
• Mistake 4: Designing a hypercluster-level architecture first without operational team capabilities
  Prevention: Start with a single region/single model baseline.
  Recovery: Multi-region, multi-model, RL Separate workloads are introduced step by step.

8. Strengths and Limitations

Summary: Strength is control, limitation is complexity.

Strength

• Integrates naturally into the Kubernetes-based operating system.
• Real-world problems such as long prompts, repeated prefixes, and multi-tenancy can be handled with routing policies.
• Inference Quickstart makes it easy to quantify “where to start”.

Limit

• If the team does not have an observation system and performance verification culture, the difficulty of operation will only increase.
• Because it is a Google-centric stack, it is different from other cloud or simple serverless inference strategies.
• Even if the public figures are good, for smaller products a managed API like Bedrock may be better in terms of total cost.

My judgment is clear. For teams where monthly usage is still small and where product learning speed is more important than model differentiation, this architecture may be premature. On the other hand, if your team experiences repetitive enterprise prompts, long contexts, and multi-tenant cost pressures throughout the day, you will have to design your routing layer now, or you will end up purchasing additional GPUs again and again.

9. Points to study more deeply

Summary:The order of study is routing·cache·benchmark analysis rather than model server.

1. Understand how NTPOT, TTFT, and throughput curve are converted to recommended values in the GKE Inference Quickstart document.
2. Read llm-d's optimized baseline, predicted latency scheduling, tiered prefix cache guide.
3. Distinguish between the role of vLLM standalone operation and llm-d higher-level orchestration.
4. Compare how much the inference APIs from Bedrock or Vertex AI reduce operational burden with managed alternatives.
5. First, extract the prompt length distribution and response length distribution from the actual service log.

10. Action Checklist + Author's Perspective

Summary: This stack is not about finding a ‘good model’, but about establishing ‘good operating standards’.

• Do you know the P50/P95 input token length of our service?
• Have you defined your TTFT and TPOT targets in numbers?
• Is there an advantage to cache-aware routing due to a high repeat prefix ratio?
• Isn't the operating labor cost greater than the GPU/TPU cost?
• Are sample sets and observation dashboards for actual traffic replays prepared?
• Have you compared the total cost of ownership (TCO) of a managed API versus your own operation?
• Aren’t long text input and short text input grouped under the same serving policy?

Definition of Done: Based on the actual prompt distribution, the three configurations of baseline·cache-aware·KV offload are compared and all P95 delays and costs must be documented to complete the introduction review.

Author's Perspective: I recommend this combination to “teams starting to really run inference on Kubernetes.” Conversely, if the number of model calls is still small and the speed of experimentation is more important in the early stages of a startup, it is better to use the managed API first. Self-management brings not only freedom but also responsibility.

Reference material

• What’s new in GKE at Next ’26 (Google Cloud, 2026-04-22)
• AI infrastructure at Next ’26 (Google Cloud, 2026-04-22)
• Analyze model serving performance and costs with GKE Inference Quickstart (Google Cloud Docs, accessed 2026-04-23)
• llm-d GitHub repository and guides (llm-d, accessed 2026-04-23)
• Introducing Anthropic’s Claude Opus 4.7 model in Amazon Bedrock (AWS, 2026-04-16)