Google·Marvel AI chip commentary: In the inference era, why AI infrastructure teams must now redesign the memory bottleneck first, rather than FLOPS

Publication date: 2026-04-20 | Category: Development information

1) Problem definition

Key takeaway: As AI services grow, the bottleneck shifts from the amount of model computation itself to how quickly weights and KV caches are moved and held.

AI Times reported on April 20, 2026 that Google is discussing a memory processing unit (MPU) paired with TPU and the next-generation TPU with Marvell. The reason this news is important is not simply because a new chip is coming out. What is more important is that Google has now publicly revealed that building a single compute chip is not enough when looking at its inference infrastructure.

This article is a commentary for AI infrastructure teams, platform engineers, MLOps leads, and backend teams who need to manage inference costs. The scope is Why memory bottlenecks have become the core of the inference competition, what the TPU+memory auxiliary chip structure is trying to solve, and what judgment criteria are needed compared to existing GPU-centric designs. Conversely, the hardware separation design referred to here may be excessive for small-scale demo services by individual developers.

2) Conclusion first

Key summary: The essence of this news is not that Google will increase the number of chips, but that it will optimize inference performance by separating it into calculation performance and memory performance Conversion.

• Teams you should consider right now:Teams running long context, large models, multimodal inference, agent-type workloads
• Team with better observation still: Initial service with small model size, cost and latency sufficient with one type of GPU
• My judgment: Going forward, the key question for inference infrastructure will be “How do we reduce memory bandwidth, cache maintenance, and data movement paths” rather than “Which chip delivers more TFLOPS?”

In other words, the infrastructure team must now look at not only model selection and chip selection, but also computation layer, memory layer, and inter-chip connection layer. Without this perspective, it is difficult to explain why purchasing more GPUs does not reduce actual service latency and costs as expected.

3) Core structure decomposition

Key summary: The structure that Google is aiming for is closer to “a structure that separates calculation and memory managers” rather than “one big brain.”

1. Computation layer: TPU is the center of matrix computation and model execution.
2. Memory layer: The MPU mentioned in the article focuses on data movement and alleviating memory processing bottlenecks.
3. Connection layer: How the connection is made between the TPU and the memory auxiliary chip determines the actual performance.
4. Batch layer: Long contexts, large KV caches, large batches, multimodal inputs increase memory pressure.
5. Operation Layer: The goal is to reliably handle more inference requests on the same rack with the same power than the benchmark score.

To put it simply, for a novice developer, the TPU is a calculator, and the MPU is closer to a warehouse manager that helps the calculator not stop waiting for data. No matter how fast the calculator is, if the necessary data is not received on time, the overall speed will be slow. This problem is especially severe in inference of large language models. This is because memory access becomes much more important in the decoding stage where subsequent tokens are extracted than before the first token.

4) Explanation of design intent

Key summary: Google's design intention is not to boast about computational performance, but to reduce waste caused by memory walls when inference demand surges.

Google has already brought the inference era to the forefront with the Ironwood TPU released in 2025. According to the official blog, Ironwood is the first TPU designed specifically for inference, offering 192GB of HBM capacity per chip and 7.37TB/s of bandwidth. However, the same Google Cloud document also directly explains that HBM can still be a bottleneck for memory-intensive workloads. This means that even if you significantly increase the memory, the problem is not over.

This is where the MPU discussion in the article becomes meaningful. In inference workloads, model parameters, KV caches, embeddings, and intermediate activations are constantly moving. Instead of increasing only the TPU, Google appears to be trying to increase overall efficiency by optimizing this data movement layer separately.

• What you get: Alleviating memory bottlenecks, improving inference throughput, and potentially improving efficiency/power ratio
• Giving up: Increased chip structure complexity, software optimization burden, increased supply chain and packaging difficulty
• Practical interpretation: In the future, inference hardware is likely to have more combination designs with multiple roles rather than a single general-purpose accelerator.

Google is not an exception to this trend. In an official announcement in late 2024, Marvell cited up to 25% more compute area, up to 33% more memory, and up to 70% lower interface power with its custom HBM architecture. In other words, the entire industry is already past the stage where “only the calculation chip needs to be fast”.

5) Evidence and comparison

Key takeaway: The comparison is not based on brand, but on how to solve the memory bottleneck.

By combining official data, the criteria for judgment become clearer.

• Google TPU7x documentation: 192 GB of HBM per chip, approximately 7.37 TB/s, and states that HBM may still be a bottleneck
• NVIDIA Blackwell official page: Expands up to 576 GPUs with 5th generation NVLink, highlighting 130TB/s GPU bandwidth in NVL72
• Marvell official announcement: Custom HBM architecture claims up to 25% additional compute area, 33% memory increase, and 70% interface power savings
• Groq official page: Brings to the forefront the need for a custom silicon stack dedicated to inference, not just GPUs

In the end, the difference is this. NVIDIA has a strong tendency to bundle large GPUs with a huge connection network, Groq is pushing a chip specialized for inference, and Google appears to be moving towards further segmenting the memory hierarchy based on TPU. In other words, the axis of competition is shifting from “Who makes the bigger GPU” to “Who can get around the memory wall smarter?”

6) Step-by-step execution method

Key takeaway: What the team should do now is not wait for a new chip, but first decompose whether the current inference bottleneck is compute or memory.

1. Divide the workload into two.
   Distinguish whether it is prefill-centered or decode-centered. Long interactive services usually have larger decode bottlenecks.
2. View memory metrics separately.
   Don't just look at GPU utilization, measure HBM usage, KV cache size, and token throughput by batch size.
3. Reduce data movement path.
   Check model partitioning, cache offloading, prompt compression, and session expiration policy together.
4. Tune request patterns before hardware.
   It is easy to reduce memory pressure by separating paths where short responses are important from those that require long inferences.
5. Test alternative architectures on a small scale.
   Compare GPU single-path, TPU-path, and inference-specific providers in the same evaluation set.

#Minimum operational inspection questions
1. Is the p95 delay time of our service due to prefill or decode?
2. When the batch is increased, GPU utilization increases, but why does the cost per token not decrease?
3. What percentage of total memory does the KV cache eat up?
4. What percentage of total requests are from long context users?
5. Will adjusting the cache expiration policy reduce HBM pressure?

Many teams make mistakes here. We expect that costs will automatically improve as hardware generations change. However, in practice, it is often more effective to tinker with request patterns, cache policies, and model routing first.

7) Pitfalls and mistakes

Key takeaway: Failure of inference infrastructure often comes from looking at memory bottlenecks too late rather than from lack of computation.

• Mistake 1: Thinking that if GPU utilization is high, it is being used well
  Prevention: Cost per token, decode speed, and memory pressure must be considered together. Recovery: Re-interpret the bottleneck by adding batch size and KV cache metrics.
• Mistake 2: Treating long context requests with the same path as regular requests
  Prevention: Isolate long context-only paths or high-memory nodes. Recovery: Add routing rules to distribute HBM pressure
• Mistake 3: Comparing vendors with FLOPS numbers alone
  Prevention: Look at memory capacity, bandwidth, inter-chip connectivity, and software maturity together. Recovery: Recreate the scorecard and include the inferential workload metric:
• Mistake 4: Scaling agented workloads without a cache policy
  Prevention: Design cache lifetime based on session length and reuse rate. Recovery: Lower costs by separating old sessions and long context requests.

8) Strengths and limitations

Key takeaways: Memory-centric design is right for the inference era, but not every team needs to follow a dedicated architecture right away.

• Strengths: Targets real-world inference bottlenecks more accurately, with significant room for improvement in throughput relative to power and rack space.
• Strengths: Particularly meaningful for memory-intensive workloads such as long context, multimodal, and agentic services.
• Limitations: Requires hardware and software co-optimization, general-purpose ecosystem may follow late.
• Counterexample: Short response-centric, small model-centric services may still be simpler and cheaper to run on regular GPUs.

So my recommendation is simple. Teams operating large inference services must now have an indicator system in terms of memory bottlenecks, but for small services, it is better to modify request design and cache policy rather than replace hardware.

9) Points to study more deeply

Key takeaways: The next step in learning is not memorizing chip names, but understanding operational metrics that explain why inference is becoming memory intensive.

• Impact of KV cache size and long contexts on actual cost
• Difference between HBM bandwidth and chip-to-chip interconnect on decode performance
• How to compare general-purpose GPU, TPU, and inference-specialized chips using the same evaluation set
• How to design session reuse and cache policies in agent-type workloads

10) Execution Checklist + Author’s Perspective

Key takeaway: The default posture of infrastructure teams in the inference era is not to wait for bigger chips, but to numerically check and bypass memory walls.

• Are we separating the bottlenecks of our service into prefill and decode?
• Do you collect HBM usage and KV cache metrics in addition to GPU or TPU utilization?
• Can long context requests and regular requests be routed through different paths?
• When comparing hardware, are you looking at memory capacity, bandwidth, and interconnect in addition to FLOPS?
• Are the cache expiration and reuse policies for agent-type sessions documented?
• Have you optimized the current request pattern and cache policy before introducing new hardware?

Definition of Done: Once the team has numerically identified whether the current inference bottleneck is computation or memory, and has documented an operational strategy for separating long contexts from regular requests, the first round is ready.

My recommendation is clear. This Google/Marvel news should not be read as news of a new chip, but as a signal that the infrastructure evaluation standard has already shifted to focus on memory. Therefore, it is better for mid- to large-sized AI service teams to first fix memory bottleneck observations, cache policies, and request routing systems. Conversely, if the service is still small, it is a priority to accurately measure bottlenecks in the current stack rather than being swayed by new chip news.

Reference material

• AI Times - Google in discussion with Marvell to develop two types of AI chips...Preparing for increased inference demand (Article date: 2026-04-20, Confirmation date: 2026-04-20)
• Google Blog - Ironwood: The first Google TPU for the age of inference (Published: 2025-04-09, Updated: 2025-04-23, Checked: 2026-04-20)
• Google Cloud Docs - TPU7x (Ironwood) (Confirmation date: 2026-04-20)
• Marvell - Breakthrough custom HBM compute architecture to optimize cloud AI accelerators (Published date: 2024-12-10, Confirmed date: 2026-04-20)
• NVIDIA - Blackwell Architecture (Confirmation date: 2026-04-20)
• Groq - Fast, low cost inference with custom silicon (Confirmation date: 2026-04-20)