Anthropic-Fractile Commentary: As the demand for inference explodes, why memory movement costs and supply chain negotiation power must be considered before purchasing GPUs

Publication date: 2026-05-03 | Category: Development information

AI Times reported on May 3, 2026 that Anthropic is discussing introducing an inference chip with British AI chip startup Fractile. On the surface, it may seem like just another investment/supply contract news. However, there are other signals that are more important in practical terms. Now, rather than buying more GPUs for learning, frontier model companies are starting to think about how to distribute inference to which chips and how to secure supply chain negotiation power

This article is a commentary for AI platform team, infrastructure engineers, CTO, and model serving operations personnel. The scope is Anthropic's diversification background, why inference-specialized chips such as Fractile are attracting attention, and by what criteria GPU, TPU, Trainium, and dedicated inference chips should be used. This is not an article that predicts the success of a specific company's investment or determines the contents of a non-disclosure agreement.

1. One-line problem definition

Key takeaway: Bottlenecks in the inference era are closer to memory movement, response latency, supply intensity than to FLOPS.

Large model services like Claude no longer operate solely on lab benchmarks. As numerous users ask questions simultaneously and the number of long-term requests such as code generation and task automation increases, the question for infrastructure teams changes from “Can we train a model?” to “Can we send out responses cheaply, quickly, and reliably?” This change is evident in the fact that Anthropic is considering new inference chip companies such as Fractile even after signing a multi-gigawatt TPU contract with Google and Broadcom on April 6 and announcing an up to 5GW compute expansion agreement with AWS on April 20.

Scope of application: Large-scale inference serving, code/agent workloads, international region expansion, cost-sensitive LLM products. Not applicable: Small-scale single GPU experiments, on-device models, teams with small inference traffic yet requiring hardware routing.

2. First, conclusion

Key summary: In the future, the team will design for training, general-purpose inference, and low-latency inference rather than “one best chip”. It is advantageous.

• The right team for you right now: Teams whose monthly inference costs are rapidly increasing and who are already feeling regional service delays or supply chain concentration risks
• Team that is still too much: Product-market suitability has not yet been verified, and the inference request volume is small, so one or two types of GPUs are sufficient
• My judgment: Anthropic's Fractile review is not simple chip shopping, but It is difficult to handle the surge in inference demand with only an Nvidia-centered procurement structure It is better to view it as an operational signal. That’s right.

Therefore, I recommend the order below to most AI service teams. 1) First, separate the inference workload into delay-sensitive and throughput-sensitive types, 2) lay GPU/TPU/Trainium as the basic floor, and 3) test the dedicated inference chip on top of that only in specific sections. It is risky to rely on one startup chip for the entire service from the beginning, and conversely, if you rely on GPU alone for everything, you may be at a disadvantage in terms of cost and negotiating power.

3. Decomposition of core structure

Key takeaway: Inference infrastructure should be divided into training layer, general-purpose inference layer, low-latency inference layer, and overflow layer, not just one chip.

3-1. Training layer

This is still focused on large GPUs, TPUs, and large clusters like Trainium. Anthropic announced on April 20, 2026 that it is already using more than 1 million AWS Trainium2 chips, and that it will secure nearly 1GW of additional capacity based on Trainium2·3 by the end of the year. The key to the training layer is extreme flexibility and massive parallelism.

3-2. General purpose inference layer

Most product requests go here. It is a mixed layer of long context, code generation, document summaries, and general chat responses. The reason why GPU, TPU, and Trainium are still powerful is because has a wide range of response even when the model architecture changes, and the software ecosystem is mature

3-3. Low-latency inference layer

Requests where First Token Delay (TTFT) and Token Throughput per Secondare very important require different criteria, such as real-time chat, voice chat, and agent mid-loops. Fractile claims on its homepage that it will use a structure that physically mixes memory and computation, making frontier model inference up to 25 times faster and reducing the cost to one-tenth. In this layer, minimizing memory movement is more important than versatility.

3-4. Overflow/Negotiation Layer

This layer is closer to procurement and operations strategy than technology. Having more than one provider makes unit price negotiation, regional distribution, and failure bypass easier. This is why Anthropic mentions AWS, Google/Broadcom, and NVIDIA GPUs, but is also looking at additional startup chips. In other words, chip diversification is both a performance experiment and a means of securing negotiation power.

4. Description of design intent

Key takeaways: Anthropic's choice is not so much a “post-NVIDIA statement” as it is a move to Ready for multi-layer routing tailored to growing inference demands It’s close.

Anthropic has publicly acknowledged infrastructure pressures going into 2026. In a notice on April 20, it was revealed that annualized sales (run-rate) exceeded $30 billion, a sharp increase from about $9 billion at the end of 2025. The number of business customers spending more than $1 million annually has exceeded 1,000. With speeds like this, “just buy more GPUs” won’t solve the problem. This is because as the number of customers increases, the risk of failures during peak hours, capacity shortages in specific regions, and supplier price increases also increases.

Fractile are attracting attention. Fractile explains that it aims for both low latency and high throughput by physically intersecting memory and compute. In simple terms, this means reducing the cost of traveling back and forth between distant memory and chips each time the model answers. Inference is repeated much more frequently than training, so getting this structure right can have a significant impact on overall service cost and delay.

However, what is gained and what is given up is also clear.

• What you get: Potential cost savings for specific inference segments, faster responses, and stronger supply chain bargaining power
• What you give up: Versatility, software compatibility, mass production verification stability
• Practical interpretation: It is safer to first view startup chips not as full replacements but as auxiliary axes that reduce specific bottlenecks.

5. Evidence and Comparison

Key takeaways: The basis for comparison is not model benchmarks, but Delays, supply security, flexibility, unit price negotiation power

The evidence can be organized as follows.

• Anthropic Announcement (2026-04-06): Signed a multi-gigawatt next-generation TPU contract with Google and Broadcom, and announced that capacity will begin in 2027.
• Anthropic announced (2026-04-20): Announced up to 5GW new compute contract with AWS, nearly 1GW Trainium2·3 capacity by year-end, and over 1 million Trainium2 units in use.
• AI Times (2026-05-03): Anthropic is discussing introducing an inference chip with Fractile, and said that SRAM-based architecture and minimal data movement are the key differentiators.
• Fractile homepage (checked 2026-05-03): They claim to make frontier model inference up to 25 times faster and at 1/10 the cost, and that token processing is increasing more than 10 times per year. Explain.
• Financial Times requoted article (Fractile, 2025-03): Morgan Stanley believed that more than 75% of future U.S. data center power and computation demand could be inferred, and Barclays predicted that the frontier AI inference CAPEX would be 2025. It was estimated that it could increase from $122.6 billion to $208.2 billion in 2026.

What these numbers say is simple. From now on, How to classify inferences in detail and send them to the appropriate silicon will become a more important competitive advantage rather than who buys a larger GPU cluster.

6. Actual operation flow / step-by-step execution method

Key takeaway: Chip adoption is a matter of creating workload classification table and routing rules before purchasing.

1. Split your inference requests into three categories:
   Examples: real-time chat/voice, code/agent mid-loop, batch summary/report generation. Even with the same "inference", the required delay and throughput are different.
2. List your TTFT and tokens per second goals in each category.
   Example: Real-time chat might have a TTFT of 300ms or less, batches might have a 3seconds or less reduction, and agent loops could have a 20% reduction in overall completion time. There is
3. Measure the current cost per chip and the cost of failure together.
   Don't just look at the token price, you should also look at the churn costs due to peak hour failures, lack of regions, and worsening delays.
4. Test dedicated inference chips from 5 to 15% range.
   It is safer to start with the most delay-sensitive or highly repeatable requests and leave the rest to the GPU/TPU.
5. Document vendor routing rules.
   For actual operation, write down which request goes to which chip and where to fall back in case of failure.

#Example: Inferred Routing Draft
workloads:
  - name: realtime-chat
    ttft_ms_max: 300
    throughput_priority: high
    preferred_silicon: inference-optimized
    fallback: gpu-general
  - name: coding-agent-loop
    ttft_ms_max: 800
    throughput_priority: medium
    preferred_silicon: trainium-or-tpu
    fallback: gpu-general
  - name: nightly-batch-summary
    ttft_ms_max: 5000
    throughput_priority: low
    preferred_silicon: cheapest-available
    fallback: gpu-spot

To put it simply as a novice developer, chip selection ultimately comes down to load balancer policy. Rather than putting all your requests into one piece of equipment, it will be much easier to decide if you write down which requests are most important first.

7. Pitfalls

Key summary: Rather than overconfidence in new chips, a common failure is Blanket application without distinction of workload

• Mistake 1: Calculating the cost of inference only in GPU time
  Prevention: Calculating the cost together with response delay, lack of regions, and vendor dependency
  Recovery: Past 30-day failures and peaks Recalculate the total cost by reapplying the time data.
• Mistake 2: Leaving the startup chip as a company-wide default right away
  Prevention: Pilot from a region with high repeatability and low failure impact.
  Recovery: Reduce the pilot scope to one workload and Force a fallback.
• Mistake 3: Making decisions based solely on benchmark numbers rather than versatility
  Prevention: Look at model change speed and software porting cost together.
  Recovery: 2-3 types of actual service models Re-measure and suspend supported unstable routes.
• Mistake 4: Not creating a fallback path despite diversification
  Prevention: Insert the fallback order for each chip, region, and cloud in the operating document.
  Recovery: Disability training Run game day to verify automatic fallback.

8. Strengths and Limitations

Key takeaways: Specialized inference chips are cost- and latency-resistant, but they cannot replace the entire general-purpose AI infrastructure.

• Strengths: In certain inference patterns, reduced memory movement and low-latency response can provide significant benefits.
• Strengths: Can increase negotiation power with large suppliers, regional distribution strategy, and supply chain resilience.
• Limitations: New vendors may lack mass production, drivers, operational tools, and failure response experience.
• Limitations: If the model structure changes quickly, certain chip optimizations may become a drag.
• Counterexample: Teams with small traffic and still in the experimental phase may want to look at model quality and product conversion first, rather than a multi-layer silicon strategy.

9. Points to study more deeply

Key takeaways:The next step is to compare chip brands.Quantifying the inference patterns of our servicesThis is what I do.

• Which requests in our service are TTFT critical and which requests are total completion time critical
• What percentage of inference requests repeat the same pattern in nature
• How much does chip porting cost when model architecture changes
• In case of regional failure or supply shortage, can it be bypassed to another chip within a few days?
• Are you distinguishing whether the cost reduction is token price or improved reliability during peak hours?

10. Reference

• AI Times - Antropic, on the verge of securing computing, considers partnership with British chip newcomer (Created date: 2026-05-03)
• Anthropic - Anthropic expands partnership with Google and Broadcom for multiple gigawatts of next-generation compute (Created date: 2026-04-06)
• Anthropic - Anthropic and Amazon expand collaboration for up to 5 gigawatts of new compute (Created date: 2026-04-20)
• Fractile - Radically Accelerate Frontier Model Inference (Confirmation date: 2026-05-03)
• Fractile - How ‘inference’ is driving competition to Nvidia’s AI chip dominance (Financial Times requoted) (Written date: 2025-03)

11. Action Checklist + Author's Perspective

Key takeaways: Inference A supply chain strategy is successful when it documents workload classification, fallback design, and procurement negotiating power before chip adoption.

• Real-time, agent, and batch inferences were separated
• The TTFT, throughput, and cost upper limits of each workload are written in numbers
• The roles of GPU/TPU/Trainium/dedicated inference chips are divided into layers
• Documented at least one fallback path and region bypass path
• The startup chip was decided to start with a limited pilot range rather than the entire
• We created a procurement table that compares supplier unit price and failure risk together
• We decided to reexamine the repotting cost when changing the model architecture

Definition of Done: If the routing criteria for each workload, role for each chip, fallback path, pilot scope, and procurement comparison table are documented and the actual cost/delay A/B test is run in one inference section, the first preparation is considered complete.

My Recommendation: This move by Anthropic does not mean that you should follow and buy from a specific startup. Rather, it is closer to the meaning of Now, the AI ​​infrastructure team must design the inference routing and supply chain structure in more detail than the model. It is realistic to place large cloud chips on the bottom and place dedicated inference chips first in delay-sensitive sections. Conversely, if the service scale is still small, it is better to establish a workload measurement system before the chip portfolio.