KAIST AI agent power cost explanation: Why agent introduction should measure repeated calls, GPU idleness, and energy budget before accuracy

Publication date: 2026-07-05 | Category: Development information

1. One-line problem definition

Key line: The cost problem of AI agents is not that the model is expensive, but that the amount of calculation and power usage explodes as the model call, tool execution, and waiting time are repeated within a single user request.

AI Times introduced the HPCA 2026 paper by KAIST researchers on July 5, 2026 and reported that an AI agent using 70B-class LLM consumed an average of 348.41 Wh per question and consumed up to 136.5 times more energy than a simple question-and-answer generation AI. The same study states that the response time of AI agents increases by up to 153.7 times, and that GPUs can spend up to 54.5% of the total execution time waiting without computation while external tools operate.

This article is a commentary for developers, platform engineers, CTOs, and PMs who want to add AI agent functionality to their products. The scope includes power, latency, and GPU idle instrumentation of agent workloads, benchmark design, and pre-launch gates. This is not an article that reproduces the entire KAIST paper or predicts electricity rate policy.

To put it simply, for a novice developer, a chatbot is closer to a counselor who receives questions and answers once. Agents are more like workers who search, calculate, rethink, run code, and check the results. Workers may look smarter, but it costs more electricity and time.

2. First, conclusion

Key line: Agents are risky when released as “features that provide slightly better accuracy” and should be viewed as “a new workload that requires managing energy/delay/success rate per request together”

An agent is meaningful for problems that actually require multiple steps, such as complex document analysis, code modification, research automation, and shopping, reservation, and task automation. In this case, there is a basis for paying more than a simple chatbot.

Conversely, it is not recommended to attach an agent to functions that require one or two model calls, such as answering FAQs, short summaries, simple classification, or creating a set template. The improvement in accuracy is small, and only delay, power, and operating costs are likely to increase.

My judgment is clear. The agent introduction review table after 2026 should not only include task success. LLM call count, tool latency, GPU idle percentage, Wh per request, cost per successful request must be on the same line.

3. Decomposition of core structure

Key one line: The power cost of the agent increases significantly in “iteration loops” and “tool waits” rather than in the model itself.

1. User Request Layer: Users have goals such as “Find and compare data,” “Fix the code,” and “Find available reservation times.” On the surface, it's just one request, but internally it's split into multiple subtasks.
2. Planning/Inference Layer: The agent plans what to do first. Methods like ReAct, Reflexion, LATS, and LLMCompiler repeat planning, action, observation, and replanning.
3. Model call layer: Call LLM again at each step. A simple chatbot may end with a single call, but the agent makes multiple calls, from pre-search planning, post-search judgment, error recovery, and final answer.
4. Tool execution layer: Search, calculator, code execution, environments like WebShop, API calls are interjected. At this point, the GPU is on waiting for the next LLM call, but may not be able to work sufficiently.
5. Infrastructure measurement layer: Power meter, GPU utilization, token count, wall-clock latency, tool latency, and retry count must be recorded together to account for actual costs.

What makes the KAIST study important is that it defined agents as a data center workload problem rather than a model accuracy problem. An agent is not a single line of prompts, but an execution pattern that the server and GPU must repeatedly process.

4. Description of design intent

Key line: Agent architecture is a choice to “spend more reasoning at runtime” to solve difficult problems, at the cost of power, time, and operational complexity.

Why does the agent think several times instead of answering at once? For complex problems, it is often more accurate to check intermediate results and go back if incorrect, rather than generating the answer right away. Even when a person writes a report, it goes through the following steps: data retrieval, drafting, review, and revision.

What this design gains is flexibility. Agents can search for things they don't know, offload calculations to external tools, and view and re-correct the results of code execution. What you give up instead is predictability. Even for the same user request, it is difficult to know in advance how many loops it will run, how long which tool will take, and how many retries will occur after failure.

The KAIST study's figures of 348.41Wh, 136.5x, 153.7x, and GPU idle up to 54.5% show this trade-off in numbers. “Smarter answers” ​​aren’t free. Operations teams must determine whether a 3% improvement in accuracy justifies a 10x increase in energy.

5. Evidence and Comparison

Key one-liners: Agent adoption decisions should be compared to chatbots, fixed workflows, and batch processing.

AI Times reported that KAIST researchers introduced that data center power demand could reach approximately 198.9GW, assuming 13.7 billion AI agent requests per day. Rather than a prediction of the actual future, these numbers should be read as a warning sign showing how power bottlenecks could become larger if agent calls were scaled up to the scale of web searches on current methods.

AgentBench GitHub repository is also of practical importance. The repository provides ReAct, Reflexion, LATS, LLMCompiler implementations and hotpotqa, webshop, math, and humaneval workloads, and assumes an OpenAI-compatible LLM server and vLLM-based endpoint. In other words, you have a starting point to turn your research results into internal benchmarks for your team, rather than just consuming “number of articles.”

6. Actual operation flow / step-by-step execution method

Key line: Before releasing the agent function, energy/delay profiling experiments, at least small, must be run separately from accuracy evaluation.

1. Choose 20 to 50 representative tasks.
   Don’t just include easy questions, but include long documents that real customers will ask, tool calls that can fail, and tasks with ambiguous conditions.
2. Set the baseline.
   Execute the same task using a simple chatbot, fixed workflow, and agent method, respectively. Don't just look at success rate, look at LLM calls per request and p95 delay together.
3. Leaves traces
   For each step, llm_call_start, tool_call_start, tool_call_end, Leave events for llm_call_end, retry, final_success.
4. Attach GPU and power indicators.
   If it is your own server, collect GPU utilization, power draw, and memory usage. If you only use an external API, even if you cannot see the actual Wh, the number of calls, number of tokens, delay, and retry rate are used as proxy metrics.
5. Sets the energy budget gate.
   Example: Release criteria in numbers, such as 8 or fewer LLM calls per successful operation, p95 45 seconds or fewer, tool wait 40% or fewer, and 2 or fewer failure retries. Leave it.

#agent_energy_profile.yaml example
workload: "customer-research-agent"
baselines:
  - simple_chat
  - fixed_workflow
  - react_agent
measure:
  - task_success_rate
  - llm_calls_per_task
  - tokens_per_successful_task
  - p50_latency_seconds
  - p95_latency_seconds
  - tool_wait_ratio
  - gpu_idle_ratio
  - watt_hours_per_task
release_gate:
  min_success_rate: 0.82
  max_llm_calls_per_task: 8
  max_p95_latency_seconds: 45
  max_tool_wait_ratio: 0.40
  max_retry_count: 2

AgentBench, you need a Python 3.13.9 environment, OpenAI-compatible LLM server, vLLM endpoint, and agent type and workload settings of config.yaml. Instead of trying to replicate the size of the paper from the beginning, it is more realistic to start with the team's 20 actual feature candidates.

7. Pitfalls

Key one line: Agent operation failure is not due to a bad model, but because iterative loops and waiting times are not viewed as pre-launch numbers.

1. Trap: Release only after seeing task success.
   Prevention:Number of LLM calls per successful task, number of tokens, next to success rate, p95 Always add delay and retry rates.
   Recovery: Reclassify failure causes into model quality, tool failures, too many loops, too many contexts and reduce the most expensive failures first.
2. Pitfall: Tool call time is considered free.
   Prevention: Set tool wait ratio as a separate indicator. GPU waiting time is also an infrastructure cost.
   Recovery: Slow tools are dequeued asynchronously, and cacheable search and calculation results are reused.
3. Pitfall: Let it “solve itself” with no loop limit.
   Prevention: iteration_limit, reflection_limit, max_replan, Leave different upper limits, such as max_depth, for each function.
   Recovery: If it fails more than a certain number of times, it is passed on to human approval or fixed workflow rather than a small model.
4. Pitfall: I think it has nothing to do with the power problem because it uses an external API.
   Prevention: Even if you can't see Wh directly, the number of tokens, number of calls, Latency and retry rates are viewed as alternative indicators of cost and power.
   Recovery: When we detect provider-specific cost spike signals, we enable model routing, caching, and batch processing to immediately lower them.

8. Strengths and Limitations

Key line: An energy metering perspective makes agent adoption decisions realistic, but not every team needs to have a data center-grade power meter.

Strengths. First, rather than giving a feel for why an agent is slow or expensive, we can explain it in step-by-step numbers. Second, accuracy improvements and cost increases can be compared in the same table. Third, you can find invisible waste by just looking at the token price, such as tool waiting and GPU idling.

limit. The 348.41Wh in the KAIST study is a value based on specific research conditions, 70B level LLM, agent implementation, and workload. This does not mean that agent requests for all products use the same power. It is difficult for teams that only use external APIs to directly measure GPU power and idle rate. Additionally, even if more energy is used, it can be justified business-wise if high value-added tasks are successfully handled.

So the point is not “Agents should not be used.” “Use agents only for tasks where there is a reason to use them, and measure the cost of iterative inference before release.”

9. Points to study more deeply

Key line: The next learning order is agent pattern, test-time scaling, GPU utilization, and benchmark reproducibility.

• ReAct: Basic agent pattern that alternates between thinking and acting. As tool usage increases, so does the number of calls.
• Reflexion: This is a structure where you try again through self-reflection after failure. Accuracy can increase, but the cost increases without reflection_limit.
• LATS: This method explores multiple candidate actions. Search depth and number of samples add up quickly.
• LLMCompiler: An approach to structuring and re-planning work plans. Managing max_replan and chat history is important.
• GPU Idle Percentage: Time when GPU is on but not calculated. On agents, this value can be large due to waiting for external tools.
• vLLM OpenAI-compatible server: It serves an open source model like an API and is an execution base that can be used as a reference when running your own benchmark.

10. Action Checklist + Author's Perspective

Key line: I think release approval for agent features should be changed to “success rate/delay/energy budget contract” rather than “accuracy demo”.

• The same task was compared with a simple chatbot, fixed workflow, and agent method.
• Records the number of LLM calls and tokens per successful operation.
• View tool wait ratio and retry count on the dashboard.
• If it is your own server, GPU utilization, memory usage, and power draw are collected together.
• Loop upper limits such as iteration_limit, reflection_limit, max_replan, and max_depth are determined for each function.
• Even if a success rate of more than 85% is achieved, the release is postponed if p95 is delayed or the cost exceeds the budget.
• When limit is exceeded, fall back to one of the following: small model, cache, batch queue, human approval, or fixed workflow.
• We look at 20 failure samples every week and separate “model problems” from “operational problems”.

Definition of Done: If the agent function candidate satisfies all the success rate, p95 delay, number of calls per successful task, retry rate, tool wait ratio, cost, or Wh/task criteria in the 30-day test, and if the automatic stop or fallback path operates when the criteria is exceeded, it is considered ready for the first release. View.

My recommendation: Prioritize agents only on tasks that are complex enough for customers to pay for them. For tasks that can tolerate slightly higher latency, such as internal automation or research, it is better to start with asynchronous batches. Conversely, attaching unlimited agents to a real-time chat UI is risky in terms of both cost and user experience.

11. Reference

• AI Times, reporting on KAIST AI agent's power cost investigation (Published date: 2026-07-05, Confirmed date: 2026-07-05)
• IEEE Xplore, The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspective(HPCA 2026, confirmation date: 2026-07-05)
• VIA-Research AgentBench GitHub repository (HPCA 2026 artifact, confirmation date: 2026-07-05)
• EurekAlert, KAIST unveils hidden energy cost of AI agents (Confirmation date: 2026-07-05)
• vLLM document, OpenAI-Compatible Server (Confirmation date: 2026-07-05)