Semiconductors Lens: Why Stateful, Bursty Agents May Push CPU Capacity in Data Centers, with Google Cloud’s Mark Lohmeyer

Electronicseng Semiconductors summary: For chip, server, and supply-chain teams, the practical question is no longer whether AI inference needs accelerators; it is whether long-lived, stateful software agents create a new CPU control-plane demand. The remarks by Google Cloud executive Mark Lohmeyer in EE Times provide that angle.

Arm-based Axion CPU

From Conversation to Compound Execution

The distinction starts with the interaction pattern. A short query has bounded scope. A long-running task has branches, retries, tool calls, memory state, and possibly several people touching the same workflow. Lohmeyer framed the baseline this way: “Chat is relatively straightforward—there’s a question and an answer, and it relies heavily on inference across GPUs or TPUs,” and he added that “With agents, the nature of those interactions is changing quite a bit.”

In an agent pipeline, a high-level user intent may be split into subtasks. Those subtasks can invoke APIs, visit websites, generate code, evaluate outputs, or wait on callbacks. Those operations are CPU-facing orchestration work, while model generation remains accelerator-heavy. In other words, the CPU role expands from host management into the execution graph itself.

Bursty, Idle, Stateful: The New CPU Scheduling Problem

The key phrase for infrastructure planners is not peak compute alone. It is duration plus unpredictability. Lohmeyer said: “Agents are often bursty, and often idle,” before explaining the operational profile in full.

“They don’t do anything for [say] a day or two until you come back to them, or maybe until some API responds… It’s unique from older CPU workloads, in that it needs to be stateful over a long period of time, but is often idle during that period of time, and then when it’s not idle, it’s highly bursty.”

For data-center operators, that profile breaks a simple capacity model. A service may be idle but cannot be treated as disposable, because context may need to survive days or weeks. A burst may arrive without warning because an external system replies or a user reopens a session. This is why “Demand is already very high and is growing rapidly,” still has to be answered with cost discipline: “So, how do we provide the infrastructure that can support that growth and do it cost-effectively?”

Semiconductor Implications: Core Density, Storage Attach, and Platform Choices

From a component and supply-chain perspective, the workload shape can influence CPU selection. Agents need enough general-purpose compute to manage code execution, API access, application logic, and state coordination. They also need low-latency access to context and storage so reloads are fast. The image above identifies one example hardware context: an Arm-based Axion CPU. The mention is relevant because Arm, Axion, and cloud-specific CPUs are part of the broader question about how data-center silicon aligns with workload-optimized infrastructure.

Compute Split, Locality, and CPU Share

Lohmeyer separated the new workload of running the agent from the pre-existing workload of accessing websites and APIs. The second category already existed, but its consumer was usually human. Agents can multiply sessions and requests quickly. “Agentic workloads will have a meaningful impact on both of those elements,” he said.

Latency placement also changes the engineering problem. “Ideally, you’d like to have those agents and their context, the applications and data, relatively close to each other from a latency perspective, for performance reasons,” he said. However, enterprises remain distributed, with public cloud, private cloud, and multiple clouds. That keeps inter-data-center networks and high-performance fabric relevant.

He summarized the infrastructure requirement: “Dynamic, stateful agents will need to have a new type of computing infrastructure that supports them,” adding, “They’re accessing all the existing apps and data, so the demand there is growing, and all this needs to be connected together through the network. Google is investing in all of these elements.”

CPU Versus Accelerator Balance

The exact CPU-to-accelerator ratio is still moving. As organizations shift from training foundation models to agentic inference, the CPU portion rises. “We expect both [CPU and TPU/GPU] workloads to continue to grow rapidly, but the proportion of CPUs is going from a relatively small portion to a meaningful portion, and we think that will continue to evolve as agents expand into new use cases and new environments,” he said.

For hardware planners, that means TPU/GPU capacity should not be viewed as replacing CPU capacity. It may be viewed as part of a mixed pool where CPU-side orchestration becomes large enough to influence core counts, memory bandwidth, storage attachment, and virtualization overhead.

Co-design Across the Stack

Because the workload is mixed, single-chip answers are unlikely to be sufficient. Lohmeyer pointed to workload-optimized infrastructure: “Google Cloud has always had a philosophy of workload-optimized infrastructure,” and said, “This is more important than ever in the agentic era.”

A practical mechanism is hibernation. If an agent is idle, its state can move to storage, freeing CPU resources for other agents. When the agent wakes, storage must return the state quickly. “This requires a new type of orchestration in software, and in this environment, having really high-performance storage that’s attached to that server, so I can restart it quickly, is important,” he said.

MicroVMs as a Safe Packing Boundary

Google Cloud uses microVMs, described as a slimmed-down virtual machine, to isolate agents packed onto the same server. The density example is concrete: rather than 100 traditional VMs, the same hardware may support 1,000 microVMs. That changes the calculus for cloud CPUs, hypervisors, memory bandwidth, and secure execution.

“Security and governance for agents is absolutely critical, and Google is investing a lot in every layer of the stack,” he said. The virtualization layer matters because many agents may touch APIs, code, and external websites on the same substrate. A smaller attack surface and streamlined microVM feature set are important; egress controls and network-level gateways are also part of the design.

Comparison: Conversational Inference Versus Agentic Workload

Dimension	Conversational Inference	Agentic Workload	Engineering Implication
Session behavior	Short question and answer	Long-running, multi-step intent	State persistence matters more than one-shot latency
CPU role	Host and inference coordination	Orchestration, tools, code, APIs, and context management	CPU share may rise as agents expand use cases
Utilization pattern	Closer to request-response bursts	Idle for days, then highly bursty when triggered	Capacity models need hibernation and rapid restore
Data placement	Model and prompt path dominate	Agent, apps, data, and services should be close for latency	Network locality becomes a hardware and cloud planning input
Isolation	Service boundary is simpler	Many agents can be packed on one substrate	MicroVM boundaries and egress controls shape CPU overhead

Common Misconceptions

• That accelerators make CPUs less important. In agentic systems, CPUs carry orchestration, code execution, API access, and state handling around the accelerator inference path.
• That idle sessions cost nothing. Long-lived sessions need retained context and fast restore, which affects memory, storage, and scheduler design.
• That agents are only chatbots with longer prompts. A real agent can create subtasks, wait on external systems, and act across many applications.
• That security is only an application-layer problem. Packing many agents onto shared hardware pushes isolation into virtualization, microVMs, gateways, and egress controls.
• That a fixed CPU-to-accelerator ratio is enough. The ratio is workload-dependent and is still evolving as agents move into more environments.
• That locality is optional. Agents benefit when their context, applications, and data are close enough to limit latency, even if the overall enterprise remains distributed.

FAQ

Why do CPUs matter if inference runs on GPUs or TPUs?

Because the agent path includes more than token generation. Tool calls, API access, code execution, website interaction, and state management are CPU work. The accelerator path still matters, but the overall compute mix changes.

Does “bursty” mean the CPU can be shut off?

Not in the simple sense. The source describes a pattern where agents may be idle for a day or two, then become highly bursty. The state must remain available, so the platform needs orchestration and restore mechanisms rather than ordinary idle shutdown assumptions.

Why is storage performance part of the CPU discussion?

Because hibernating an idle agent means moving state to storage and restarting it quickly when needed. High-performance storage attached to the server becomes part of the CPU utilization strategy.

Why are microVMs relevant for agent infrastructure?

MicroVMs can create a secure boundary between many agents on one server. They are described as slimmer than traditional VMs, with a smaller attack surface and performance benefit. The example density shift is from 100 traditional VMs to 1,000 microVMs.

Does Arm-based Axion CPU answer every agentic workload?

No. It is an example of workload-optimized data-center silicon. The larger point is that agents may need different CPU, memory, storage, and network combinations depending on how state, bursts, and tool access are orchestrated.

What should engineers watch first when planning for agents?

Watch the split between accelerator inference and CPU orchestration. Then measure session lifetime, restore latency, storage bandwidth, isolation overhead, and network distance to applications and data.

Action Recommendations for Engineers and Procurement

1. Map which agent steps are model inference, API access, code execution, and state management. Do not collapse them into one AI capacity line.
2. Build a CPU share metric for agentic inference, not only for host management. The CPU proportion may become meaningful as use cases expand.
3. Design session models around days or weeks of idle state, not minutes of request-response traffic.
4. Validate restore latency from storage. Hibernation is useful only if state reload is fast enough to protect user experience.
5. Compare traditional VM boundaries with microVM boundaries. Ask what the security surface, performance overhead, and achievable density become.
6. Evaluate network locality between agents, applications, data, and external services. Distributed systems will still need high-performance interconnection.
7. Include egress controls and governance in infrastructure requirements, because agents can call APIs, visit websites, and run code.
8. Use TPU/GPU and CPU capacity as a coupled planning problem. One may grow, but the other should not be assumed flat.
9. Review platform choices such as Arm-based Axion CPU as one possible workload-optimized path, not a universal rule.

The strategic takeaway for the Semiconductors column is that agent infrastructure is not just another AI application; it changes where compute is consumed. As Lohmeyer put it, “This will allow our customers to scale their business, but do so with a good business model, good profitability, but without having to sacrifice performance as they are scaling,” and “These capabilities will be very important going forward.”