AgentSTH V7 Performance
After months of profiling agentic AI workloads and systems, we developed AgentSTH V7. This is a suite of tests that represents the types of agentic AI workloads we see in many trace runs, performing real-world tasks ranging from coding and infrastructure management to creating financial models and more. We also focus on running different shapes on processors to stress the CPU in ways that traditional benchmarks do not. Since we are not testing the LLM side here, it turns out that much of this looks very similar to how multi-tenant server CPUs are used outside agentic AI workflows.
Let us start with single-core results:
Limiting our agnetic AI workload to a single core means we are not exercising much of these workstation/server systems other than the CPU architecture. This also gives us the most architecturally pure look at agentic performance. The Xeon 658X fares the best of all of the Xeons, which is what we would expect from it having the newest CPU architecture and the best memory. It is not quite enough to top the charts altogether, as the Threadripper 9980X takes the top spot with a roughly 5% performance lead.
Meanwhile, grouping results by whether they are throughput-, coordination-, or memory-bound reveals a bit more nuance. With just 1 core in use, the Xeon 658X slightly trails the Threadripper processor in both coordination-heavy workloads and trails by a bit more still in throughput-heavy workloads. On the other hand, the 658X takes the top spot for memory-heavy workloads. This is somewhat surprising, as a single-core workload will not readily reveal the benefits of the platform’s greater number of memory channels. Either way, it gives the Xeon a roughly 14% edge here.
Next up, let us look at the agentic AI workload executed across 4 CPU cores. Often when agent harnesses are run, or perhaps better said, when hyper-scalers look at agentic AI sandboxes, micro VMs, or other ways to put multiple clients on a cloud instance, those tend to be 1-4 core, and rarely even 8 core partitions. So we want to look at that view here:
The Xeon 658X continues to lead the pack among Xeons. The gap between it and the Threadripper 9980X widens. It is now some 16% behind by our scoring results.
The dimensionally-sorted results give us a bit more insight. The 658X is falling further behind in both throughput and coordination. Meanwhile, the memory-bottlenecked results are effectively a wash, with the Emerald Rapids architecture Xeon actually taking the top spot. It should be noted, again, that the memory is more of a “mostly memory performance influenced” bucket, not a pure memory bandwidth test like STREAM.
Finally, we will scale things up once more, to 16 cores.
Interestingly, despite the growing divergence skipping from 1 core to 4, the Xeon 658X and Threadripper 9980X are actually converging again with agentic workloads spread over 16 CPU cores. The bad news for Intel is that it still does not take the top spot here, as the Threadripper system edges it out by barely a percent (and really, we are inside our margin of error at this point). Meanwhile, the newest Xeon widens its advantage over the older Xeons.
Going to the grouped dimensions finds a few different things going on. The Xeon is still trailing in throughput; however, it is now leading in the coordination-bottlenecked benchmarks. 16 cores is larger than a single AMD CPU core CCX domain, so this is likely the benefit of the Xeons’ mesh fabric stretching its legs now that we are working with a larger number of CPU cores. The platform’s larger number of memory channels is also making itself more felt, as the 658X tops the charts while the 4-channel Threadripper is now in last place.
Pulling all of this data together, here is a look at how well the performance of each CPU scales in AgentSTH V7 starting from 1 CPU core.
None of the chips here gets particularly close to perfect scaling. Though by the time we get to 8 CPU cores, the Xeon 658X is at least the top dog among the Xeons. It is not, however, scaling better than the Threadripper processor, which sees the largest performance gains going from 1 CPU core to 8.
Anyone know why core 0 is such an outlier in the latency graph?
Not sure why anyone would pick this over AMD’s offering. We are not in 2006 anymore where games or apps would run slower on AMD due to software optimisation.
@justsomeguy
I have a suspicion it’s power management-related. But we admittedly haven’t had a chance to do any further diagnostics on the issue to properly isolate it.
It is good to see Intel somewhat back in the game.
Quite possible in the workstation market we are approaching a point where gains in CPU just really do not matter all that much. Can you really keep 30, 60, or 100 cores busy all the time? That is an immense amount of compute (of that sort), and for quite a lot of use we are likely to see diminishing return.
The next episode may be more about hosting one or more GPUs or LLM-custom PCIe cards, and the bandwidth to shuffle data to memory of across the PCIe bus. Need lots of PCIe lanes … but maybe not so much more CPU compute.
For many of us, current high prices limit our choices. But when the AI-bubble bursts, there will be a flood of disused GPUs on the market, and lots of folk wanting to lash together rigs that can make use.