From Olympus to Vera: NVIDIA’s Scalable Coherency Fabric 2
Beyond the Olympus core, we have the larger Vera chip design, which is another area where NVIDIA has been light on the technical details up until now.
One point of differentiation for the company has been that Vera’s CPUs are all on the same piece of silicon. While NVIDIA employs chiplets on Vera, they are disaggregated based on function: CPU cores, I/O, and memory controllers. This is rather distinct from current AMD and Intel chip designs, both of which split their CPU cores over multiple chiplets – up to a dozen, in the case of AMD.
But how do all of those CPU cores talk to each other? We finally have an answer to that with a look at NVIDIA’s scalable coherency fabric (SCF). Which, at a high level, effectively splits up a Vera compute chiplet into four quadrants/clusters.
Each quadrant, in turn, is anchored by a single coherency switch node (CSN), which is responsible for routing traffic between the attached CPU cores, L3 caches, memory controllers, and I/O controllers. The CSNs are also attached to each other for when traffic needs to flow to CPU cores or memory controllers that are not part of the local cluster.
As laid out in NVIDIA’s diagram, each CSN is paired up with two clusters of CPU cores (an interesting 11 cores each), as well as two blocks of L3 cache (20.5MB/each). Finally, each CSN has a single Memory Switch Node (MSN) leading to the respective memory controller chiplet adjacent to the MSN, as well as a Bridge Switch Node (BSN) which leads to I/O. Interestingly, because Vera only has a single I/O chiplet and a single NVLink-C2C interface, both are shared by pairs of BSNs. Meanwhile this also means that a given BSN is attached or one or the other, but not both – meaning that the number of hops traffic has to go through to get from a CPU core to either of the I/O interfaces will depend on which interface it is closer to. Unlike local memory accesses, not all of the cores are equidistant from the I/O exits.
Overall, the SCF offers a massive amount of bandwidth. According to NVIDIA, there is 3.4TB/second of bandwidth along the fabric, just over 2.8x the amount of bandwidth for the complete external memory subsystem. As a result, NVIDIA says that the fabric is fast enough to allow the chip to effectively function as one large CPU despite the clustered organization, with each CPU core able to reach into other L2 cache pools or MSNs without a significant latency penalty. In other words, allowing NVIDIA to employ a distributed cache (and distributed coherency) architecture while still maintaining latencies similar to smaller monolithic designs.
NVIDIA’s own cache latency diagram seems to back this up. While there is clearly still some benefit to staying local, we do not see the usual “block” patterns of large latency differences which we see in other CPU designs with multiple CPU clusters. Most cores seem to be able to reach another within 40ns, with only half-a-dozen core pairings exceeding 80ns. Though in some respects NVIDIA has an easier job here with Vera, as the 88 core CPU design means that there are far fewer CPU cores than the massive 128+ core designs employed by AMD and Intel.
It looks to me like someone set the colour scale on the core to core latency table so the entire table was green. Even so, the chip seems well designed for tightly coupled parallel workloads. Since AI is different than the cloud-style micro-service throughput targeted by existing processors, I think it makes sense for Nvidia to fabricate their own.