Broadcom has its new 800GbE generation out, the Broadcom Thor Ultra. What is more, we were able to see not just the OCP NIC, but also the chip that powers it live at OCP Summit 2025. What is more, this is an Ultra Ethernet Consortium (UEC), a big leap toward next-generation networking focused on bringing high-performance Ethernet to displace technologies like InfiniBand.
Broadcom Thor Ultra 800GbE NICs Launched
First off, we were able to see an OCP NIC 3.0 at OCP Summit 2025. You will notice a lot of heatsink area here because between an 800G NIC chip and an 800G optical module, these can use a decent amount of power.
We can see that this NIC is a SFF with Ejector Latch design and has a single 800Gbps port. We did not get to ask, but this looks like an OSFP112 port with up to 112G PAM4 SerDes.
On the other side, we have the OCP NIC 3.0 connector. Something that is a bit different with this one is that this is a PCIe Gen6 x16 NIC since it is an 800GbE NIC. Since the NVIDIA B300 PCIe is not officially launched yet, the only PCIe Gen6 x16 cards we have right now are the NVIDIA ConnectX-8’s that just arrived in the studio and are earmarked for use in our new Keysight CyPerf box hopefully bringing our traffic generation capabilities from 1.6Tbps today to over 2Tbps.
A magical Broadcom briefcase also yielded the BCM57708 chip for Thor Ultra.
Key features are packet multipathing, out-of-order packet delivery, selective retransmit, and congestion control.
On the packet level multipathing side, this helps load balance the network.
The out-of-order packet delivery is another technology that helps as packets take different paths through the network.
If you saw our recent Broadcom Tomahawk Ultra at Hot Chips 2025 piece, the programmable congestion control will make a lot of sense. To understand this a bit more, it might actually be worth checking that piece out as the Thor Ultra is designed to work with the Tomahawk line.
There is also a programmable pipeline that can help handle the congestion control.
800G is not just about adding new offloads. Instead, it is also about running large-scale and high-speed clusters, so many of the innovations beyond the line rate are designed for large deployments.
Final Words
Broadcom is also going to have a standard PCIe x16 card version of Thor Ultra, but it was neat to see the hardware in-person this close to launch. With NVIDIA ConnectX-8 cards arriving in the lab, and seeing 800G Thor Ultra cards in-person this week, the 800GbE generation is now upon us. What is also becoming clear is that NICs leading the charge to new generations of PCIe.
I have a real dummy old man question: OoO packet delivery – How does this work? Does it just assume the receiver will reorder all the packets once they are all delivered? Is it like a UDP type thing where it’s fire and forget, doesn’t matter if it gets there?
@James
I have to think that OoO packet delivery exists because in a fabric the source to destination isn’t always a straight line, to maximize a bunch of 400GB links with 800GB endpoints you would have many different paths to target. I imagine this is a way of sequencing packets via some header info (?) I don’t know but just a guess that packets can be sequenced like frames…they’ve just moved that up the osi model to allow for routed fabric. These nics likely have deeper buffers and the chip can re-order the packets on the fly before getting that out to the bus as a stream. There’s going to be a finite amount of OoO packets that can exist, but running into that defines that your fabric core has had something go wrong and it’s not a nic problem. But this could alleviate smaller issues of congestion.
Infiniband’s solution was always to focus upstream at the switch/fabric design level, on these sorts of things to reduce latency. This feels more like a heavy lift in silicon at the nic to solve it, but I’m no engineer. Also ethernet was designed for reliability in a less controlled environment – trying to make it work like infiniband is a battle against its purpose – but NV owns mellanox now and you work with what you have. Or someone will try to reinvent a wheel.
I’ll skip this crap, until fiber-on-chip becomes reality.
Power consumption of these things is insane and is solely pushed by the a cloud of morons, seeking to get to their personal Terminator first.