Today we are going to take a look at the FS N9510-64D. This is a 64-port 400GbE switch powered by the Broadcom Tomahawk 4 switch chip. In this piece, we are going to look at networking hardware that is 4-8x faster than what many are using these days. Let us get to it, and some of the challenges we faced in the process.
Getting to 400GbE the Video
Getting this working was not as simple as one may assume. While the switch ran a fairly Cisco-like Broadcom/ FS management stack, the real challenge came with getting everything working. This was an extremely expensive piece to do, and we barely got it done before our 400GbE NICs had to go to FedEx. Still, we have a little video about it:
We always suggest opening this in its own tab, window, or browser for the best viewing experience.
FS N9510-64D 64-port 400GbE Switch External Overview
The switch itself is massive. It is a 4U unit. The bottom 2U of the switch is used for the 64x QSFP-DD ports. The top 2U is used for cooling.
We are going to quickly note here that the right rack ear was damaged on the inbound shipment to STH. DHL seemingly dropped the package on its corner out of the airplane it took to get here.
The top of the switch is a giant air vent. One can see it here with the rear fans removed. This is to cool the massive Broadcom Tomahawk 4 switch chip heatsink.
On the left side of the switch, there are status LEDs, a management port, serial console port, and a USB port.
The switch uses QSFP-DD optics and DACs. We previously looked at the FS 400Gbase-SR8 400GbE QSFP-DD optics.
These look small when plugged in, but they are actually larger than their QSFP28 predecessors.
On the rear of the switch, we have power and cooling, both of which are substantial efforts.
The fans are dual-fan modules.
They are also hot-swappable as we would expect in a $55,000 switch.
The Great Wall power supplies are 1.3kW 80Plus Platinum units. They are typical power supplies one might find in a modern high-end server.
Next, let us get inside the system and see how this switch is built.
That heatsink is crazy. Makes me wonder what is going to happen (en mass) first: co-packaged optics, or water-cooled switches?
Question: How much does the Tomahawk 4 weigh?
Just musing..but is it odd a 64 port 400GbE switch (and other enterprise or service provider level equipment) is presented on a site called “Serve the HOME” Don’t get me wrong, I love the content here!
Yeah, that is an interesting switch. The thing is so big that it needs a standard server chassis just for that MASSIVE heatsink and 8,000-plus pin Broadcom Tomahawk 4 switch ASIC. Very cool. 🙂
Page 1, paragraph 4: “The top 4U is used for cooling.” should read “The top 2U is used for cooling.”
Page 3, paragraph 3: “We will quickly note that while we were using Intel Xeon “Sapphire Rapids”” – missing end of sentence.
Sure hope this has a better warranty than “30 days then we send it back to China for a few months” I experienced with an obviously much less expensive switch.
as mentioned before, use the 2x200G cx-7 nic and a cheap ($150) QSFP-DD to 2 x QSFP56 DAC.
Don’t waste your time with the flat top OSFP weirdness!
@Kwisatz: It’s definitely a misleading site name and it causes so much confusion it’s an FAQ in the “about” section: https://www.servethehome.com/about/
“STH may say “home” in the title which confuses some. Home is actually a reference to a users [sic] /home/ directory in Linux. We scale from those looking to have in home or office servers all the way up to some of the largest hosting organizations in the world.”
I do wonder how many of us ended up here because we were after information about home server equipment. I guess judging by the number of non-enterprise mini-PC and 2.5 GbE articles, quite a few of us.
I wonder how often 400Gb ports are saturated in this context – aren’t latency and rate more interesting?
It would be useful if you would cover the management features of the switch. Sure, the heatsink is cool and all but that’s not really functional or useful to users, is it?