At STH, we have done a number of switch teardowns. Today, we have something special. We are getting to look at a switch that has 32x QSFP-DD ports each capable of running 400GbE bi-directional traffic. In this article, we are going to take a look at the Innovium Teralynx 7 based switch, and then see it pass 12.8Tbps of traffic in both directions. This is the same type of switch being deployed by hyper-scalers today.
Video Version
As we have been doing in this series, we have an accompanying video:
As always, we urge our readers to open this video on YouTube for a better viewing experience.
Inside an Innovium Teralynx 7-based 32x 400GbE Switch
First a quick bit of background. Innovium is the upstart in the data center switch market directly targeting a segment long dominated by Broadcom Trident and Tomahawk product lines. In terms of success, with the 400GbE generation, Innovium has now shipped over 1M ports and has made sizable inroads in a market where Broadcom had a virtual monopoly.
The switch we are looking at today is based on the Innovium Teralynx 7. This is a 12.8T switch product. Innovium also has the lower-end Teralynx 5 which is more for Top of Rack and other applications where as Teralynx 8 is the next-gen for 800GbE.
In terms of key features, we have Flashlight which is the company’s telemetry and analytics solution. Part of Innovium’s key differentiation is being able to handle this level of analytics alongside its high-performance/ low latency Terascale switch fabric and its Innovlex programmable pipeline.
Recently, Innovium raised $170M in a funding round valuing the company at over $1B due to its portfolio and market success thus far.
With that, it is time to get to the hardware.
Innovium Teralynx 7-based 32x 400GbE Switch Hardware Overview
As is customary, we are splitting our hardware overview into external and then internal sections. Simply due to the nature of video versus still images, you can see more angles in the accompanying video. One will notice that this is a switch that says “Innovium” but it is made by one of the dozens of OEM/ODM partners that Innovium has. We were borrowing the switch, and this is what we could get. We expect most hyper-scalers to not have this paint and that the Taiwanese ODM will not brand their product the Innovium TERASwitch 7.
Innovium Teralynx 7-based 32x 400GbE Switch External Hardware Overview
The 1U switch has its front mostly filled with 32 ports. These are QSFP-DD ports. QSFP-DD is required for 400GbE and is likely to be the standard our readers will see most often.
On one side of the switch, we have the status LEDs and a big reset button. On a relative scale, this is gargantuan compared to many we have seen.
On the right side, we get a management RJ45 port, a USB port, and a serial console port.
Moving to the rear of the switch, we can see that this 1U chassis rear is almost all handles, latches fans, and power connectors.
The fans are, as one would expect, hot swappable. They also have status LEDs to help diagnose a failed fan module.
The power supply we will discuss more around in the power consumption section, but this is a 1.3kW 80Plus Platinum redundant power supply configuration.
After that look at the outside of the switch, next we will take a look inside before getting to power and performance.
Looks like it is 1M SERDES shipped and not 1M 400G ports.
I’d say this is one of the best high-end switch reviews I’ve ever seen. I’d like to see more on SONiC installs and ops. Good work though.
How STH kills time before the AMD Milan launch ——-> Oh here’s some ultra amazing switch.
Second the request for articles on SONiC installs and ops. These switch teardowns are fantastic.
OMG, 32x 400Gbps. And in the meantime small business/office users are waiting for fanless 10Gbps switch with more than just 2-4 10GigEs.
That market share number for Innovium feels a little too high. Any idea why?
If you are looking closely, it is not total switch market share.
This is really cool. I can help with the interconnections. We should talk if your are interested in QSFP-QQ 400G connectors, cables and optics.
That switch consumes more power than my entire rack at home lol. Great review though!