FS N8560-32C 32x 100GbE Switch Internal Hardware Overview
Opening this switch is difficult. There are ~26 screws to un-do in order to get inside the switch, making it one of the least accessible units we have reviewed to date. The next switch we are planning to review is the Facebook Wedge100, also a 32-port 100GbE unit that is designed for the hyper-scaler. That is a simple latch to open and is a stark contrast.
The key feature of the switch is the switch chip. This is a Broadcom Trident 3-X7 (BCM56870) switch chip which is standard in this generation. Some other units like the older Dell S5148F-ON switches utilized older Cavium/ Marvell switch chips instead of Broadcom and were retired relatively quickly. Broadcom is the leader in the merchant switch silicon switch space. It is good that FS is using that solution here and not a lower-volume Marvell switch chip or a lower-quality bargain brand like Nephos. We were happy to see the Trident 3-X7 used here.
Here is the Trident 3 key specs slide. This is 32x 100GbE so it is the X7 that takes the 128x25G SerDes and makes 32x 100GbE ports.
The main switchboard has a variety of connections to the management board. This is just a fun picture where we can see two SFF-8643 style connections for higher-speed communications and then a ribbon cable for lower-speed connectivity.
Between these cable landings and the PSU we have a Xilinx Artix-7 FPGA.
Aside from managing the fans and power supplies, this management board has a number of different modules.
First, there is a M.2 SSD and two slots available. Based on the model number, this looks like an Intel (now Solidigm) Intel D3 S4510 240GB SATA SSD with onboard PLP. Even though the system is hard to get into, we see a nice tool-less M.2 retention mechanism here.
The BMC sits on its own PCB with memory and storage as well as a Broadcom 1GbE PHY. We often see BMCs on their own daughterboards in switches. We do not usually see BMCs with heatsinks on them as they are low power parts.
The management processor on these is on its own module with two SODIMM slots. One of the more interesting pieces is that FS is using an Intel Xeon D-1527 here, not a lower power and performance Intel Atom that we see on some switches. In older eras, the Atom’s have had things like the AVR54 bugs while the Xeon D line has been relatively more reliable.
Between the Xeon D daughterboard and the hot-swap fan landings is a variety of chips from Broadcom and Lattice. A lot of times Broadcom sells platforms to switch vendors with heavy bundling incentives so it is common to see a lot of Broadcom if the main switch chip is Broadcom.
Overall, this was better internal construction and component choices than we were expecting.
Next, let us get to the software side.
FS N8560-32C 32x 100GbE Switch Software Overview
Getting into the switch requires a standard RJ45 console cable or connecting a management interface. The default IP is 192.168.1.1 and the default username/ password is admin/ admin. One is told that the password is too weak, but is not immediately prompted to change the password. We usually like when these are set to DHCP address acquisition since that makes configuration, if the service port is already configured, easy. There is a pretty wide variation on how this is handled in the industry.
The Intel Xeon D-1527 has a ton of performance for the management of the switch. Idle utilization is in the 1% range.
The CLI is fairly standard, but it is something that will take a bit of getting used to coming from other vendors.
From a performance perspective, we got up to running half the switch ports at 100GbE speeds without issue before running out of optics to test with. We were using a relatively simple iperf3 configuration, so this was basically just validating that the L2 speeds were correct. We have now tested many Trident-3 switches, so the performance is more of a known quantity.
FS has a fairly huge feature list that does not just include things like VXLAN, but also includes features like PFC that are important in modern networking, especially for 100GbE speeds.
Given the breadth of features in a platform like this, our suggestion is to check out the documentation for the features and the CLI. Some are very open to CLIs, and some want only specific CLI experiences so this is going to be a preference item.
Next, let us get to the power consumption, value proposition, and our final words.