FS N8560-32C 32x 100GbE Switch Review

8

FS N8560-32C 32x 100GbE Switch Internal Hardware Overview

Opening this switch is difficult. There are ~26 screws to un-do in order to get inside the switch, making it one of the least accessible units we have reviewed to date. The next switch we are planning to review is the Facebook Wedge100, also a 32-port 100GbE unit that is designed for the hyper-scaler. That is a simple latch to open and is a stark contrast.

FS N8560 32C Front Open
FS N8560 32C Front Open

The key feature of the switch is the switch chip. This is a Broadcom Trident 3-X7 (BCM56870) switch chip which is standard in this generation. Some other units like the older Dell S5148F-ON switches utilized older Cavium/ Marvell switch chips instead of Broadcom and were retired relatively quickly. Broadcom is the leader in the merchant switch silicon switch space. It is good that FS is using that solution here and not a lower-volume Marvell switch chip or a lower-quality bargain brand like Nephos. We were happy to see the Trident 3-X7 used here.

FS N8560 32C Broadcom Trident 3 X7 BCM56870
FS N8560 32C Broadcom Trident 3 X7 BCM56870

Here is the Trident 3 key specs slide. This is 32x 100GbE so it is the X7 that takes the 128x25G SerDes and makes 32x 100GbE ports.

Broadcom Trident 3 X5 X7
Broadcom Trident 3 X5 X7

The main switchboard has a variety of connections to the management board. This is just a fun picture where we can see two SFF-8643 style connections for higher-speed communications and then a ribbon cable for lower-speed connectivity.

FS N8560 32C Control Board To Switch Board Link
FS N8560 32C Control Board To Switch Board Link

Between these cable landings and the PSU we have a Xilinx Artix-7 FPGA.

FS N8560 32C Xilinx Artix 7 FPGA
FS N8560 32C Xilinx Artix 7 FPGA

Aside from managing the fans and power supplies, this management board has a number of different modules.

FS N8560 32C Xeon D M.2 And Management
FS N8560 32C Xeon D M.2 And Management

First, there is a M.2 SSD and two slots available. Based on the model number, this looks like an Intel (now Solidigm) Intel D3 S4510 240GB SATA SSD with onboard PLP. Even though the system is hard to get into, we see a nice tool-less M.2 retention mechanism here.

FS N8560 32C M.2 SSD And Open Slot
FS N8560 32C M.2 SSD And Open Slot

The BMC sits on its own PCB with memory and storage as well as a Broadcom 1GbE PHY. We often see BMCs on their own daughterboards in switches. We do not usually see BMCs with heatsinks on them as they are low power parts.

FS N8560 32C Management
FS N8560 32C BMC

The management processor on these is on its own module with two SODIMM slots. One of the more interesting pieces is that FS is using an Intel Xeon D-1527 here, not a lower power and performance Intel Atom that we see on some switches. In older eras, the Atom’s have had things like the AVR54 bugs while the Xeon D line has been relatively more reliable.

FS N8560 32C Intel Xeon D 1527 Control Plane
FS N8560 32C Intel Xeon D 1527 Control Plane

Between the Xeon D daughterboard and the hot-swap fan landings is a variety of chips from Broadcom and Lattice. A lot of times Broadcom sells platforms to switch vendors with heavy bundling incentives so it is common to see a lot of Broadcom if the main switch chip is Broadcom.

FS N8560 32C Rear Area View From Rear
FS N8560 32C Rear Area View From Rear

Overall, this was better internal construction and component choices than we were expecting.

Next,  let us get to the software side.

FS N8560-32C 32x 100GbE Switch Software Overview

Getting into the switch requires a standard RJ45 console cable or connecting a management interface. The default IP is 192.168.1.1 and the default username/ password is admin/ admin. One is told that the password is too weak, but is not immediately prompted to change the password. We usually like when these are set to DHCP address acquisition since that makes configuration, if the service port is already configured, easy. There is a pretty wide variation on how this is handled in the industry.

FS N8560 32C Admin Admin Logon
FS N8560 32C Admin Admin Logon

The Intel Xeon D-1527 has a ton of performance for the management of the switch. Idle utilization is in the 1% range.

FS N8560 32C Show Cpu
FS N8560 32C Show Cpu

The CLI is fairly standard, but it is something that will take a bit of getting used to coming from other vendors.

FS N8560 32C Show Interfaces
FS N8560 32C Show Interfaces

From a performance perspective, we got up to running half the switch ports at 100GbE speeds without issue before running out of optics to test with. We were using a relatively simple iperf3 configuration, so this was basically just validating that the L2 speeds were correct. We have now tested many Trident-3 switches, so the performance is more of a known quantity.

FS has a fairly huge feature list that does not just include things like VXLAN, but also includes features like PFC that are important in modern networking, especially for 100GbE speeds.

FS N8560 32C Features
FS N8560 32C Features

Given the breadth of features in a platform like this, our suggestion is to check out the documentation for the features and the CLI. Some are very open to CLIs, and some want only specific CLI experiences so this is going to be a preference item.

Next, let us get to the power consumption, value proposition, and our final words.

8 COMMENTS

  1. I’m more worried about the lack of NRTL marks (UL, Intertek, etc) on the power supply! I’d imagine that might be a good reason why they don’t have 80+ rating.

    I’d imagine that most colos require those marks for any equipment you bring in, too- I know the handful I’ve worked with do.

  2. The FS switches I’ve bought (S5860-20SQ) have been rock solid for nearly 2 years at this point. Optics as well. Just my 2 cents. Great value for the money.

  3. This website does make me laugh. It’s called Serve the Home but here we are reviewing a 32 x 100Gbe switch! How many home users have this kind of gear?!

  4. FYI- on page 2 in the CLI section it cuts off between the screen shots:

    “The CLI is fairly standard, but it is something that”

  5. @John R: …How many home users have this kind of gear?!…

    I concur, but it’s not 0%.

    I’m using a 36-ports FDR (56Gbps) switch equivalent to ~35 to 45GbE(*). Extremely useful for SMB, VM, database, large files over the Intranet. Due to to the supply chain SNAFU, used 100GbE is too expensive at the moment.

    (*) used IB hardware is anyway and still less expensive than brand new “prosumer” 10 GbE.

    STH forum is an excellent source of support information when you use this type of hardware at home :-)

  6. Would love to see them support ONIE for SONiC etc.

    Also what is three-layer sub-interface to VxLAN is that layer-3 sub-interface to VxLAN support? or do they support some form of triple vLAN tagging.

LEAVE A REPLY

Please enter your comment!
Please enter your name here

This site uses Akismet to reduce spam. Learn how your comment data is processed.