FS N8560-32C 32x 100GbE Switch Internal Hardware Overview
Opening this switch is difficult. There are ~26 screws to un-do in order to get inside the switch, making it one of the least accessible units we have reviewed to date. The next switch we are planning to review is the Facebook Wedge100, also a 32-port 100GbE unit that is designed for the hyper-scaler. That is a simple latch to open and is a stark contrast.
The key feature of the switch is the switch chip. This is a Broadcom Trident 3-X7 (BCM56870) switch chip which is standard in this generation. Some other units like the older Dell S5148F-ON switches utilized older Cavium/ Marvell switch chips instead of Broadcom and were retired relatively quickly. Broadcom is the leader in the merchant switch silicon switch space. It is good that FS is using that solution here and not a lower-volume Marvell switch chip or a lower-quality bargain brand like Nephos. We were happy to see the Trident 3-X7 used here.
Here is the Trident 3 key specs slide. This is 32x 100GbE so it is the X7 that takes the 128x25G SerDes and makes 32x 100GbE ports.
The main switchboard has a variety of connections to the management board. This is just a fun picture where we can see two SFF-8643 style connections for higher-speed communications and then a ribbon cable for lower-speed connectivity.
Between these cable landings and the PSU we have a Xilinx Artix-7 FPGA.
Aside from managing the fans and power supplies, this management board has a number of different modules.
First, there is a M.2 SSD and two slots available. Based on the model number, this looks like an Intel (now Solidigm) Intel D3 S4510 240GB SATA SSD with onboard PLP. Even though the system is hard to get into, we see a nice tool-less M.2 retention mechanism here.
The BMC sits on its own PCB with memory and storage as well as a Broadcom 1GbE PHY. We often see BMCs on their own daughterboards in switches. We do not usually see BMCs with heatsinks on them as they are low power parts.
The management processor on these is on its own module with two SODIMM slots. One of the more interesting pieces is that FS is using an Intel Xeon D-1527 here, not a lower power and performance Intel Atom that we see on some switches. In older eras, the Atom’s have had things like the AVR54 bugs while the Xeon D line has been relatively more reliable.
Between the Xeon D daughterboard and the hot-swap fan landings is a variety of chips from Broadcom and Lattice. A lot of times Broadcom sells platforms to switch vendors with heavy bundling incentives so it is common to see a lot of Broadcom if the main switch chip is Broadcom.
Overall, this was better internal construction and component choices than we were expecting.
Next, let us get to the software side.
FS N8560-32C 32x 100GbE Switch Software Overview
Getting into the switch requires a standard RJ45 console cable or connecting a management interface. The default IP is 192.168.1.1 and the default username/ password is admin/ admin. One is told that the password is too weak, but is not immediately prompted to change the password. We usually like when these are set to DHCP address acquisition since that makes configuration, if the service port is already configured, easy. There is a pretty wide variation on how this is handled in the industry.
The Intel Xeon D-1527 has a ton of performance for the management of the switch. Idle utilization is in the 1% range.
The CLI is fairly standard, but it is something that will take a bit of getting used to coming from other vendors.
From a performance perspective, we got up to running half the switch ports at 100GbE speeds without issue before running out of optics to test with. We were using a relatively simple iperf3 configuration, so this was basically just validating that the L2 speeds were correct. We have now tested many Trident-3 switches, so the performance is more of a known quantity.
FS has a fairly huge feature list that does not just include things like VXLAN, but also includes features like PFC that are important in modern networking, especially for 100GbE speeds.
Given the breadth of features in a platform like this, our suggestion is to check out the documentation for the features and the CLI. Some are very open to CLIs, and some want only specific CLI experiences so this is going to be a preference item.
Next, let us get to the power consumption, value proposition, and our final words.
I’m with you. Why didn’t they make the handles read for port to power airflow?
I’m more worried about the lack of NRTL marks (UL, Intertek, etc) on the power supply! I’d imagine that might be a good reason why they don’t have 80+ rating.
I’d imagine that most colos require those marks for any equipment you bring in, too- I know the handful I’ve worked with do.
The FS switches I’ve bought (S5860-20SQ) have been rock solid for nearly 2 years at this point. Optics as well. Just my 2 cents. Great value for the money.
This website does make me laugh. It’s called Serve the Home but here we are reviewing a 32 x 100Gbe switch! How many home users have this kind of gear?!
FYI- on page 2 in the CLI section it cuts off between the screen shots:
“The CLI is fairly standard, but it is something that”
John R: some of us do, at least. Used 100G switches were actually pretty cheap for a brief while, before the supply chain mess hit.
@John R: …How many home users have this kind of gear?!…
I concur, but it’s not 0%.
I’m using a 36-ports FDR (56Gbps) switch equivalent to ~35 to 45GbE(*). Extremely useful for SMB, VM, database, large files over the Intranet. Due to to the supply chain SNAFU, used 100GbE is too expensive at the moment.
(*) used IB hardware is anyway and still less expensive than brand new “prosumer” 10 GbE.
STH forum is an excellent source of support information when you use this type of hardware at home 🙂
Would love to see them support ONIE for SONiC etc.
Also what is three-layer sub-interface to VxLAN is that layer-3 sub-interface to VxLAN support? or do they support some form of triple vLAN tagging.