Edgecore AS7712-32X Switch Overview A 32x 100GbE Switch

9
Edgecore AS7712 32X Broadcom BCM56960 Tomahawk 3.2Tbps Chip Heatsink
Edgecore AS7712 32X Broadcom BCM56960 Tomahawk 3.2Tbps Chip Heatsink

In this article, we are going to take a look inside a bare metal switch, the Edgecore AS7712-32X. This is a bare-metal switch with ONIE and broad network OS support so you can use it with most major NOS distributions. It features 32x 100GbE ports that can break out to 25GbE ports in a rack making the switch extremely flexible. In our overview, we are going to look at the switch and see how it was built.

Edgecore may not be an overly well-known brand if you have always used Cisco hardware. The company is a subsidiary of Accton which is one of the big networking ODMs. When you read reports about hyper-scale companies like AWS and Facebook leaving their Cisco gear and instead of going to merchant silicon, that is the market that Accton has excelled in. An example of this collaboration between hyper-scale and Edgecore is the Facebook Minipack which is a switch Facebook is using for 100GbE and 400GbE networking. While you may have never heard of Edgecore or Accton, their hardware is running many of the hyper-scale services you use every day.

Edgecore AS7712-32X Video

We took some video of the switch if you are on the go and prefer listening to learn about the switch. Check that out here:

That video gives a few more angles than we can show in photos just due to being video. Of course, we have more detail in our piece here so keep reading to dive in.

Edgecore AS7712-32X Switch External Overview

Looking at the front of the switch, we can see the 32x QSFP28 100GbE ports. Where we used QSFP+ for 40GbE, we now use QSFP28 for 100GbE switching. Those 32 ports dominate the front of the switch.

Edgecore AS7712 32X Front View
Edgecore AS7712 32X Front View

We wanted to show three important features if you use this switch. There is a console, out-of-band management, and a USB port. All three are important. The serial console port is a standard feature in the industry. One can also see the management port and USB port. An essential part of the bare metal switch and ONIE is loading your own operating system. As a result, these features are often used to accomplish that first step in using the switch. After the NOS is installed, that management port is also the out-of-band management port for the switch.

Edgecore AS7712 32X Console Management USB
Edgecore AS7712 32X Console Management USB

Moving to the rear of the switch we can see a fairly simple, but designed to be an easily serviceable switch. There are six numbered hot-swap fans that run in a 5+1 redundant manner so even in the unlikely event that a fan fails, the switch can operate while that fan is being replaced.

Edgecore AS7712 32X Rear
Edgecore AS7712 32X Rear

One will also see two power supplies. These are also hot-swappable units and have internal fans to keep them cool in the switch as well.

Edgecore AS7712 32X Hot Swap PSUs
Edgecore AS7712 32X Hot-Swap PSUs

For most users, this is all they will see of the Edgecore AS7712-32X. We, however, wanted to share a look inside.

Edgecore AS7712-32X Switch Internal Overview

Looking at the internal overview of the switch we can see a layout where we see the ports and the main switch chip PCB, we have the management portion, then we have the cooling control in the rear of the switch.

Edgecore AS7712 32X Internal Overview
Edgecore AS7712 32X Internal Overview

One item we wanted to note here is just how many fairly large ICs there are in a switch like this. We are going to talk about them next, but we just wanted to point out that there are chips from Intel x86 and Altera CPLDs/ FPGAs which is now part of Intel. There is the Broadcom switch chip. One can even see the SmartFusion Coretex-M3/ FPGA SoC onboard. There is a ton going on here.

Edgecore AS7712 32X Internal Overview Big Chips Labeled
Edgecore AS7712 32X Internal Overview Big Chips Labeled

The main switch chip here is the Broadcom Tomahawk BCM56960 which is a 3.2Tbps switch chip that can do 6.4Tbps full-duplex. That fits nicely with 32x 100GbE ports. The Broadcom Tomahawk was perhaps the first widely-adopted 100GbE switch chip. Although there are new generations and Broadcom is pushing 12.8Tbps versions of newer Tomahawk chips for 400GbE switches, this is an extremely popular switch chip.

Edgecore AS7712 32X Broadcom BCM56960 Tomahawk 3.2Tbps Chip Heatsink
Edgecore AS7712 32X Broadcom BCM56960 Tomahawk 3.2Tbps Chip Heatsink

We wanted to quickly point out that this PCB has more than just the switch chip and ports. One will also see three low-power Altera (now Intel) Max V CPLDs, There is one more of these chips on the fan PCB for a total of four that we saw.

Edgecore AS7712 32X Intel Altera Max V
Edgecore AS7712 32X Intel Altera Max V

That is not the only Intel-Altera product. The switch also has a Cyclone IV FPGA next to the primary switch chip.

Edgecore AS7712 32X Intel Altera Cyclone IV
Edgecore AS7712 32X Intel Altera Cyclone IV

Perhaps the most interesting feature is the Intel Atom C2538 chip onboard serving as management for the switch. This is a low power quad-core SoC that is part of Intel’s Rangely line. It runs at 2.4GHz and has a 15W TDP. You will notice the small heatsink on the SoC. Edgecore can get away with this since the switch is moving so much air.

Edgecore AS7712 32X Intel Atom C2538 CPU
Edgecore AS7712 32X Intel Atom C2538 CPU

An item we wanted to note is that since these are Intel Atom C2000 series products, one needs to be aware of the Atom C2000 Series Bug. In 2016 about three years after they launched, Cisco identified an issue where the Atom C2000 series had a chance to cease working after years in operation. There is a board-level workaround and in 2017 an Intel Atom C2000 C0 stepping came out fixing the AVR54 bug. The problem is that devices such as switches and firewalls are not the easiest items to replace and the chips are everywhere. At STH, we even fell victim to the Intel Atom C2000 AVR54 Bug. Datacenter power is reliable, however, we suggest that you ensure that your Edgecore AS7712-32X has a newer chip or the workaround. If not, plan accordingly.

Since the Atom C2000 series was launched in 2013, it utilized DDR3 memory, not the newer DDR4 that current chips all use. We can see two 8GB SODIMMs for a total of 16GB of memory onboard.

Edgecore AS7712 32X 16GB RAM DDR3 SODIMM
Edgecore AS7712 32X 16GB RAM DDR3 SODIMM

Since a management controller in this type of switch looks a lot like an embedded x86 server, we also have a 64GB mSATA SSD.

Edgecore AS7712 32X 64GB MLC MSATA SSD
Edgecore AS7712 32X 64GB MLC MSATA SSD

Since this is a switch, we also have an Actel SmartFusion A2F200M3F chip with FPGA logic and an Arm Cortex-M3 processor.

Edgecore AS7712 32X Actel SmartFusion ARM A2F200M3F
Edgecore AS7712 32X Actel SmartFusion ARM A2F200M3F

Going back to our original diagram, one can see just how many good size logic ICs there are in this switch and how it has been designed for flexibility. One way to think about these bare metal switches is as low-power servers that control high-speed networking subsystems.

Edgecore AS7712 32X Internal Overview Big Chips Labeled
Edgecore AS7712 32X Internal Overview Big Chips Labeled

You can see that the switch has airflow guides to keep PSU airflow and main switch airflow separate. Airflow is specifically directed around the Broadcom Tomahawk switch chip and the QSFP28 ports to ensure the higher-power components get enough cooling.

One can see internally the hot-swap fans that have their own PCB to monitor fan operation and ensure that failures are diagnosed and replacement is relatively easy. If you were wondering, this switch can run well over 60dba like many servers and other data center equipment.

Edgecore AS7712 32X Fan PCB And Counter Rotating Fans
Edgecore AS7712 32X Fan PCB And Counter Rotating Fans

One nice feature is that the Edgecore AS7712-32X has LED light channels so one can easily see which fan has failed.

Edgecore AS7712 32X Fan Status LED Light Channels And Layerd PCBs]
Edgecore AS7712 32X Fan Status LED Light Channels And Layered PCBs

At this point, you have probably noticed that there is a fan PCB, a switching PCB, and a management controller PCB with the Atom SoC. Edgecore sells the AS7712-32X with a different CPU subsystem. Specifically, the AS7716-32X has an Intel Xeon D-1518 chip onboard. Building switches in this modular fashion allows Edgecore to gain efficiencies when creating switch lines with different port configurations and different CPU options.

Edgecore AS7712 32X 550W 80Plus Platinum PSU Internal
Edgecore AS7712 32X 80Plus Platinum PSU Internal

The power supplies are 80Plus Platinum rated units. The cooling for these is not redundant, but there are two power supplies each of which can run the entire switch leading to a 1+1 redundant setup.

Edgecore AS7712-32X Switch NOS Options

The Edgecore AS7712-32X comes pre-installed with Open Network Install Environment (ONIE) so you can load your network OS of choice onto the switch. From Edgecore, here are the options one has:

  • Open Network Linux (ONL), an open-source, OCP reference NOS.
  • Netvisor ONE Network operating system from Pluribus Networks.
  • Big Monitoring Fabric 5.6.0 and later version from Big Switch Networks.
  • Cumulus Linux 3.1 and later versions.
  • SONiC open-source network software.
  • PicOS 2.7.1 and later version from Pica8 Inc.
  • OcNOS 1.2 and later version from IP Infusion.
  • Ixia Vision Edge OS 4.7.3 and later.
  • SnapRoute CN-NOS, Cloud Native – NOS software.
  • Infinera CNOS.
  • ArcOS 3.3.1 and later versions.

In our next lab build-out, this switch will be used with SONiC, however, the idea has been tossed around using ONL.

While the Edgecore AS7716 has better hardware with the onboard Xeon D, it only is compatible with a subset of the above. At some point, we want to see the open networking world becomes more CPU agnostic. These days if you want to use many of these OSes, you need not just an x86 CPU, but also a specific model or line. This is happening over time, but there are still some hardware constraints. The Edgecore AS7712 is popular because it works with many OSes that have been tested and validated on the switch.

OCP Accepted Logo
OCP Accepted Logo

We wanted to take a quick second to point out that this is an OCP Accepted design. That means that the switch complies 100% with an OCP approved specification and the design files are open-sourced and available. This has become a popular switch for that reason.

Final Words

Hopefully, you enjoyed this tour of the Edgecore AS7712-32X switch. We thought this is an important piece since the AS7712-32X is extremely popular and runs many of the popular networking distributions out there. For some of our readers looking at next-generation open networking, we wanted to show how a switch like this is designed at a high-level and some of the reasons why. For others, this is just a look at a cool piece of hardware that you will know more about it should you ever encounter it in a data center.

9 COMMENTS

  1. Patrick, I think the video is in either “Unllisted” or “private” mode – it does not show on your channel in YouTube.

  2. Hetz, changing audio levels so re-uploading the video. Likely on the channel in <10 min. Sorry about that. If you wanted to share it, give it a few minutes until the updated version is swapped in.

  3. One quick note. Edge-Core’s hardware is relatively solid, but their software and firmware are anything but. I’ve got ~100 of this switch (with a PowerPC CPU), and about 10 ECS4510-52T.

    Variety of problems:
    1. SNMP daemon crashes when snmpwalk is run.
    2. Leaving a DAC cable connected to the switch, but not at the other end can result in the link flapping up and down over and over and then the switch rebooting, or even hardlocking.
    3. The software is generally kinda crap for those of us used to Arista/Cisco/etc, and the CLI is not the same between different switch product lines.
    4. Their ONIE version tends to crash a fair bit for some reason.
    5. Some versions of their switch software have progressive link failures. Sometimes the link lights stay on even if you pull a cable, other times the lights work, but no traffic flows. Have to reboot the switch.

    Just my 2 cents. I’d stay very far away.

  4. What switches and NOS are you using? I’ve found that there is a big difference in terms of finished product quality in something like Cumulus and OcNos vs say Sonic.

  5. Big problem with these switches is heat when populated fully with both fiber and dac modules. The cage airflow is insufficient and causes the switch to overheat and die. In some cases the cages got so hot that the paint flaked off the front in our DC.

LEAVE A REPLY

Please enter your comment!
Please enter your name here

This site uses Akismet to reduce spam. Learn how your comment data is processed.