At Flash Memory Summit 2015 we saw a number of interesting designs where companies were trying to cram as much PCIe/ NVMe flash in a single system as possible. Practically, there are only so many PCIe lanes that Intel provides on each CPU, up to 40 PCIe 3.0 lanes on each Xeon E5 CPU. With x4 devices, that allows for an easy expansion path to 10 drives per processor. If one wants to go higher, a different architecture is needed. Avago recently purchased not just LSI but also PLX. With the LSI acquisition Avago has the ability to expand SAS implementations. With the PLX acquisition, Avago has the ability to expand PCIe lanes.
The key component is the Avago/ PLX PCIe switch line. At a very high-level, these chips operate similar to a network switch where one can have one or more uplink ports and hang a number of devices off of the switch. Here is a nice shot of what this practically looks like from a company called Serial Cables. One can see four SFF-8644 external PCIe cables in the rear each carrying 4x lanes for a total of 16 lanes uplinked to a system. Convieniently, the same number of lanes in a standard PCIe x16 slot. The front of the board has 12x SFF-8643 connectors. So we have 48 lanes to use to connect drives. The Avago switch chip is therefore likely a 64 PCIe port switch.
The benefits of this are large. One can use these chips to increase capacity (on this board) by a factor of 3x. Taking this to a larger level, there were a few systems on display from One Stop Systems. William recently reviewed a One Stop Systems PCIe expansion appliance focused on desktops. The company had a system build to OpenCompute specs that could handle m.2 SSDs:
One can see no less than seven Avago/ PLX chips there. The company also showed off expansion chassis that could handle 2.5″ top loading PCIe/ NVMe SSDs using special adapters:
Here is the same system full of add-in-card type SSDs:
A number of vendors did show off traditional looking 2U and 4U appliances that had 24+ PCIe drives installed. Although we could not take pictures of the chips due to the display cases, simple math deduced and we confirmed with company reps that, these systems were indeed using Avago PCIe switch chips.
Of course, SAS has been using expanders and large topologies for years. One of the more interesting SAS arrays was the SanDisk IF100 which we got to see in person for the first time:
The IF100 does not use Avago PCIe chips but is certainly a possibility where these earlier stage PCIe designs are heading.
Expect to see more PCIe switch chips from Avago in larger storage systems in the near future. The PLX technology has been around for years and using Avago switches does provide some unique capabilities e.g. dual ported PCIe/ NVMe drives are available that when combined with Avago switches and the right chassis can provide Active-Active controller NVMe drive systems.