The Broadwell-DE Intel Xeon D-1500 series and SR-IOV

One hot topic in the STH forums has been that of the Intel Xeon D-1500 series’ support for SR-IOV. We have some updates to share on support in the Intel Xeon D-1500 series given the new chips’ release earlier this week.

What is SR-IOV?

SR-IOV is away to have a Single Root Function (e.g. an Ethernet Port) appear to be multiple separate physical devices. This is useful for virtualization because one can reduce virtualization overhead using a single port rather than adding ports and assigning them to each VM. A useful case here is adding 10GbE to a server than having the NIC shared with multiple VMs.

Here is a good overview paper from Intel on SR-IOV if you are looking for a deeper explanation. If you just want the diagram of what is going on, here is a good one from that paper:

Is SR-IOV absolutely necessary for virtualization? No. It is a PCI-SIG standard technology to accelerate virtual machine networking.

Intel Xeon D-1540 and Xeon D-1520 Support for SR-IOV

For those that may have missed this in the forums, we did get confirmation from Intel that SR-IOV was indeed not supported by the Intel Xeon D-1540 and D-1520 launch SKUs. Here is the slide we got from Intel for the Broadwell-DE launch. See the SR-IOV support under Optimized for Virtualization.” It is also in the block diagram on the right hand of the slide.

We sent a note to Intel after hearing rumors that Broadwell-DE SR-IOV support was not working. We sent the following to Intel for confirmation:

I heard from a few sources that VT-d or SR-IOV pass-through of the Xeon D’s 10Gb NICs is not possible at this time. I just wanted to confirm this is indeed the case.

and received the following response:

Hi Patrick,

To follow up below, SR-IOV is not enabled in the Intel® Xeon® Processor D-1520 and D1540 products that Intel announced earlier this year.(sic)

With the release of 8 new Intel Xeon D-1500 parts yesterday, we wanted to see if the status has changed.

New Intel Xeon D-1500 series support for SR-IOV

On 9 November 2015 Intel released eight new Intel Xeon D-1500 series SKUs. The primary focus was on on the storage and networking segments. Since SR-IOV was featured prominently in the release materials for Broadwell-DE for the March 2015 launch we wanted to see if the status had changed with the new parts. We asked for clarification and received this earlier today:

Hi Patrick,

SR-IOV is enabled on all SKUs we announced yesterday, and VT-d is also a feature of the CPUs.

We did ask for a clarification regarding whether there is a new D-1520 and D-1540 stepping that will enable SR-IOV on those platforms and we received the following:

At this point, there are no plans to enable SR-IOV in D-1540 and D-1520.

That is at least interesting considering the fact it was a feature touted in launch materials. It is possible that the SR-IOV in launch materials meant Xeon D processors that would eventually be released.

Conclusion

If you read through the 30+ page Intel Xeon D discussion thread on the STH forums or the SR-IOV with Xeon D testing thread you will see folks running into challenges with SR-IOV. At the end of the day, Facebook is one of the larger Xeon D customers an in its Yosemite Xeon D server it actually uses a Mellanox ConnectX-4 card for networking (see the lower left of this picture and here for the description.)

Performance on the onboard 10Gb ports for a lot of use is still good for many use cases. For example, in our Proxmox VE (KVM + LXC) and Ceph cluster we have in our Fremont, CA datacenter can push well over 1GB/s storage traffic over the NICs and to the VMs. In the initial cluster we used three Supermicro X10SDV-TLN4F based systems which have been working extremely well to the point additional nodes will be added this week.

We are happy to see that Intel has enabled SR-IOV on the new chips but wanted to get this information out so folks are at least aware of the earlier chip limitations.