Today we are going to discuss a change to server security that is going to make waves in the home lab and secondary markets for servers and components in the future. During our recent Dell EMC PowerEdge C6525 review we briefly mentioned that AMD EPYC CPUs in the system are vendor locked to Dell EMC systems. This is not a Dell-specific concern. We have confirmed that other vendors are supporting the feature behind this. For the large vendors, their platform security teams are pushing to build more secure platforms for their customers, and that is going to have future impacts on the secondary server market and home labs.
In this article, we are going to cover the basics of what is happening. We are going to discuss the motivations, and why this is going to be more common in the future. Finally, we are going to discuss what those in the industry can do to keep the secondary server market operating well. If you work with partners or resellers who dip into used parts bins or even have the potential to purchase grey market CPUs, send them this article or accompanying video. The current market has a large disconnect between what some large customers are asking for, and large vendors are delivering on and what others in the market know is happening.
This is an important topic. To ensure that we can cover those who like to read/ skim and those who like to get information via audio, we have an accompanying video:
Feel free to pop open that video on YouTube and check it out or to send to those who prefer not to read.
Background: How we learned this was a “thing”
In 2018 we did a Dell EMC PowerEdge R7415 review and as part of that review, we started our normal process of trying different CPUs in the system. Early in that process, we used an AMD EPYC 7251 CPU, the low-end 8-core model, and noticed something curious. It would not work in our other test systems after.
After a bit of research, we found it was because Dell EMC was vendor locking the chips to Dell systems. We did not know exactly why, but we were told was a security feature. At this point, and even to this day two years later, not every vendor takes advantage of all of the AMD EPYC security features. What that practically means is that what we saw with the Dell EMC system is not what we saw with other systems. For example, we were able to interchangeably use CPUs in Supermicro and Tyan systems, but we could not use those systems once they went into a Dell EMC server.
We found we were not alone. Laboratories, VARs, and other organizations were finding that transferring AMD EPYC CPUs from one vendor’s system to another, was not the simple process it was on the Intel Xeon side. It did not always work.
We knew it was a security feature and thought that most who are buying servers would be informed of this by their sales reps or channel partners. After I personally got a lot of texts, e-mails, instant messaging, and comments on our C6525 video and article, I realized that this actually may be a situation where many people do not know what is going on.
This experience that we had, apparently is one that is not overly common yet. That makes sense because the systems that utilize the enhanced security levels are still largely new, and being used by their first buyers. Also, AMD still has a smaller market share than Intel. A big reason, by the way, that Intel Xeon does not have this issue is that they do not have the security feature that AMD has. Vendors have come out and stated that their AMD EPYC systems are more secure than their Intel Xeon systems, and this behavior is a byproduct of that enhanced security.
Next, we are going to dive into the feature of AMD processors (and what will be more common in future CPUs from other vendors.)
AMD EPYC Secure Processor Platform Secure Boot (PSB)
Let us start with the high-level slide. This is effectively the same slide on the AMD Secure Processor that we saw with the AMD EPYC 7001 series launch, but this is from the EPYC 7002 series. AMD EPYC CPUs may be x86, but they have an embedded Arm Cortex-A5 microcontroller that runs its own OS that is independent of the main system. This AMD Secure Processor is the backbone of AMD’s security push as it provides features such as key management and hardware root of trust for the platform.
AMD spends time patching this solution to make it more secure, but it is generally fairly hard to reach without some extremely low-level access in a system. We are going to come back to the “Enables hardware validated boot” line shortly, but it is important to understand that this secure processor underpins many of AMD’s best security features.
For example, at STH we use EPYC’s Secure Memory Encryption and Secure Encrypted Virtualization heavily. With AMD EPYC, we do not have to manually manage keys. Instead, the ephemeral keys are managed for us by the AMD Secure Processor. This is the basis for what is really the building wave of confidential computing offerings such as Google Cloud Confidential Computing Enabled by AMD EPYC SEV. Intel has its secure boot features and SGX that will be enhanced greatly with Ice Lake Xeons, but for now, AMD has this capability while Intel does not. When big vendors say AMD is more secure, the AMD Secure Processor is a cornerstone of those offerings.
Let us discuss that “Enables hardware validated boot” line. While traditionally CPUs just fire up in whatever platform they are in, AMD has intelligence in their CPU due to the Arm-based AMD Secure Processor. EPYC CPUs are designed to be a bit more intelligent about the platforms they are in, and interact with server platform security to act as this root of trust that would not be possible if they effectively just booted up in any system.
Here is a statement from AMD describing the AMD Platform Secure Boot.
The AMD Platform Secure Boot Feature (PSB) is a mitigation for firmware Advanced Persistent Threats. It is a defense-in-depth feature. PSB extends AMD’s silicon root of trust to protect the OEM’s BIOS. This allows the OEM to establish an unbroken chain of trust from AMD’s silicon root of trust to the OEM’s BIOS using PSB, and then from the OEM’s BIOS to the OS Bootloader using UEFI secure boot. This provides a very powerful defense against remote attackers seeking to embed malware into a platform’s firmware.
An OEM who trusts only their own cryptographically signed BIOS code to run on their platforms will use a PSB enabled motherboard and set one-time-programmable fuses in the processor to bind the processor to the OEM’s firmware code signing key. AMD processors are shipped unlocked from the factory, and can initially be used with any OEM’s motherboard. But once they are used with a motherboard with PSB enabled, the security fuses will be set, and from that point on, that processor can only be used with motherboards that use the same code signing key. (Source: AMD statement to STH)
That is a lot to take in. We asked HPE about this. Their response mirrored what the above was describing. HPE firmware, when a system is first turned on, performs this binding process where the AMD EPYC CPU expects to see HPE signed firmware. If you alter the HPE firmware on the system, the check fails and the system will not work. That means if your HPE motherboard fails, you can replace it and put your CPU in another HPE motherboard with signed HPE firmware. It also means if the server platform’s firmware is not signed by HPE, the processor will see it as evidence of tampering and not work.
Edit: 2020-09-09 – HPE clarified that they are doing this in a different manner than Dell after initially confirming that they were using the AMD PSB feature. After this went live, HPE sent us the following:
HPE does not use the same security technique that Dell is using for a BIOS hardware root of trust. HPE does not burn, fuse, or permanently store our public key into AMD processors which ship with our products. HPE uses a unique approach to authenticate our BIOS and BMC firmware: HPE fuses our hardware – or silicon – root of trust into our own BMC silicon to ensure only authenticated firmware is executed. Thus, while we implement a hardware root of trust for our BIOS and BMC firmware, the processors that ship with our servers are not locked to our platforms. (Source: HPE)
What is at least interesting there is that HPE was initially claiming feature parity with Dell to us, and from the comments on this article were saying they used this feature in sales pitches, but now are saying they are not blowing the eFuses.
Here is where the concern develops, and not necessarily for AMD, the OEM, or most of the initial customer base. Customers want more security. The OEMs want to create a secure hardware environment because that is what their customers want. AMD is implementing an advanced security solution beyond what Intel Xeons have giving the OEMs and end-customers what they want. Effectively, when these are sold as new systems, this is exactly what everyone involved wants.
If everyone is getting what they want, then where is the concern, that is what we are going to cover next.