Today, Bloomberg upped the stakes in its contentious story around Chinese tampering of Supermicro hardware, citing Yossi Appleboum CEO of Sepio Systems. Here is that story. I reached out to Mr. Appleboum for comment via telephone. Whereas the Bloomberg story singles out Supermicro servers, Mr. Appleboum’s sentiment is that this is an industrywide issue. Other very large server and networking manufacturers are certainly impacted, perhaps more so. He also stated that as an industry, or a society, we have two options: we go with the narrative that a US company, Supermicro, is the only one impacted as the Bloomberg reporting suggests, or we recognize that this is a persistent threat that impacts the entire hardware supply chain that underpins the lynchpin communications infrastructure of our global economy.
Background Perspective for Our Readers
Before we proceed, I wanted to present some personal bias. At STH, we review servers from every major server vendor. As a result, I speak with a lot of people in the industry. I have visited manufacturing operations for different companies and we also work with some major component suppliers. STH focuses on deeper technical content, but this needs to be an industry discussion. For our more technically minded readers, here is what an RJ-45 network port, at the heart of today’s story looks like:
For the record, I personally believe that there are hardware and software attack vectors against baseboard management controllers, as well as other components in modern systems. I believe that there have been attacks on BMCs and network hardware against every major server vendor in the US, China, and elsewhere. Positioning this industrywide issue as a Supermicro issue is irresponsible at best. Case in point, we unveiled iDRACula Vulnerability Impacts Millions of Legacy Dell EMC Servers which allows BMC firmware tampering on Dell EMC PowerEdge 13th generation and prior if a malicious actor has either physical access or remote access with valid credentials. This is not a single vendor issue. This is a global issue that participants in the industry must tackle.
My Discussion with Yossi Appleboum
In the 4th paragraph of the Bloomberg piece cited above, I noticed that he seems to refer to this as not just a Supermicro problem. I asked Mr. Appleboum about this:
We found it in different vendors, not just Supermicro. We found it not just in servers, in different variations, but hardware manipulation on different interfaces, mostly in network related. We found it in different devices connected to the network, even Ethernet switches. I am talking about really big what are considered to be major American brands, many compromised through the same method.
This is why I think that Supermicro has nothing to do with that. In many cases, by the way, it is not through manufacturing, it is after through the supply chain.
People think of the supply chain in a very narrow sense between the manufacturer and the customer. Supply chain never ends. There are technicians, there are integrators, there are people that work in your facilities. We have seen after installation, after the fact attacks where someone switched something already installed. This is why Supermicro would have no idea what happens later in the supply chain.
I asked Yossi if the supply chain can include those integrating racks or those moving boxes. He said:
Totally possible. We have been a witness, or have been involved in an investigation, not in the US by the way, but who cares. Eventually, the one who switched the box was a guy who got money to switch boxes during the shipment.
We have a problem. The problem is the hardware supply chain. All of us are dealing with what happened to Supermicro, and whether Amazon knew or did not know. That is not the main issue for me. The main issue is that we have a problem. It is global. This is why I think Supermicro is suffering from the big players. I am talking about the really big players who know that they have the same problem, and they are kind of using the story right now to throw Supermicro under the bus instead of coming out and saying that it is a global problem, let’s fix it and find a solution.
I asked Mr. Appleboum for his thoughts around the silicon root of trust or hardware root of trust that hyper-scale companies like Google (with Titan) or other vendors like HPE and Dell EMC are implementing in their newest servers. I asked if that is the answer to the supply chain problem.
It’s a good start but again since you do not really what is going on in your supply chain, you don’t know where it is going to happen in your supply chain. Having a root that will vouch for the hardware is a good start. The problem is that when you get the hardware how can you make sure the product was not compromised? Someone can replace modules that validate hardware with other modules that say it is okay. We have seen things like that in our past.
There are two things that we need to do. One as a society come back to the ones that manufacture for us and say that they cannot let this happen or it will be the end of business for you. That is politics. The second one is technology-wise, this is why we started Sepio. I am not going to do a marketing pitch around that. Companies should have a policy that says my hardware is an immediate suspect, just like my software. We are spending $100B on software related attacks, but near zero for hardware attacks. That is irresponsible and that is the problem that we need to fix.
I asked Mr. Appleboum if he had a sense of how much happens in the manufacturing process versus after.
I don’t have the numbers or statistics around how much it happens in the manufacturing process or how much it happens after. My educated guess is that most of it happens later because it is much easier and much harder to find the source. If you do this through the manufacturing, eventually there is a name and address to go back and ask questions. If it happens after, you go where? Who is going to answer your questions?
Summarizing Supermicro’s fault in this, he said:
I think they are innocent and someone is using them to dilute the story instead of mitigating the threat. Please help me, them, and everyone else to understand that the problem is bigger. Dealing with this as a Supermicro problem will ruin the opportunity to face the reality that we need to fix it.
Mr. Appleboum said that those that have been in Israeli intelligence, (where Mr. Appleboum has worked for) or the CIA, being in the news is a nightmare. From the discussion, I got the sense that Mr. Appleboum was taking my call because he wanted to clarify how the story was being told instead of for personal or professional gains. In our discussion, he came across as someone who shares the belief that this is an industry-wide problem, not a Supermicro problem.
The line that stuck with me during our discussion happened when I confirmed he agreed to be quoted in this. Mr. Appleboum replied:
I want to be quoted. I am angry and I am nervous and I hate what happened to the story. Everyone misses the main issue.
This is extremely strange. I believe that these types of attacks are more common than we often assume. Bloomberg’s reporting has done an excellent job bringing this to top of mind. At the same time, it begs the question of whether we hang this solely around Supermicro, and angle Bloomberg seems to be advocating by not providing balanced reporting. The other option is that we recognize this is an industry problem. Further, it is going to be an industry problem for the foreseeable future that impacts all companies that provide the infrastructure we enjoy every day.
I strongly believe there needs to be an SEC investigation here. This is a dangerous precedent to set where equities are being impacted by selective reporting and strong denials from companies who have obligations to give truthful statements to shareholders. As a global market leader, the US SEC needs to ensure that companies are making truthful statements and that markets are not being manipulated for financial gain. This very well may not be the case here, but this is the case where a lot of parts are not aligning warranting some assurances from regulators that manipulation is not happening.
My takeaway from the conversation was fairly simple. First, I immediately re-read today’s Bloomberg piece and thought, this is very different than the impression I had after reading that piece. Second, for STH readers who are mainly in the IT industry, it is worth having a conversation with your teams about this. Everyone in the industry knows that there are multiple facets to security, and the hardware supply chain is one of them. Bloomberg did the industry a service by highlighting the need for greater hardware security and scrutiny.
We should take this as an opportunity to take simple steps such as ensuring that BMC IPMI interfaces are not publicly routable, egress from BMC networks is restricted, and default passwords are not left on production machines. If a BMC were compromised, as discussed in the original report, competent IT setups from large organizations would not give packets egress from BMC networks. There are thousands of publicly routable BMCs on the Internet right now across vendors. This should be the point where that number goes to zero.