Today we are going to more thoroughly address the Bloomberg Businessweek article alleging that China targeted 30 companies by inserting chips in the manufacturing process of Supermicro servers. Despite denials from named companies and the technology press casting some reasonable doubt on the story, Bloomberg doubled down and posted a follow-up article claiming a different hack took place. In this piece, we are going to present a critical view of Bloomberg’s claims, as supported by anonymous sources, in order to allow our readers to decide for themselves the credibility of Bloomberg’s reporting in this case.
Technical Lightness or Inaccuracy
This is a long article. In the first section, we are going to discuss why there are some fairly astounding plausibility and feasibility gaps in Bloomberg’s description of how the hacks worked. The weakness in this section of the Bloomberg article makes it extremely difficult to navigate and it is light on details. We are going to evaluate some of the parts in isolation, and also discuss some of the logical outcomes. In our first investigative piece, Bloomberg Reports China Infiltrated the Supermicro Supply Chain We Investigate, we went into some detail about why a motherboard and hardware for a motherboard is a very difficult way to hack a BMC. If you have not read our Explaining the Baseboard Management Controller or BMC in Servers that should be a precursor to reading the next section. STH has a relatively technically minded audience, so we are going to assume our audience has at least the knowledge imparted in that article.
The Lynchpin of How Bloomberg’s Device Activates is Not Plausible
We are going to focus on a few key parts of one of the opening paragraphs from the story where functionality is described.
Since the implants were small, the amount of code they contained was small as well. But they were capable of doing two very important things: telling the device to communicate with one of several anonymous computers elsewhere on the internet that were loaded with more complex code; and preparing the device’s operating system to accept this new code. The illicit chips could do all this because they were connected to the baseboard management controller, a kind of superchip that administrators use to remotely log in to problematic servers, giving them access to the most sensitive code even on machines that have crashed or are turned off. (Source: Bloomberg with emphasis added to highlight key points for discussion)
That first part starting with “telling the device…” is nonsensical. If you are in the industry or read our Basic BMC and IPMI Management Security Practices piece, you would know that this is false. If you have an unsophisticated network or a lack of understanding about the topic, you may think that this is how BMC’s are networked:
Even smaller organizations with a handful of servers generally have segregated BMC networks. That basic starting point, from where large companies take further steps, looks something like this:
The key here is that the companies named are all sophisticated, and will have better protections than your average small to medium enterprise. Bloomberg’s report describes an attack that is not possible at the companies listed in the article.
Even the notion of being able to switch channels (a basic part of the IPMI interface) is well-known and network administrators will prevent BMC leakage even if a BMC is compromised so it cannot get egress by switching network ports.
One of the biggest reasons people in the server management industry doubt the Bloomberg report is that it is not plausible for these chips to function in the networks of their intended targets, allegedly “30 companies.” Amazon AWS noted as much in their rebuttal response to this article when they said: “The remaining two non-critical issues with the web application were determined to be fully mitigated by the auditors if customers used the appliances as intended, without exposing them to the public internet.” (Source: Bloomberg)
We are going to get to how hard it is to pull this type of hack off later in this article. For now, the important piece is that the attack, as described in the passage above, would not work at its intended targets. Standard industry practice guards against this attack vector.
Impossible Access to Code on Crashed or Turned Off Servers
The next inaccuracy to this paragraph is the line describing BMCs as “giving them access to the most sensitive code even on machines that have crashed or are turned off.” That is not how this technology works.
Baseboard management controllers or BMCs are active on crashed or turned off servers. They allow one to, for example, power cycle servers remotely. If you read our piece Explaining the Baseboard Management Controller or BMC in Servers BMCs are superchips. They replace a physical administrator working on a server in a data center for most tasks other than physical service (e.g. changing failed hard drives.)
At the same time, the sensitive data on a system is in the main server complex, not the BMC. When the BMC is powered on, hard drives, solid state drives, the server’s CPU (for decrypting data) and memory are not turned on. If you read our embedded systems reviews, such as the Supermicro A2SDi-16C-HLN4F 16-core Intel Atom C3955 mITX Motherboard Review, we actually publish power figures for when a system is on versus when the BMC only is active. In that review, the BMC powered on utilizes 4.9W of power. SSDs each have idle power consumption generally above 1W and hard drives use considerably more even at idle. The point here is that when the server’s BMC is turned on, and the server is powered off, it is trivially easy to measure that the attached storage is not powered on and accessible.
When a server is powered off it is not possible to access a server’s “most sensitive code.” OS boot devices are powered off. Local storage is powered off for the main server. Further encrypted sensitive code pushed from network storage is not accessible, and a BMC would not authenticate.
This line from the Bloomberg is technically inaccurate because a powered off server’s storage with its sensitive code has no power and cannot be accessed.
We have discussed two patently false technical details in the Bloomberg article. Anyone involved in the server industry will know this as they are common foundational elements regarding how servers work. Beyond the false points in the Bloomberg article, there are a number of other elements that are at best implausible.