Bloomberg today came out with an industry shocker. The Big Hack: How China Used a Tiny Chip to Infiltrate U.S. Companies. In that article, Bloomberg reports that the PLA managed to infiltrate Supermicro’s supply chain and add small chips that allowed Chinese agencies to hack into 30 companies such as Apple and Amazon. The company also published, in a different article, statements from Amazon, Apple, and Supermicro strongly rebutting the story. See The Big Hack: Statements From Amazon, Apple, Supermicro, and the Chinese Government. Something is certainly strange here, and at STH, we review more server platforms than anywhere else on the Internet, including those from Supermicro. We also, by chance, started diving into the BMC security space more recently so it is clearly time to investigate.
What is this hack?
According to Bloomberg, the hack involves a small IC inserted into the Supermicro motherboard PCB. In previous generations, this would have been a surface mount component. The story claims current generations have these devices embedded in PCB.
There, of course, has to be much more than a simple chip. That chip needs to tap into electrical signals both for power and for data transfer. That means that not only must a component be inserted, but also PCB wires. Bloomberg says it is in line with memory to CPUs to intercept some password validation code. By changing this code in Linux, it allows remote attackers to access the servers and potentially phone home.
That is a little strange frankly from a technical standpoint. Where could these chips be located?
DRAM memory traces are very complex. 288 pins per DDR4 DRAM module (not all are data of course), times 8 modules per CPU times two CPUs and that is a lot of pins to monitor from such a small IC. Even in the older 240-pin DDR3 generation, with 16 modules there is no way a small IC can monitor that many wires. Also, memory traces in motherboards are often an area where PCB designers spend a lot of time to get correct lengths and timings on the wires. Inserting a small IC would not be the easiest feat there.
The other candidates are more probable. The first is using the onboard SATADOM wires. SATADOMs are small flash memory devices used to load base operating systems. SATA cables are 7-pin designs with three ground wires and two A/B +/- pairs. Supermicro SATADOM connectors have an extra power capability.
This would be a lower pin count option to exploit. The problem, of course, is that most large shops encrypt data on the SATADOMs. Most SATADOMs do not have self-encrypting capabilities which means it is host encryption. The Bloomberg article said that the hardware would intercept storage to CPU transfers. If the data is encrypted when transferred, it would be nearly impossible in that IC footprint to crack reasonable encryption and change the OS in-line.
The final, and perhaps most likely vector would be the BMC. We have a piece Explaining the Baseboard Management Controller or BMC in Servers. A hardware chip that could impact the BMC firmware is more probable.
Each BMC has local storage ever since the 1998 IPMI 1.0 spec was announced.
This is generally a very small flash module for storage, often a few MB in size. The BMC usually runs a flavor of Linux. Getting root access to the BMC is bad, but it is not the same as getting full access to the main server OS.
The BMC has root console access to the server. It is on before the server boots. It can mount media and has network access. Think of it as an administrator sitting at the machine, but bringing that functionality anywhere in the world.
BMCs are amazingly hacked devices. The Bloomberg story’s comments from Amazon and Apple both point to the BMC and IPMI firmware/ management interfaces. We think this is the most likely vector.
The bad news is that BMC’s are extremely dangerous. They are also pervasive with a few points under 100% of servers having them these days. The Bloomberg article cites the well-known Supermicro BMC/ IPMI vulnerabilities. Supermicro is not alone. Every Dell EMC PowerEdge server (edit: 13th generation and older, the new 14th generation has a fix to prevent this) has a local and remote exploit available that the company can mitigate with patches, but cannot fix. We broke this story with iDRACula. If you think you are safe with HPE or Lenovo servers, here are BMC vulnerabilities for other vendors.
The security community, as a whole, knows that BMCs are both useful if not mandatory in today’s infrastructure. As a result, the security community, and major hyper-scale vendors are putting a lot of effort in researching security solutions.
One of the more interesting bits is that if it is a BMC vulnerability or anything that “phones home” over a network interface, one would expect that security researchers would have seen it. There are companies that put boxes on networks just to see what network traffic they create. Supermicro tends to build common designs that it ships to multiple customers. It would be slightly interesting if only some Supermicro servers, e.g. for certain customers were impacted. If China did not do this, it would have been caught earlier. If China did limit to a few customers, it would be difficult to target them at PCB. As we will show shortly, Supermicro PCBs are used across products.
Bottom line, if this Supermicro attack vector is to the BMC, then the Bloomberg story is no bigger than the Dell EMC PowerEdge iDRACula story or any others. Saying there is a vulnerability in a BMC is like saying the sun is hot.
Some higher-resolution areas of MicroBlade BMCs
We had some similar generation Supermicro MicroBlades where we could provide higher-resolution photos of their BMC areas. This is where the hacked chips are located on the board that Bloomberg depicts. This also shows that a Supermicro PCB is spun for multiple products. That makes it extremely difficult to target a specific customer at the time of PCB construction. Here we have two different products built on the same underlying PCB.
For our less technical readers, this is what the actual PCB looks like. For our more technical readers, you may want to see for yourself.
Here are two MicroBlades of that era the Supermicro B1SD1-TF and the B1SD2-TF. The “2” represents that the PCB houses two complete server nodes. We highlight this because if the attack is present on this platform, presumably it would require a second inserted chip which would not be required on the B1SD1-TF.
There are a ton of ICs there. I know we have STH readers who will want to look. Have at it.
The Counterpoint Published Outside of the Main Story
Bloomberg posts statements from companies, not in their main article, but linked in a separate article.
Amazon, Apple, and Supermicro all deny that this is happening.
Just for a taste, here is an excerpt from Apple’s statement:
“We are deeply disappointed that in their dealings with us, Bloomberg’s reporters have not been open to the possibility that they or their sources might be wrong or misinformed. Our best guess is that they are confusing their story with a previously-reported 2016 incident in which we discovered an infected driver on a single Super Micro server in one of our labs. That one-time event was determined to be accidental and not a targeted attack against Apple.” (Source: Bloomberg/ Apple)
This is a little strange. All three are public companies. A simple “no comment” would have sufficed. Or a “we would not be allowed to comment on your classified source story” perhaps. Supermicro one can dismiss their lack of knowledge to perhaps the intelligence community not wanting to alert anyone there. Apple and Amazon went beyond a simple “no recollection” or “no comment” type response. They should not be allowed to make these types of responses if they are untrue since they would be potentially misleading investors. Even if they could not speak about the issues, they did not have to go into the depth that they did.
Indeed, when we broke iDRACula the persistent (with mitigations) non-fixable Dell EMC PowerEdge issue impaction tens of millions of their servers, we held the entire story while Dell EMC’s confirmation went through legal and management approvals. Having just broken a similar story, the responses from parties are in an absolutely sharp contrast.
Where the Bloomberg Piece Makes No Sense
There is one area where the Bloomberg piece makes no sense. Supermicro servers are procured for US Military contracts and use to this day. Supermicro’s government business is nowhere near a large as some other vendors, but there are solutions providers who sell Supermicro platforms into highly sensitive government programs.
If the FBI, or other intelligence officials, had reason to believe Supermicro hardware was compromised, then we would expect it would have taken less than a few years for this procurement to stop.
Assuming the Bloomberg story is accurate, that means that the US intelligence community, during a period spanning two administrations, saw a foreign threat and allowed that threat to infiltrate the US military. If the story is untrue, or incorrect on its technical merits, then it would make sense that Supermicro gear is being used by the US military.
First and foremost, I think we need to call for an immediate SEC investigation around anyone who has recently taken short positions or sold shares in Supermicro. With the accompanying Supermicro stock price hit that was foreseeable prior to the story, if anyone knew the story would be published, and acted on that non-public or classified information, the SEC needs to take action. There seems to have been over 20 people that knew about this.
Further, with public companies making statements on the impact, unless there is a valid national security/ classified reason that they gave the responses they did, there is a mismatch. Apple and Amazon did not say “no comment” they called Bloomberg’s account false. The SEC needs to investigate here as well to see if these were publicly misleading statements.
Second, we need a formal investigation into why, if this is a true and serious threat, why it was not flagged in military procurement years ago.
There are parts of the Bloomberg story, the rebuttals from Amazon, Apple, and Supermicro, and logical reasoning which point to one key takeaway: server security is a big deal. Perhaps the bigger takeaway is that this is a 21st-century battleground that is active every day. Government agencies from China, the US, Russia, Israel, and others all have ways to impact servers and more broadly computing devices. We know the Intel management engine has been compromised. There are reports of Lenovo laptops phoning home data. It would be naive to think that any major world power is not working to get information from compute devices whether they are from Supermicro or another vendor. It is probably better to assume your server is compromised and start with that.