During our initial iDRACula vulnerability coverage, we had a Q&A interview with one of the discoverers Jon Sands (see the interview here.) For those who missed the original article, it is worth spending a few minutes reading through iDRACula Vulnerability Impacts Millions of Legacy Dell EMC Servers. The basic premise is that iDRAC 8 and older versions have both hands-on hardware and remote access potential for tampering with the BMC firmware images allowing persistent remote and controlled access to servers undetectable by iDRAC users or the OS itself.
After the duo that found the vulnerability stepped forward, Adam Nielsen wanted to offer a viewpoint on iDRACula and the broader implications it has for the industry. Here is his perspective that we wanted to add to the discussion. It offers a view of the challenge that the industry is currently facing weighing the need for remote access to manage and automate large numbers of servers with the security concerns putting a small embedded computer (BMC) in the devices to act as a remote administrator.
Here is what Adam sent us:
Adam Nielsen’s Perspective on iDRACula and its Implications
As one of the pair who recently discovered what is now being called the iDRACula vulnerability, I would like to take this opportunity to discuss a few of the issues this has raised.
As many readers will be aware, the iDRAC is a small Linux computer that has full control over a PowerEdge server. The iDRAC can do everything from powering the server on and off to making software-controlled USB devices appear on virtual USB ports connected to the main processor, such as a virtual USB keyboard for the remote console or a virtual CD drive to boot a disk image uploaded through the iDRAC’s web interface.
In other words, not only can the iDRAC turn your servers off, it can boot them from a remote disk image, plug in a virtual auto-run USB stick while the OS is running, wipe the BIOS/firmware flash chips so the server won’t boot, and even turn all the fans off in the hope that something overheats and is permanently damaged.
The iDRAC then, is critical to the security and survival of the whole system.
During the journey of discovering how to gain root access to the iDRAC, we learned just how much effort Dell has put in to restrict access to this important device. There are checksums, verifications, and backups at every step of the way, and seemingly all designed to keep everyone out of the iDRAC – attackers and owners alike.
While it makes sense that nobody should have access to something this crucial, the problem is that there will always be vulnerabilities. It is inevitable that given enough time, someone will always find a way in. The problem arises when the attacker finds a way in, but the owner is still locked out.
This is the situation we have now. An attacker with one-time access to the server, either physical or remote with a valid login, can completely take over the iDRAC, permanently. So then how do you know, as the owner of a PowerEdge server, that your system has not been compromised? What if yours already has? There’s no antivirus for an iDRAC, and you can’t get a shell on the Linux system it runs, so how could you tell? Don’t forget that the iDRAC can send and receive packets through the server’s onboard NICs, so even isolating it to a management VLAN won’t necessarily limit an infected device’s communications with the outside world.
The only way you would have any hope of confirming the security of your servers is if you, as the owner of the system, had full unrestricted root access to the iDRAC. You could then go in and look for yourself.
You could see what processes are running, you could check any startup scripts, and yes, eventually you might even be able to run a malware scan on the iDRAC itself.
But sadly none of this is possible, because Dell and other manufacturers go to such lengths to restrict access from everyone, that it ultimately only hurts their own customers. Those who purchase servers from these companies are not given full access to their own devices, so nobody can check for themselves what exactly these embedded computers are really doing.
The latest generation of iDRACs was not affected by this vulnerability because access to them is locked down even more. They still have the same control over the server, and there are no doubt vulnerabilities waiting for attackers to find, but it’s now even harder for everyone to confirm that they can really trust their own servers. Is your server’s embedded controller phoning home? Is it sending data off to another country, or even to your own government? How would you know if you can’t go in and check for yourself?
If the two of us can compromise the iDRAC then you can bet that we aren’t the first, and the real question you should be asking both yourself and your server manufacturer, is how can I get root access to my own server to go and look for myself?
Yeah but nobody is gonna actually monitor this themselves. You need vendors like Dell to provide software frameworks to monitor if someone’s gotten into a BMC. If you have 1-10 servers, MAYBE you can do it. Once you’re at 1000s it’s too hard to monitor, even if you have access, without vendor support.
What I read here is that BMCs need the same AAA security features that are present on all other operating systems. What point is having Authentication and Authorization without Accounting? Yeah… I agree.
OpenBMC anyone?
I feel like this series leaves out citing a big load of prior research on BMC security to make this look like a STH exclusive. There have been way worse BMC security findings recently.
Today’s BMCs are at about the security level as the free-with-contract router your isp gives you. You have no real control or visibility of their guts, and when its pwned you either have to junk it or break out your own flashing hardware. The only difference is that crummy router can be replaced with another crummy one for $50 or less, but the Dell/HP/Whatever BMC is glued inside hundreds of $5k++ servers running your critical business functions.
Maybe worse, because some consumer routers can be reflashed with open source firmware that can be configured properly and locked down, have actual logs etc.
worst additional article i have ever read since it essential brings nothing new to the table. Unless they are willing to actually do a full disclosure there is no point of providing them additional space since they only add to the damage that dell has already provided.