An Interview with the
Vampire Discoverer of iDRACula
The STH forum member, and his colleague on taking advantage of iDRACula, did not want to be publicly named. There is a lot of concern in the media about “anonymous sources.” What I can share is that this STH forum member did confer with the Dell EMC iDRAC firmware team this week to confirm the vulnerability and method they used. STH is now the largest enterprise hardware review site and we, therefore, have many readers who have professional lives that intersect with hobby interests, such as working on servers.
In this instance, since Dell EMC was able to validate this work, and confirm to me that it was indeed a vulnerability, we are going to keep the STH forum member’s identity out of this article to respect their wishes for anonymity. They are not a security researcher doing this for a bounty, this was an accidental discovery which was handed over to Dell EMC. With that said, I asked for a Q&A interview about the vulnerability with the explicit understanding that STH is not going to be sharing a how-to guide on reproducing it.
[Edit: 7:20PM Pacific, 28 September 2018] The two folks who discovered this vulnerability Jon Sands and Adam Nielsen notified us that they wanted to be credited by name. This interview is with Jon Sands. Adam Nielsen is the colleague mentioned.
STH: Tell us a bit about yourself and your colleague.
My colleague and I currently work in the IT sector but have no association with the security realm. This all stemmed from wanting more granular control over the fan control logic in some 12th generation Dell PowerEdge servers, which is handled by “platformdata” files in the iDRAC filesystem.
I began searching on Google to see if anyone else has achieved the root access required to do this, but all I could find was a post buried in an old Dell mailing list from my colleague, hinting that he achieved said root access on DRAC a couple of generations ago. The method had been long since patched, but I emailed him anyway to see if he’d be interested in helping out. Not only was he interested, but he had just picked up the exact same Dell PowerEdge 12G server I had. He was also interested in more granular thermal profile control, so we set out to find a way in, so to speak.
STH: Tell us about what iDRAC is and why it is an area that vendors spend so much time with security?
iDRAC is Dell EMC’s brand of lights-out server management. It allows you to have a unified interface for managing your server, inventory, chassis hardware, remote console & OS installs, etc. It’s a totally separate computer (running Linux, in this case) with its own storage, RAM, and everything. I think a lot of people don’t realize this, and how privileged its access to your server hardware is. That’s why IPMI/BMC controllers have become such a security focus, when not secured they become a giant backdoor to your hardware.
STH: What can you share about the attack vector? Keeping in mind that we will not print details that allow others to reproduce.
There are two attack vectors: one local with physical access, and one remote (via the web UI). The local method does not require any valid login, with physical access logins can be bypassed quite easily as you’d expect. With the remote method, you must already have a valid DRAC login. Using either method, you can elevate privileges to a root Linux shell, which then allows you to load in whatever replacement filesystem and code you want. This bypasses Dell’s usual checks. The bad part about this is it can all be done so it’s totally hidden from other iDRAC users. You can load in the compromised firmware that “phones home” and they will have no clue.
Since iDRAC is an isolated embedded computer, wiping the BIOS, OS drives, or anything else will have no effect. Depending on the iDRAC flash partition you load your modifications in, even using the iDRAC web UI to flash fresh firmware will not get rid of your modifications. When I spoke to Dell they said they will be pushing out a patch for this in a new release.
Please keep in mind you need either physical access to the PowerEdge server. To do this remotely, you must have existing iDRAC credentials.
STH: What can a malicious user do with a hacked iDRAC?
There’s the obvious stuff like remotely power your server on and off, but also more malicious things like capture your server’s console contents, send keyboard input, mount your own storage into the OS running on the server, that kind of thing. Anything you can do from the usual iDRAC UI, you can do undetected with no record of your existence from other’s point of view, unless they were monitoring network traffic.
STH: What server generations are vulnerable, given current firmware, to iDRACula?
It seems like this impacts all Dell EMC PowerEdge 12th and 13th generation products. Given BMC security has become more sophisticated as new BMC hardware is more capable, one might expect older generations to be vulnerable as well. After speaking with a lead firmware engineer at Dell, these exploits are absolutely impossible on current Dell EMC PowerEdge 14th generation servers and future generations. They went to extreme lengths to prevent these kinds of situations, even using a hardware root of trust from the bottom up. Due to technical limitations of the BMC CPU, these preventative measures were not possible in 12th and 13th generation products, or anything before that.
STH: Is what you found applicable to other vendor’s remote management solutions?
It’s been a long time since I looked at Supermicro and HPE equipment, but the last products I did look at, the physical method was absolutely possible. I have not done any investigation into the remote methods, nor on more recent products, so I can’t say. I will say from my previous experience, Dell has taken this kind of vectors more seriously than anyone else, so it certainly would not surprise me.
STH: Are there any broader implications for those purchasing servers?
The interesting vector here that I think a lot of people (especially in the second-hand/ homelab community) ignore is the past history of the equipment they buy. In about 2 hours I can compile a custom image that reports anything I want back to me, flash it in a way that is totally undetectable to the next user without physical tools to read raw EMMC flash, then list it on ebay. The purchaser will have a server that is reporting back to me and giving me extremely privileged access to their server & console, and they’ll never have a clue. They can flash the BIOS, replace hard drives, and it’ll still be there. They can even use the iDRAC management web page to flash new iDRAC firmware, and it’ll still be there (until they flash a new firmware that addresses this exploit specifically.)
This of course also assumes the iDRAC can hit a network gateway that gets out to the internet to report back, another vector a lot of people ignore. The solution here is obviously to stop putting your IPMI equipment anywhere near the Internet (even if you have inbound connections blocked, that’s still not enough). To ensure you’re getting untampered hardware in the first place, you need to make sure you’re purchasing directly from Dell EMC or an approved vendor (or Supermicro, or HP, or whatever your preferred flavor.)
8) Is there anything else that you want to share about the vulnerability and experience?
It was much more difficult than I think either of us had anticipated, especially compared to other IPMI & embedded devices I’ve used. Even with complete physical access, they had some novel protections in place. It’s clear the embedded team at Dell EMC spent a good amount of time thinking about all of this and did everything they could given the hardware capability. As previously mentioned, Dell EMC PowerEdge 14th generation and up have made all of this impossible. Dell’s response has also been very quick and communicative. It was an enjoyable puzzle to put together, working with my colleague via email on two opposite sides of the globe turned it into an enjoyable hunt. Props to him, and props to Dell for reaching out and handling this gracefully!
[Edit: 9:00AM Pacific, 30 September 2018] We published a follow-up piece from Adam Nielsen with his perspective on iDRACula. You can read that here: Broader Implications of iDRACula Vulnerability a Perspective
Dell EMC’s Comment on the iDRACula Finding
Here is the official comment we received from a Dell EMC spokesperson when we notified Dell EMC that we would be running a piece on this.
Hi, Patrick. I’ve been able to meet with our team to get a better understanding of this post. Of course, we take this and any vulnerability reported to us very seriously, as we take pride in developing systems that are as secure as possible. Through communication with the poster, we learned that he identified two areas that he felt were potential vulnerabilities. Applied remotely, with admin rights for iDRAC, he had downloaded an older firmware version with a known vulnerability and created root user access. This known vulnerability in older iDRAC firmware has already been addressed in subsequent firmware releases. The second was through physical access, as you know. We believe exposure is limited, but we are taking extra precautions to reduce any further risk.
We are going to reserve the right to post an update to this section if the company comes back with an additional response after having read this piece.
Next, we are going to discuss why this is a broader industry concern, and give our final thoughts on the iDRACula class of vulnerabilities.