An Interview with the Vampire Discoverer of iDRACula
The STH forum member, and his colleague on taking advantage of iDRACula, did not want to be publicly named. There is a lot of concern in the media about “anonymous sources.” What I can share is that this STH forum member did confer with the Dell EMC iDRAC firmware team this week to confirm the vulnerability and method they used. STH is now the largest enterprise hardware review site and we, therefore, have many readers who have professional lives that intersect with hobby interests, such as working on servers.
In this instance, since Dell EMC was able to validate this work, and confirm to me that it was indeed a vulnerability, we are going to keep the STH forum member’s identity out of this article to respect their wishes for anonymity. They are not a security researcher doing this for a bounty, this was an accidental discovery which was handed over to Dell EMC. With that said, I asked for a Q&A interview about the vulnerability with the explicit understanding that STH is not going to be sharing a how-to guide on reproducing it.
[Edit: 7:20PM Pacific, 28 September 2018] The two folks who discovered this vulnerability Jon Sands and Adam Nielsen notified us that they wanted to be credited by name. This interview is with Jon Sands. Adam Nielsen is the colleague mentioned.
STH: Tell us a bit about yourself and your colleague.
My colleague and I currently work in the IT sector but have no association with the security realm. This all stemmed from wanting more granular control over the fan control logic in some 12th generation Dell PowerEdge servers, which is handled by “platformdata” files in the iDRAC filesystem.
I began searching on Google to see if anyone else has achieved the root access required to do this, but all I could find was a post buried in an old Dell mailing list from my colleague, hinting that he achieved said root access on DRAC a couple of generations ago. The method had been long since patched, but I emailed him anyway to see if he’d be interested in helping out. Not only was he interested, but he had just picked up the exact same Dell PowerEdge 12G server I had. He was also interested in more granular thermal profile control, so we set out to find a way in, so to speak.
STH: Tell us about what iDRAC is and why it is an area that vendors spend so much time with security?
iDRAC is Dell EMC’s brand of lights-out server management. It allows you to have a unified interface for managing your server, inventory, chassis hardware, remote console & OS installs, etc. It’s a totally separate computer (running Linux, in this case) with its own storage, RAM, and everything. I think a lot of people don’t realize this, and how privileged its access to your server hardware is. That’s why IPMI/BMC controllers have become such a security focus, when not secured they become a giant backdoor to your hardware.
STH: What can you share about the attack vector? Keeping in mind that we will not print details that allow others to reproduce.
There are two attack vectors: one local with physical access, and one remote (via the web UI). The local method does not require any valid login, with physical access logins can be bypassed quite easily as you’d expect. With the remote method, you must already have a valid DRAC login. Using either method, you can elevate privileges to a root Linux shell, which then allows you to load in whatever replacement filesystem and code you want. This bypasses Dell’s usual checks. The bad part about this is it can all be done so it’s totally hidden from other iDRAC users. You can load in the compromised firmware that “phones home” and they will have no clue.
Since iDRAC is an isolated embedded computer, wiping the BIOS, OS drives, or anything else will have no effect. Depending on the iDRAC flash partition you load your modifications in, even using the iDRAC web UI to flash fresh firmware will not get rid of your modifications. When I spoke to Dell they said they will be pushing out a patch for this in a new release.
Please keep in mind you need either physical access to the PowerEdge server. To do this remotely, you must have existing iDRAC credentials.
STH: What can a malicious user do with a hacked iDRAC?
There’s the obvious stuff like remotely power your server on and off, but also more malicious things like capture your server’s console contents, send keyboard input, mount your own storage into the OS running on the server, that kind of thing. Anything you can do from the usual iDRAC UI, you can do undetected with no record of your existence from other’s point of view, unless they were monitoring network traffic.
STH: What server generations are vulnerable, given current firmware, to iDRACula?
It seems like this impacts all Dell EMC PowerEdge 12th and 13th generation products. Given BMC security has become more sophisticated as new BMC hardware is more capable, one might expect older generations to be vulnerable as well. After speaking with a lead firmware engineer at Dell, these exploits are absolutely impossible on current Dell EMC PowerEdge 14th generation servers and future generations. They went to extreme lengths to prevent these kinds of situations, even using a hardware root of trust from the bottom up. Due to technical limitations of the BMC CPU, these preventative measures were not possible in 12th and 13th generation products, or anything before that.
STH: Is what you found applicable to other vendor’s remote management solutions?
It’s been a long time since I looked at Supermicro and HPE equipment, but the last products I did look at, the physical method was absolutely possible. I have not done any investigation into the remote methods, nor on more recent products, so I can’t say. I will say from my previous experience, Dell has taken this kind of vectors more seriously than anyone else, so it certainly would not surprise me.
STH: Are there any broader implications for those purchasing servers?
The interesting vector here that I think a lot of people (especially in the second-hand/ homelab community) ignore is the past history of the equipment they buy. In about 2 hours I can compile a custom image that reports anything I want back to me, flash it in a way that is totally undetectable to the next user without physical tools to read raw EMMC flash, then list it on ebay. The purchaser will have a server that is reporting back to me and giving me extremely privileged access to their server & console, and they’ll never have a clue. They can flash the BIOS, replace hard drives, and it’ll still be there. They can even use the iDRAC management web page to flash new iDRAC firmware, and it’ll still be there (until they flash a new firmware that addresses this exploit specifically.)
This of course also assumes the iDRAC can hit a network gateway that gets out to the internet to report back, another vector a lot of people ignore. The solution here is obviously to stop putting your IPMI equipment anywhere near the Internet (even if you have inbound connections blocked, that’s still not enough). To ensure you’re getting untampered hardware in the first place, you need to make sure you’re purchasing directly from Dell EMC or an approved vendor (or Supermicro, or HP, or whatever your preferred flavor.)
8) Is there anything else that you want to share about the vulnerability and experience?
It was much more difficult than I think either of us had anticipated, especially compared to other IPMI & embedded devices I’ve used. Even with complete physical access, they had some novel protections in place. It’s clear the embedded team at Dell EMC spent a good amount of time thinking about all of this and did everything they could given the hardware capability. As previously mentioned, Dell EMC PowerEdge 14th generation and up have made all of this impossible. Dell’s response has also been very quick and communicative. It was an enjoyable puzzle to put together, working with my colleague via email on two opposite sides of the globe turned it into an enjoyable hunt. Props to him, and props to Dell for reaching out and handling this gracefully!
[Edit: 9:00AM Pacific, 30 September 2018] We published a follow-up piece from Adam Nielsen with his perspective on iDRACula. You can read that here: Broader Implications of iDRACula Vulnerability a Perspective
Dell EMC’s Comment on the iDRACula Finding
Here is the official comment we received from a Dell EMC spokesperson when we notified Dell EMC that we would be running a piece on this.
Hi, Patrick. I’ve been able to meet with our team to get a better understanding of this post. Of course, we take this and any vulnerability reported to us very seriously, as we take pride in developing systems that are as secure as possible. Through communication with the poster, we learned that he identified two areas that he felt were potential vulnerabilities. Applied remotely, with admin rights for iDRAC, he had downloaded an older firmware version with a known vulnerability and created root user access. This known vulnerability in older iDRAC firmware has already been addressed in subsequent firmware releases. The second was through physical access, as you know. We believe exposure is limited, but we are taking extra precautions to reduce any further risk.
We are going to reserve the right to post an update to this section if the company comes back with an additional response after having read this piece.
Next, we are going to discuss why this is a broader industry concern, and give our final thoughts on the iDRACula class of vulnerabilities.
Holy shit. Good job presenting. I’ve sent this to our team as a reminder we need to take this kind of security seriously and not go lax over time. I saw the post in the forums before it was taken down and I’d not put 2 and 2 together.
Dell is supposed to be the best at this………
I’ve know you could do this for awhile on regular ipmi servers.
Convenience comes with a price.
We were about to buy R730 but now we’re going to look at the R740 or R740xd.
So if I buy a new PowerEdge R630 today, ship it to our South American hub the person installing it tomorrow could do this to the server before I even see it is online? iDRAC is pre-boot so I’d have no way of knowing.
Is this an end of month Dell 14th gen sales promotion?
Was there a CVE submitted for this? Or does it not need one as an old firmware with a known vulnerability was used?
Aaron: a CVE is in the process of being submitted by dell, yes. While the remote process does involve rolling back to an older firmware, the issue/actual exploit lies in the fact the latest firmware has some holes that when used, allow you to then re-flash to the latest firmware and keep exploits in place undetected
Dell is still selling idrac 8 in new servers that can be exploited like this
Ok, so by handing a thief the key to your car and/or the key to your garage (because you chose to trust them)…. They can now take the motor out and modify it and put it back (hey that sounds fun)…. Helllllo you gave them the keys!!! They can turn it on or off anytime and have their way with it at will just by utilizing the key!! Or better yet, if I’m the owner and I hold the key I can have MY way with MY server, my choice of handing out the key or leaving the door open is the obvious security vulnerability at large. While a valid security flaw this is just a bit over fluffed. People need to learn more about enterprise best practices to prevent this, such as management vlans blocked from the internet, video surveillance and physical access control… rather than only buy 14th gen servers or learn the hard way.
John, you’re missing a huge part of the server deployment process, the supply chain. You can guard against physical access all you want once you have the equipment in-hand, but it doesn’t matter if anyone who had it previously had bad intentions, especially given as-of-now anything malicious they could potentially load will survive standard firmware re-flashes. Using your analogy, it’s more akin to buying a car that runs unsigned code on the engine control unit, and a pissed off employee at the dealer/tow company/etc loaded in their own modified firmware. The car seems perfectly fine, you keep it secure in the garage and nobody else has the keys, but one day that modified code is going to slam you into a wall.
Is this likely to happen to rinky-dink IT companies with not much to steal from? Doubtful, the chances are very low. is this likely to happen to large providers, financial institutions, etc? Yes, it already has happened. There is a reason Facebook, Microsoft, OpenBMC etc have spent millions solving the unsigned firmware issue, they have seen the damage it can do. 5 minutes of physical access to hardware to load something that will never be cleaned away is much easier for an entity to achieve in supply chains then you think, just ask any of the victims of previous exploits
This is why you never ignore defense in depth. Physically lock and secure your datacenters. IPMI traffic should be in an isolated vlan with no direct internet or end user access.
I was so excited to read about a new BMC vulnerability but this doesn’t apear to be anything new at all. Speaking strictly about the article, it seems to be a sensationalized piece about “scary BMCs oooh”. Nothing the security world doesn’t know about already. And please for the love of… stop naming these things. “integrated Dell Remote Access Controller unauthorized load access”… really??? You’re kinda grasping with that one.
If you’ve got physical access to a server you’ve already won. Not even talking about the iDRAC there are other interesting attack surfaces available as well (Internal SD card? Internal USB port? etc) but yeah the completely hidden and persistent ones are the best.
Speaking as someone who has actually discovered such a vulnerability in 2016 (CVE-2016-5685, local iDRAC8 root shell: https://www.cvedetails.com/cve/CVE-2016-5685/) I’m not sure how downgrading to an old, known to be vulnerable iDRAC release and exploting it is news. Did I miss something or is the -f[orce] flag that can be used to downgradeFW not a thing with the Dell Update Package anymore? I’m asking honestly. I haven’t checked. But that’d be an easy way to downgrade all the things by booting via Live USB.
I mean, if you managed to get root on the iDRAC on the old pre-patched version all by yourself (without hints) then in that case mad props to you for figuring it out. The fact that you’re now attacking a persistent file system and that this can’t be quickly fixed by reflashing to the same or newer version of the Firmware is known to Dell (and should be pretty obvious to anyone really) but clearly it wasn’t meant to be broadcast to the world…
Finally, the 14th Gen servers can and will be exploited… It will be harder to make these changes persistent but nothing says “challenge accepted” like a promise of complete protection from security bugs. If you have an iDRAC9 with a very early initial release of the FW you can drop to a root shell pretty easily too if you “invoke” some “debug” stuff: https://pbs.twimg.com/media/DIm-fhAUQAE8VaC.jpg
Luckily that was a very early FW release and was quickly patched…
Anyway, good on you for keeping BMCs on peoples radars as attack surfaces.
Is there an official dell support page for this? What is the CVE number?
THanks
“Please keep in mind you need either physical access to the PowerEdge server. To do this remotely, you must have existing iDRAC credentials.”
If someone has physical access to your servers you have bigger problems
Ditto iDRAC credentials..
@SA, apologies, just now seeing this. downgrading to a known exploitable firmware version is not the news, that’s however all STH was comfortable mentioning about the process and I don’t blame them. Now that a CVE has been assigned internally and a patch is coming out in a couple weeks I can probably say more: the larger exploit here is not being able to downgrade, but the process of downgrading, utilizing something that allows you to make arbitrary changes to the filesystem, and then upgrading back to the very latest “secure” firmware but keeping your changes in place undetected, persisting privilege elevation exploits regardless of how new firmware you flash to. According to dell any persistent changes like this should not have been possible, but clearly they are. There’s a couple other fun things too that wouldn’t have been good to publish months before a patch. I will probably publish more details on the github after the patch is released, but as you’ve guessed, since they don’t have hardware root of trust, one of these methods is unpatchable, and will always be possible.
And yes, very early versions of IDRAC9/14th gen firmware allowed you root access. It’s not of much use though, as you cannot load and persist anything. You can try, but on next power-on the entire boot chain from bootloader to entire EMMC flash is key checked against burned in keys in the proc and it will refuse to boot, as I’m sure you’re aware. That’s not to say there’s not some extremely niche bug in the signature checking chain somewhere, but it would certainly be far from arbitrary to exploit