A few weeks ago, we broke a story about finding something in high-end Dell EMC switches and possibly HPE-Cray supercomputer nodes that should not belong. Specifically, we found stickers that said American “Megatrands” and not “Megatrends” as the supplier company is named. As part of that story, we had a small experiment running, and this highlights just how hard it is to catch even the easy-to-spot deltas in IT equipment. After 130,000 views on YouTube, we have some data to bring to that discussion.
The Easter Egg Experiment
As part of that piece, we showed this timeline that charts our efforts and interactions with Dell EMC and American Megatrends (AMI) during the investigation:
Here, the pertinent box is the first box “October 4-7” in the Dell swimlane. This says “Dell question on if the hardware could be a rogue reseller.” That is the version from the STH main site article. We also had an accompanying video, and that is where we hid the Easter Egg as a bit of an experiment.
What others may have seen is the version of the above chart that we snuck into the accompanying video:
The difference is a small one that most will subconsciously skip. This was a small experiment to see how many people would point out the “rogue” to “rouge” swap.
After two weeks, and around 130,000 views, we have an answer: only 17. I replied to all of those who pointed it out.
Of course, not everyone will have watched and found the Easter Egg. It was actually in several places throughout the video. Just based on where people were watching on the timeline, it seems like at around 130K views it had around 100K folks seeing the chart. Many even if they have the video up are multi-tasking while it is running. Those folks are not necessarily going to see something like this. Even those that spotted the swap may not have taken the time to comment.
Still, those working in factories have other things going on so they may similarly be distracted. They may also not be incentivized to speak up even if they see something out of place. The point is, for someone to find something and speak up seems to be quite rare.
Really, the key takeaway of this unscientific experiment is just how hard it is to spot anything out of place in modern hardware. There are thousands of components and the sticker is perhaps the easiest to read.
Perhaps the biggest lesson learned here is just how vulnerable the supply chain for modern IT equipment really is. In 2022, we are going to see a bigger focus on hardware security. We recently discussed how Lenovo Vendor Locking AMD Ryzen-based Systems with AMD PSB at the consumer level after we saw it on the server and workstation side. Cloud providers are building their own hardware root of trust to help validate components and firmware. Even Intel and its partners are adding FPGAs for its PFR feature to Ice Lake Xeon generation systems.
What is clear is that this is an industry challenge and one that we are going to see taken up more this year. Relying on a supply chain to spot differences visually if it is anywhere near a 17 in 100,000 Easter Egg rate is not going to be sufficient to protect the world’s infrastructure.