Today we are going to share the result of a bit of investigation that started a few months ago on STH. The short version, it appears as though the Dell EMC S5200-ON series switches, the company’s high-end 25GbE-200GbE switches, have license/ royalty stickers that have a different company name on them than they should have. Instead of saying “American Megatrends”, they instead said “American Megatrands”. To give some perspective, this looks strange because it would be like buying a Dell notebook and getting a “Macrosoft Wandows” license sticker on it.
While we have been able to confirm these stickers on four different models in the Dell S5200-ON line, we have reason to believe they go beyond just Dell. If you happen to work on a certain US Army supercomputer by HPE Cray you may want to open up the system and have a look. More on that later since we do not have those HPE systems in the lab.
Through a fairly rough October, we validated that indeed these stickers are in the wild. Ultimately, after we brought their existence to American Megatrends (AMI) and Dell’s attention (HPE did not care enough to investigate), we now have an artifact that says that American Megatrends is honoring the license stickers and will not pursue legal action against Dell’s customers or those using them.
Since we have an official response from AMI and Dell, instead of going out there weaving untrue or bending facts as others have (I did the follow-up Yossi Appleboum interview on How Bloomberg is Positioning His Research Against Supermicro for reference, and that was eye-opening), all we can do is explain what we saw, what our process was, what the response is, and the impact now that we have all of this information. If you agree or disagree with this, all I can say is feel free to take this to whatever forum you would like online to comment on it.
Since this was found by a STH YouTube viewer, we are also going to have a video version of this piece. You can find it here:
As always, we suggest opening this in a new browser, tab, or app for a better viewing experience.
For Context: The Only Technical Background You Need
At STH, we do a lot of in-depth technical articles. This particular article is going to be around the baseboard management controller or BMC. That is actually the main chip in the complex Bloomberg was trying to highlight in its story. We have an entire article on Explaining the Baseboard Management Controller or BMC in Servers. Effectively, think of a BMC as a small computer within a computer that operates independently. That independent operation allows features on servers, switches, storage, and other IT items to power on/ off, update firmware, provide remote access instead of needing to physically troubleshoot at a box, and even do things like aggregate, log, and report sensor readings. BMCs are fundamental to modern IT.
On the Dell EMC PowerEdge side, the company’s solution is called iDRAC and that is a custom hardware and software solution.
Still, Dell actually uses the ASPEED BMCs in many of its products. Even the old Dell PowerEdge C6100 utilized ASPEED BMCs because large hyper-scale customers want industry-standard management. While iDRAC has many great features, for the majority of the industry the ASPEED BMCs are the go-to option.
Since the BMC is a small computer that runs mostly independently from the main server, storage, or network device, it needs its own OS and applications. That is where American Megatrends steps in. This is a bit of an oversimplification, but AMI MegaRAC one can think of like the Microsoft Windows of the mainstream BMC world. AMI has a number of versions, and there are different feature levels in BMCs, but at a high level, that is what is going on.
Now that we have that basic technical context,
Discovering the Dell Megatrand Issue
In August 2021, I made the decision that we needed to do more hands-on switch reviews. We had done the Edgecore AS7712-32X, Celestica Seastone DX010, Arista DCS-7060CX-32S, Innovium Teralynx 7-based 32x 400GbE Switch, and more, but it seemed like it was time to look at other high-performance switches on STH.
After chatting with Dell about reviews at the end of August, I was told someone in a Dell meeting said something to the effect of “why should we care about STH?” For context, STH is the largest editorially independent server, storage, and networking review site in the world, by a wide margin. Most of the coverage of enterprise gear is done by sites that allow vendors to write or review articles before they are published in exchange for a fee or preferential treatment. STH has never allowed this, and while it craters revenue, I am happy to do so to ensure the site keeps growing. We have been doing reviews of Dell products for years, so the comment was more of a normal process they go through when evaluating activities, and not meant in a malicious tone. Still, as an A-type personality, that immediately triggered a response of “OK, I am just going to pick a product, and we are going to do a review and get more page views/ video views than Dell.” Since I was on network switches, that became the focus.
On September 5, 2021, we purchased a few switches. First up was the Dell S5148F-ON piece where Dell used Marvell instead of a Broadcom switch ASIC. Datacenter switches that were $10K+ new are a niche category, but it still did fine for STH since we are a niche site.
With that, it was time to look at other switches and from the newer series. We looked at the subsequent Dell S5232F-ON Hands-on A Vastly Improved 32-port 100GbE Switch.
As one would imagine, we did these switches including photography, testing, and all of the B-roll as a set, and then published over time. After publishing that video, we had a very interesting comment on YouTube:
To be fair, we had both the Dell S5232F-ON and the S5296F-ON pieces done, and never had noticed that there was a misspelled sticker. A quick post on Twitter and it did not seem like there was a lot on MegatrAnds versus the correct Megatrends.
Here is the photo in question from the original piece:
Here is a closer view:
Here it is even closer. I wanted to show the full zoom-in progression just to show how amazing it is that someone saw this on the YouTube video. Also, just how hard it is to miss. This sticker should say “AMERICAN MEGATRENDS” not “AMERICAN MEGATRANDS”.
Of course, we checked the Dell EMC S5296F-ON piece that the video was already back from the editor and just waiting for a publishing slot. We do try to space these out a bit, but that is a challenge when you have spent tens of thousands of dollars on switches and the content is just sitting there waiting to be published.
Sure enough, there was an “AMERICAN MEGATRANDS” sticker here as well. You can see them in the review, but we specifically edited out this photo as well as in the S5296F-ON video when it went live in mid-October. The investigation was ongoing at this point, but we had two S5200-ON data points when we pulled the images so as not to start a firestorm. Bloomberg’s story never showed the alleged hardware, yet here was something that even a non-technical person could see is not right. It was too dangerous to put online.
On October 4, I sent a note via the American Megatrends website’s legal form alerting them to finding “Megatrands.” I was contacted by AMI on October 5, 2021. I was asked not to share the conversation, so I will respect that. All I will share is that on October 5, there was no answer as to why this was, even after AMI had enough time to do research the next day. If it was something AMI headquarters knew about as a well-known typo, they did not tell me on that call. They needed time to do more research because, just like finding a “Wandows” license sticker on a PC, this looked off to both parties.
I let Dell know via a phone conversation that I had found something that did not look right in two of its switches. We had Dell Technologies Summit on October 13, and I did not want this to become a topic. Plus, everyone at Dell was busy with that event. Instead, I scheduled a lunch with Dell on October 19 to show them what I had found. I will get more into my thought process for disclosure in a bit. During the conversation though, I was asked a few questions. One was simply to the effect of, “how do you know the switch was not tampered with by the reseller?”
That was a great question. The S5148F-ON that used a different management solution and the S5232F-ON and S5296F-ON all came from the same reseller in the southeastern US. It could have been a case of supply chain tampering on the path to me. On the other hand, remember these are 2019 era (so still very current) $10,000+ switches so investigating further is not cheap, especially since we have humble budgets compared to the hardware we review.
Meanwhile, on October 7, 2021, we purchased a Dell S5248F-ON from a reseller in the midwest. We also purchased a S5224-ON from an international reseller. These two we did not do our reviews of because we were waiting to investigate Megatrands. Sure enough, though, we saw some things.
The following week, before Dell Technologies Summit, the switches arrived. Here is the S5248F-ON from the midwest reseller:
Here is the S5224F-ON from the international reseller:
We now had four models, from three resellers, in two countries all with “AMERICAN MEGATRANDS”. This was not just a rogue reseller. It seemed to be part of the high-end Dell switch platform. At this point, I had also gone through our archives and I found a few instances of these stickers with “Megatrands” that I had never noticed previously from the 2018-2019 era or so. This appeared to be more than just Dell being impacted, but I had four of these switches sitting in our studio.
I knew all of this by Friday, October 15, 2021. American Megatrends was investigating but did not have an answer over a week later. The following week, it was time for the sit-down with Dell.