Getting this Information to Dell
The lunch meeting was set for October 19, 2021. At this point, all I knew is that I had stickers that AMI headquarters had not been able to validate the authenticity of for almost two weeks. I knew it was something in the supply chain since I found other photos from other vendors. Also, I knew that the VMware spin-off was happening on November 1, 2021. The thought occurred to me that it may present a challenge if a few days before VMware, a software vendor then owned by Dell Technologies, was to spin-off this story came out. Optically it looks bad if licensed software in solutions Dell sold with VMware had misspelled license stickers that nobody had an idea of where they came from. I was fairly sure something like this would become a priority item to respond to.
On October 19, 2021, after lunch, I pulled out my iPad and showed the pictures. Of course, one could easily and already see the images in the S5232F-ON piece, on Twitter, or elsewhere. This had been public for over two weeks at this point, and I had sent that review to Dell before the YouTube comment.
Before the lunch meeting, and after the lunch, I told Dell it would have 24 hours to get me an answer or this piece would go live. I set the 24-hour timeline to test something very specific. My thought was that if indeed Dell knew about this, and had documentation filed about the stickers, it would simply be a matter of pulling the file noting the defect it had found previously, and getting approval to send it. If Dell did not know about it, then an investigation would take place and it would take over 24 hours to get me a response. Since I had done management consulting at large tech companies for years, big companies tend to move quickly on big impact issues if it is easy and exonerating answers are on hand. If not, a big company has to involve a lot of people and even moving fast, 24 hours is usually too fast.
I wanted to get the story out because it is important either way, but I also wanted to be fair to Dell and give them an opportunity. I told Dell, STH does not need more page views from a big story like this, but it is important. Personally, supply chain security is important, but it is not something I want to do. Please just let us test the world’s server, storage, and networking gear. That is also why this is being published at a slower time of year in December 2021, instead of publishing first and asking questions later. I had sticker pictures without asking Dell. I just did not want to broadside the company since frankly a lot of great people work at Dell.
By the afternoon of October 19, Dell did not have an answer and was asking for the specific resellers I purchased the switches from. With four switches, three resellers, and two countries, my response was that was the wrong way to spend time. It had to have happened upstream. If it wanted to validate it was indeed widespread, calling the RMA shop floor would probably get them easy access to switches I did not have control over and from additional customers. Of course, this question of where I purchased the switches from would not have come up if Dell had known about and documented the typo. It only took a few hours, but I had my answer as to what Dell knew prior to me pointing this out.
I was flying on the morning of October 20, but I got a ping saying they were still working on a response. Dell said that American Megatrends, which had not confirmed this to me previously, now said that there was a label typo. I agreed to hold the story. I make a lot of typos, so I know how it goes. As someone who is dyslexic, this piece is going to have a few typos when it goes live, so I empathize with this. At the same time, I tend to spell my name correctly on official documents.
On October 21, 2021, I received this official statement from my American Megatrends US contact in the afternoon:
“AMI acknowledges the misspelling in the label on AMI’s MegaRAC PM royalty label. AMI has sent out affirmation letters, which state the royalty label is authentic and fully authorized by AMI, and that there are no legal concerns to affix the ‘American Megatrands’ royalty label onto the products. The royalty label will continue to be implemented and shipped by AMI in the future up until further notification by AMI.”
Dell sent me a letter from the American Megatrends Taiwan subsidiary:
Here is page 2:
Effectively, American Megatrends is now saying that it misprinted the labels, and no legal action would come from the misprinted labels.
There are a few things to note here:
- There is redaction. My best guess is that this is the name of the ODM that manufacturers its switches. As we have seen with the S5148F-ON (the previous generation) Dell uses 3rd parties to make these high-end switches so that makes sense. This redaction could say anything, but it feels fair to assume that it is another party, most likely the ODM.
- “The royalty label mentioned herein will continue to be implemented and shipped by AMI in the future until any further official notification issued by AMI.“
Since this is dated after I brought the issue to Dell and is using STH’s image that answers our 24-hour probe. We wanted to know if Dell had seen this and had existing documentation. It seems fair to assess that it was created in response to bringing this up in October and a subsequent conversation between Dell and AMI.
So officially, we can file this as “no big deal” then right? Maybe.
Why this Should be an Industry Wake-up Call
Taking the letter and resolution at face value, this is what had to have failed in the supply chain for me to purchase multiple high-end switches with American Megatrands labels. Again, we are taking the events as explained by the letter, at face value. In this article, we are not going to entertain the potential that someone was selling counterfeit stickers to save a bit of cost and this was swept up. That came up in numerous conversations I had with folks about this and showing them the stickers, but that is not the official version, so we will not entertain that here.
Instead, here is what had to go wrong:
- AMI Taiwan needed to get license stickers for the local market. Instead of using the “American Megatrends” MegaRAC PM sticker template, it decided to make its own that had the misspelling.
- The ODM that made the switch for Dell had to purchase the “Megatrands” stickers, and put them into the switch, not noticing that they were misspelled, or ignoring that the name on them did not match its supplier’s name.
- Dell had to accept the ODM’s switch, including the BMC board with the Megatrands stickers, and not file a bug or defect.
- Dell was asking where I purchased the switch, so it seems like nobody inside Dell’s RMA department or elsewhere that handled these switches noticed it. If they had, they would not be asking where I purchased the switch. Instead, they would know about the misspelling defect in their entire line of high-end switches.
- If either AMI, the ODM, or Dell saw this previously, there was no mention of prior documentation in this. Everyone either missed it until it came up in our review, or nobody cared enough to document it. The unredacted portion of the letter from AMI did not specifically reference some other documentation of this misspelling. If that existed, it would have been explicitly referenced and there would be no need for an “Affirmation Letter” absolving Dell and its customers of legal liability in October 2021.
- These stickers are still set to ship in future products.
A few points I think are a call to action here.
First, Dell is widely regarded as being excellent at supply chain security. Part of the company’s above-average margins on hardware come from that fact. At the same time, it feels like there is a process deficiency if something like this can go unnoticed, for years. When the Bloomberg article came out alleging spy chips back in 2018, major securities exchanges were having their heads of security drive new servers to get X-rayed looking for spy chips (they, of course, did not find any.) In contrast, Dell has something here that is easy to see with the naked eye.
This was not caught by Dell, or even the STH team at first, it was a YouTube commenter. If that is how we as an industry are catching the easy plain-to-see stuff, that should scare everyone about what may be hard to see. These are high-end switches that can go into important infrastructure. If this was not caught by the supply chain, then it makes one wonder what about that $500 PC or switch.
Second, these stickers are going to continue to ship. That makes it hard to validate a legitimate royalty sticker or a counterfeit one. This is a good item to discuss whether the stickers should be re-printed.
Third, this makes me want to have the STH team open more boxes. We found this 2-3 years after these shipped. The industry would have no idea about “Megatrands” if not for this effort.
Finally, I think that Dell and AMI did a good job getting back to me in a fairly reasonable amount of time. The Dell folks knew I was not just going to blast this out with a half story and clearly erroneously spelled stickers just for page views. One has to remember that while this piece references “Dell” and “AMI” there are great people behind those companies, many of whom I have known for years.
While Dell did a good job of getting this to resolution, HPE failed entirely, and that brings us to where else one may find these misspelled but allegedly legitimate royalty stickers. If HPE installed what they showed at a trade show, then they may be in a US military supercomputer.
John Etulain of Seattle Washington registered those 2 domain names, and it is being served HTTP using STH’s SSL cert.
Staffer of yours, Pat?
Amazon Route53.
What was HPE thinking?
MegaRAC PM is so old guys. You’re missing that. It’s almost 2 decades old and being open-sourced right?
You didn’t second source that it’s misspelled properly.
A few random remarks:
1. this whole thing seems to completely ignore the fact that latin characters are a second thought in the countries where those things are manufactured. So an honest mistake is still a very credible cause here. Would anyone in the US notice a misspelled name written in Chinese characters?
2. what would prevent a counterfeiter to print labels with correct spelling? If we’re to suspect foul play and consider a supply chain compromise, why blindly trust the correctly-spelled labels and be certain that they couldn’t possibly have been tampered with, just because the vendor name is spelled right?
3. and the fact that it hasn’t been noticed before, by the vendors and manufacturers themselves? Well, exec have probably never seen one of these products in real life, let alone the insides of it. And people on the manufacturing floor probably don’t care or know enough to report it…
I’m surprised that such a glaring defect would make it through the process that produces an incredibly precise and demanding piece of gear; but the really scary thing isn’t so much a bad batch of stickers; but the fact that evidence as weak as the sticker would even cross our minds when examining the question of whether the firmware wrapped tightly around our server or switch’s brainstem is what it ought to be or something similar but less benevolent.
I’m not aware of any really good alternatives at present, though it’s an urgent problem so the search continues; but it’s terrifying that looking at the sticker would be something other than a waste of time if you want to verify the BMC and its firmware.
I searched the web for American “Megatrands” and was quite surprised to find
mentions of the name as far back as this 2002 patent application:
https://patents.google.com/patent/US6739896B2/en
The search yielded many surprising hits which might merit more investigation.
Sorry, but the name “Serve The Home” doesn’t exactly scream enterprise content.
This response only concerns me more. Letters of this nature are extremely precise and gone over with a fine-toothed comb by management, PR, internal counsel, and often external councel for backup. Every single word is lawyered up to the nth degree.
They specifically say that “licensing is not a concern.” Fine. Wonderful.
They specifically do not say that “these systems were distributed with legitimate firmware.”
They specifically do not say that “these systems were distributed with legitimate hardware.”
If they do not specifically say these things then that means that they either do not know or that they do know and they do not want to tell you.
A suspect thit thas artacle wall wand up tranding.
As a long time reader, I love this site and the content. I do agree that ServeTheHome has always been an odd choice of name. I get that alot of homelab geeks hang on this site and it’s forums and the convos we have translate back to the enterprise. I cant tell you the amount of times someone has linked me an STH article or forum post and I have said, yeah I already seen that. I don’t think I realized until that happened a few times that this site has a pretty large following in the tech world. That said, the name has always struck me as weird. Keep up the content!
Regarding ‘ServeTheHome’, does it count if Patrick’s home probably is wired up with 100G fiber and he’s running half these servers there? 😀
I still like the content because it’s kind of a preview of some of the gear that could magically filter its way down to my homelab in a decade or so.
These are still 25Gb and 100Gb switches that companies have deployed right now. For those that think it’s just a spelling error, there’s more that the article talks about. The letter says the license is still valid so that’s fine now, but would it be valid without the letter?
All I’ll say is the big one to me is that Dell and HPE told everyone how trusted supply chains for them were when the Bloomberg piece was out in 2018. Right as they’re doing that, these stickers are getting installed and they didn’t know. That’s the real problem. They’ve told everyone they’ve got supply chain checks so their gear is more valuable. This shows that they don’t really have good checks. This doesn’t take a microscope to find.
So will the youtuber who spotted it get one of these switches as a present. I mean, he deserves it, right?
@Pete Mitchel – I first read STH maybe 10 or 15 years ago, and then it was covering stuff that you might find in the home of a serious IT professional.
It’s evolved since then.
I must be missing something. Why is anybody concerned about a “license” and why do we need a written letter absolving us (somebody?) from a licence issue? Before we had the letter were we morally questioning our “license” and turning off all equipment with wrong stickers in it because it might be “ilegal”?
Come to it, why are there even stickers on the chips? I am sure you are not buying the switch because you want to be looking at stickers. Falsifying a sticker is probably 0.0001% of complexity when working on switch hw.
As long as the device functions exactly as it should, does not do less, does not do more, functions exactly as it should who cares what the sticker says? Why does the letter not mention anything about functionality and security guarantees?
My opinion is that new gear functionality (the thing that matters) is the thing that needs 100x more QA. It has become network industry practice to release alpha software/hardware and have customers act as beta testers.
At least these don’t have the Trump chip from American MAGAtrends. 🙂
At a company, to remain nameless, a product manager and a vice president of legal got into a turf war about who was in charge of licensing stickers. To demonstrate their authority, they kept having the graphics arts depart make them mockups of the stickers with various color text.
The mockups went through endless meetings and reviews to ensure they had just the right color. When, after four months of bickering, the decision about what color text to use had to be elevated to the regional vice president. He approved and everyone went on their merry way.
Alas, after spending the last three weekends pulling all-nighters, (unpaid because he was salaried) doing emergency mockups of the stickers, a graphics artist said to his cubicle mate, “After pissing away the last four months of my life on these stickers, I’ll bet these morons won’t even notice that the name of the company is misspelled.”
So the name is spelled correctly, or it isn’t spelled correctly … either way, AMI has licensed the manufacturing out to a third-party fab who probably *does* have chinese spyware installed on the chip. If it was manufactured anywhere on that side of the world, the supply chain is compromised.
I think Patrick’s point was more along the lines of “Why didn’t Dell throw a fit when they received parts from a supplier which were incorrectly labeled, and therefore could have easily been counterfeit?”
Sounds like a business opportunity. Time to start up Armenian MangaTrends and start selling competing BMC chips lol.
Amelican Megatland
Incel Inside?
American Maga Trends sounds like a terrible 4 years.
Asian bribery of import and export officers are renown. Even for stuff made under contract. If a misspelled sticker can get past these “trusted supplier” OEM’s, then they are full of crap and the offering is just marketing BS.
I know of a US based company who went BK because an Asian supplier bribed the import officer, which caused sub par product to pollute the US supply chain. It got into commercial, military, aerospace, automotive industries and caused such a large liability, the company went under as a result.
The US has gotten so lazy and so dependent on contract work globally and don’t maintain any controls over it, that even the import inspection is contracted out which makes it highly susceptible to bribery.
Why if companies like Ford, Chevrolet, VW, Toyota, etc. Can chance a mechanical piece when It has a defect, I don’t get It that Dell, HPE and others can not chance a simple sticker on a enterprise piece of HW if the customers ask for It.
STH is an independent review site, yet bends over backwards *not* to publish an article that could hurt a (major) company whose products it reviews. Give a company 24 hours, rewrite based on the reply but get the article out within the next 24 hours. Corporate divestiture be damned. The article is important, even if the root cause is a typo. The “final” piece can always say, “our research is ongoing.”
I’d be very interested in seeing this chip be compared with a correctly-labelled chip, if any can be found, using something like ABI SENTRY counterfeit chip detector (disclosure: I worked on the software for that device).
If they’re made somewhere else, then even if it’s made to the same basic spec, there’ll likely be significantly different V/I curves on the pins.