HPE Cray “Megatrands” for the US Army Supercomputer “Freeman”
The reason I feel like HPE did a poor job, is that I gave them the same opportunity as Dell to have a 24-hour investigation window. They told me they discussed it internally and they never took the opportunity to have me share what I found here. HPE for its part showed off a HPE Cray CS500 model that also had these stickers at Supercomputing 2018. As mentioned previously, we have a pretty massive set of photos of the inside of servers, some in the studio, and others from trade shows or elsewhere.
MegaRAC PM is typically used on lower-end ASPEED BMCs than what we see in modern servers. The HPE Cray CS500 is a 2U 4-node (2U4N) server design. That means there are four dual-socket AMD EPYC nodes in a single 2U chassis. This density is what high-performance supercomputers want. At SC21, the CS500 was outfitted with a chassis management controller or CMC. This CMC allows a single point to manage all four of these nodes using a single network port. It is actually a great feature we have seen many times. Here is what one of those CMC’s looks like in a closer view.
The HPE Cray CS500, shown at SC18, had at least one supercomputer win that I could find easily. In a system that debuted at #123 in the June 2020 Top500 supercomputer list “Freeman” the HPE Cray CS500 AMD EPYC platform is noted as being used. Again, we do not know exactly what is installed, but it is listed as the CS500 in the official submission, and we saw the CS500 at SC18.
The site it is installed at is listed as the ERDC DSRC. Clicking on some detail gives us this:
As one will see, this is an Army.mil URL that takes us here:
The web page of the US Army Corps of Engineers Engineer Research and Development Center.
I know we have a lot of readers in the Washington, DC, Maryland, and Virginia areas that work for the US government, but we have fewer readers in Vicksburg Mississippi. If you happen to work there or know someone who does, it may be worth opening up a Freeman CS500 chassis to validate. The CMC is an optional feature, in those systems, but it is one that is very useful. An investigation, based on the Dell letter from AMI, will likely lead to this is OK, but it may be worth at least documenting since it is in a US military supercomputer.
Again, HPE is also supposed to also be top-tier at supply chain security as we saw in our HPE ProLiant DL380T Gen10 Trusted Supply Chain Server Teardown. I was at a wedding in Maryland this summer where one of the attendees I had never met before recognized me from YouTube and proceeded to comment on that piece and video.
We can give HPE the benefit of assuming that they already knew about the “Megatrands” sticker they showed at SC18 (and nobody online seems to have caught it for 3+ years.) Still, it is probably worthwhile for the US Army to close the loop with HPE if it has not.
In the future, it is probably worthwhile for HPE to also have a process to take supply chain reports and issue official responses. After this happened in October, I purchased the Dell Precision 3930 we recently reviewed (and several other Dell machines), but the fact that I knew how seriously HPE took this means we have not purchased any HPE products.
Final Words
All told, there is probably nothing to see here. Someone may very well have seen this before this October 4, 2021 comment on our YouTube video. It seems strange that a YouTube comment would find a supply chain story like this one.
Many are probably thinking about the possibilities here. For example, anyone can now copy the American “Megatrands” label and how would one validate it is authentic? Furthermore, what if someone decided to register the domain names so anyone doing research in the future would end up directed to another website. Surely if AMI knew of this misspelling prior to October 2021, they would have at least secured the domain name, right? It just so happened on October 22, 2021, after the AMI letter, someone did have that idea.
I wanted to ensure these domains were not sitting open and unregistered when this piece was released so I picked them up just so they did not fall into a malicious actor’s control. Someone I had shared this with suggested that if AMI really knew about the misprinted labels before I brought it to them, they would have registered the domain name at least to keep people safe in the future.
Overall, this is clearly not the industry’s finest moment. Given the state of the Internet these days, I waited until we started to get into the slower holiday traffic pattern before publishing this. On one hand, it is very important information that we investigated and uncovered. On the other hand, we know that a lot of people do not believe official statements so there is a risk that commentators may not accept the official story furnished to STH. All we can say for sure is that there are Megatrands labels the equivalent to “Wandows” license stickers being put into very high-end data center IT gear that can cost upwards of $10,000 each. We are seeing it from multiple vendors, and most notably two vendors that are supposed to be top-notch in this area.
If this can happen in the supply chain for those two companies in high-end networking gear and servers, it may be worth taking some time over the holidays to ponder if lower-cost gear could be susceptible to a Megatrands-esque event.
John Etulain of Seattle Washington registered those 2 domain names, and it is being served HTTP using STH’s SSL cert.
Staffer of yours, Pat?
Amazon Route53.
What was HPE thinking?
MegaRAC PM is so old guys. You’re missing that. It’s almost 2 decades old and being open-sourced right?
You didn’t second source that it’s misspelled properly.
A few random remarks:
1. this whole thing seems to completely ignore the fact that latin characters are a second thought in the countries where those things are manufactured. So an honest mistake is still a very credible cause here. Would anyone in the US notice a misspelled name written in Chinese characters?
2. what would prevent a counterfeiter to print labels with correct spelling? If we’re to suspect foul play and consider a supply chain compromise, why blindly trust the correctly-spelled labels and be certain that they couldn’t possibly have been tampered with, just because the vendor name is spelled right?
3. and the fact that it hasn’t been noticed before, by the vendors and manufacturers themselves? Well, exec have probably never seen one of these products in real life, let alone the insides of it. And people on the manufacturing floor probably don’t care or know enough to report it…
I’m surprised that such a glaring defect would make it through the process that produces an incredibly precise and demanding piece of gear; but the really scary thing isn’t so much a bad batch of stickers; but the fact that evidence as weak as the sticker would even cross our minds when examining the question of whether the firmware wrapped tightly around our server or switch’s brainstem is what it ought to be or something similar but less benevolent.
I’m not aware of any really good alternatives at present, though it’s an urgent problem so the search continues; but it’s terrifying that looking at the sticker would be something other than a waste of time if you want to verify the BMC and its firmware.
I searched the web for American “Megatrands” and was quite surprised to find
mentions of the name as far back as this 2002 patent application:
https://patents.google.com/patent/US6739896B2/en
The search yielded many surprising hits which might merit more investigation.
Sorry, but the name “Serve The Home” doesn’t exactly scream enterprise content.
This response only concerns me more. Letters of this nature are extremely precise and gone over with a fine-toothed comb by management, PR, internal counsel, and often external councel for backup. Every single word is lawyered up to the nth degree.
They specifically say that “licensing is not a concern.” Fine. Wonderful.
They specifically do not say that “these systems were distributed with legitimate firmware.”
They specifically do not say that “these systems were distributed with legitimate hardware.”
If they do not specifically say these things then that means that they either do not know or that they do know and they do not want to tell you.
A suspect thit thas artacle wall wand up tranding.
As a long time reader, I love this site and the content. I do agree that ServeTheHome has always been an odd choice of name. I get that alot of homelab geeks hang on this site and it’s forums and the convos we have translate back to the enterprise. I cant tell you the amount of times someone has linked me an STH article or forum post and I have said, yeah I already seen that. I don’t think I realized until that happened a few times that this site has a pretty large following in the tech world. That said, the name has always struck me as weird. Keep up the content!
Regarding ‘ServeTheHome’, does it count if Patrick’s home probably is wired up with 100G fiber and he’s running half these servers there? 😀
I still like the content because it’s kind of a preview of some of the gear that could magically filter its way down to my homelab in a decade or so.
These are still 25Gb and 100Gb switches that companies have deployed right now. For those that think it’s just a spelling error, there’s more that the article talks about. The letter says the license is still valid so that’s fine now, but would it be valid without the letter?
All I’ll say is the big one to me is that Dell and HPE told everyone how trusted supply chains for them were when the Bloomberg piece was out in 2018. Right as they’re doing that, these stickers are getting installed and they didn’t know. That’s the real problem. They’ve told everyone they’ve got supply chain checks so their gear is more valuable. This shows that they don’t really have good checks. This doesn’t take a microscope to find.
So will the youtuber who spotted it get one of these switches as a present. I mean, he deserves it, right?
@Pete Mitchel – I first read STH maybe 10 or 15 years ago, and then it was covering stuff that you might find in the home of a serious IT professional.
It’s evolved since then.
I must be missing something. Why is anybody concerned about a “license” and why do we need a written letter absolving us (somebody?) from a licence issue? Before we had the letter were we morally questioning our “license” and turning off all equipment with wrong stickers in it because it might be “ilegal”?
Come to it, why are there even stickers on the chips? I am sure you are not buying the switch because you want to be looking at stickers. Falsifying a sticker is probably 0.0001% of complexity when working on switch hw.
As long as the device functions exactly as it should, does not do less, does not do more, functions exactly as it should who cares what the sticker says? Why does the letter not mention anything about functionality and security guarantees?
My opinion is that new gear functionality (the thing that matters) is the thing that needs 100x more QA. It has become network industry practice to release alpha software/hardware and have customers act as beta testers.
At least these don’t have the Trump chip from American MAGAtrends. 🙂
At a company, to remain nameless, a product manager and a vice president of legal got into a turf war about who was in charge of licensing stickers. To demonstrate their authority, they kept having the graphics arts depart make them mockups of the stickers with various color text.
The mockups went through endless meetings and reviews to ensure they had just the right color. When, after four months of bickering, the decision about what color text to use had to be elevated to the regional vice president. He approved and everyone went on their merry way.
Alas, after spending the last three weekends pulling all-nighters, (unpaid because he was salaried) doing emergency mockups of the stickers, a graphics artist said to his cubicle mate, “After pissing away the last four months of my life on these stickers, I’ll bet these morons won’t even notice that the name of the company is misspelled.”
So the name is spelled correctly, or it isn’t spelled correctly … either way, AMI has licensed the manufacturing out to a third-party fab who probably *does* have chinese spyware installed on the chip. If it was manufactured anywhere on that side of the world, the supply chain is compromised.
I think Patrick’s point was more along the lines of “Why didn’t Dell throw a fit when they received parts from a supplier which were incorrectly labeled, and therefore could have easily been counterfeit?”
Sounds like a business opportunity. Time to start up Armenian MangaTrends and start selling competing BMC chips lol.
Amelican Megatland
Incel Inside?
American Maga Trends sounds like a terrible 4 years.
Asian bribery of import and export officers are renown. Even for stuff made under contract. If a misspelled sticker can get past these “trusted supplier” OEM’s, then they are full of crap and the offering is just marketing BS.
I know of a US based company who went BK because an Asian supplier bribed the import officer, which caused sub par product to pollute the US supply chain. It got into commercial, military, aerospace, automotive industries and caused such a large liability, the company went under as a result.
The US has gotten so lazy and so dependent on contract work globally and don’t maintain any controls over it, that even the import inspection is contracted out which makes it highly susceptible to bribery.
Why if companies like Ford, Chevrolet, VW, Toyota, etc. Can chance a mechanical piece when It has a defect, I don’t get It that Dell, HPE and others can not chance a simple sticker on a enterprise piece of HW if the customers ask for It.
STH is an independent review site, yet bends over backwards *not* to publish an article that could hurt a (major) company whose products it reviews. Give a company 24 hours, rewrite based on the reply but get the article out within the next 24 hours. Corporate divestiture be damned. The article is important, even if the root cause is a typo. The “final” piece can always say, “our research is ongoing.”
I’d be very interested in seeing this chip be compared with a correctly-labelled chip, if any can be found, using something like ABI SENTRY counterfeit chip detector (disclosure: I worked on the software for that device).
If they’re made somewhere else, then even if it’s made to the same basic spec, there’ll likely be significantly different V/I curves on the pins.