This week, eBay announced that it is building its own servers. There has been a lot of re-posting of the announcement eBay Builds Own Servers, Intends to Open Source. This is a trend among companies that are building larger sets of infrastructure. We had some thoughts beyond the simple re-post so we decided to give this announcement coverage.
In the blog post penned by Mazen Rawashdeh the company talks about how it is moving towards a more agile infrastructure to handle its data needs. Per the post, “eBay processes 300 billion data queries each day, and our data footprint is more than 500 petabytes. To put it into context, 500 petabytes is the equivalent of one trillion songs, 2.5 million hours of movies and enough to backup the American Library of Congress more than 300 times.” (Source: eBay)
eBay Builds its Own Servers
Looking at the cover image for this article and as part of the eBay announcement we see a rack of servers with A + B power. We also see power supplies located in the middle of the chassis. This picture is labeled “ODM-1-copy” in the eBay post. The other is labeled eBayodm:
Since we are the largest server hardware review site around these days, we have a lot of photos to compare this to.
Starting with the company’s cover image, these beautiful white drive trays are the classic Supermicro 3.5″ drive tray design used on almost all of their servers. Supermicro makes a few servers that do not utilize this design, for example as we saw in our Supermicro AS-4023S-TRT Review.
We did not have time to run to the STH / DemoEval data center lab to take the exact angle, but here is a photo we have from an old review at roughly the same angle. You can see those are the same drive trays, but a different color. Even the side screws to secure the server to the rails/ rack is a classic Supermicro design.
Moving to the rear image labeled “ODM 1 Copy” by eBay, we at first thought that this was a Supermicro BigTwin (our BigTwin AMD EPYC review is almost done as well.) The power Supplies are too large to be a BigTwin and the rail mounting patterns mean that the servers are 2U systems, not 4U like the FatTwin. The other rear I/O patterns also do not match up well.
We took an annotated stock image of a current generation Supermicro TwinPro and compared it to what we see in the eBay photos. Admittedly, this was easy to find since we used Supermicro Twin servers for hosting about 6-7 years ago.
You can see that the servers are exactly the same save for one detail. The stock photo uses RJ-45 networking while the eBay servers have a DAC/ optics cage (we are going to guess SFP28 not SFP+ since it is 2018.) Networking in current-generation TwinPro servers is on what Supermicro calls its SIOM design. This is similar to what we see when we review servers from Dell EMC and others where they have LOMs providing networking. For the eBay ODM analysis, this means that changing the SIOM in the company’s servers can give them SFP28 (or SFP+) cages. Supermicro only offers Intel XXV710 in the dual SFP28 form factor widely, but a customer of eBay’s size can likely get a Mellanox, Broadcom, or other NIC vendors chips implemented in SIOM.
The “Gotcha” of this Analysis
The one part that may be missing here is that eBay never said that its pictures labeled “ODM” in the post are actually its new infrastructure. They may be a previous generation stock photo for all we know. I asked via LinkedIn if they were actually the new generation of servers, but we did not get a reply.
Frankly, in the Silicon Valley Supermicro makes a lot of boxes for a lot of companies. They are known for being very flexible and quick to implement changes. For eBay to turn to Supermicro and ask to take a TwinPro and customize for their scale and needs makes a lot of sense. Supermicro is a few minutes away by car.
Perhaps the Bigger Announcement: Software Platforms
eBay has been a well-known OpenStack supporter and user for some time. As the company looks to innovate on its platform, OpenStack is not found in that article. Instead, the company seems to be moving toward open source containerized technologies:
“Open source is fueling our transformation, with technologies like Kubernetes, Envoy Proxy, MongoDB and Apache Kafka, propelling eBay to new performance levels for our customers. In a sense, open source gives us control of our destiny, offering greater capabilities to customize for eBay’s needs.” (Source: eBay)
Just about every talk I give these days has the word “container” so this makes a lot of sense. At the same time, the fact that the company sees Kubernetes as its path forward, not referencing OpenStack we feel speaks volumes.
We have a history of dissecting open source designs such as in Cavium ThunderX2 and OCP Platform Details and Second AMD Naples Server Pictures and Platform. After eBay posted its blog, we got over two dozen e-mails asking what servers those were. If the servers pictured are not of the next-gen eBay servers, then this is just fun analysis on previous systems.
Our bigger takeaway is the fact that eBay is joining a growing list of organizations like Facebook, Google, and (today’s) Microsoft that are committing to giving back to the open source communities. Since we are Silicon Valley-based, we work with startups who are now leveraging open source innovations to move technology forward. What eBay is committing to by open sourcing its innovations goes beyond building a unique infrastructure for their applications. Instead, it is one of the processions of small steps that allows new technologies to be rapidly deployed bringing new opportunities to billions worldwide.
We will bring you more as eBay announces further details.