This week, eBay announced that it is building its own servers. There has been a lot of re-posting of the announcement eBay Builds Own Servers, Intends to Open Source. This is a trend among companies that are building larger sets of infrastructure. We had some thoughts beyond the simple re-post so we decided to give this announcement coverage.
In the blog post penned by Mazen Rawashdeh the company talks about how it is moving towards a more agile infrastructure to handle its data needs. Per the post, “eBay processes 300 billion data queries each day, and our data footprint is more than 500 petabytes. To put it into context, 500 petabytes is the equivalent of one trillion songs, 2.5 million hours of movies and enough to backup the American Library of Congress more than 300 times.” (Source: eBay)
eBay Builds its Own Servers
Looking at the cover image for this article and as part of the eBay announcement we see a rack of servers with A + B power. We also see power supplies located in the middle of the chassis. This picture is labeled “ODM-1-copy” in the eBay post. The other is labeled eBayodm:
Since we are the largest server hardware review site around these days, we have a lot of photos to compare this to.
Starting with the company’s cover image, these beautiful white drive trays are the classic Supermicro 3.5″ drive tray design used on almost all of their servers. Supermicro makes a few servers that do not utilize this design, for example as we saw in our Supermicro AS-4023S-TRT Review.
We did not have time to run to the STH / DemoEval data center lab to take the exact angle, but here is a photo we have from an old review at roughly the same angle. You can see those are the same drive trays, but a different color. Even the side screws to secure the server to the rails/ rack is a classic Supermicro design.
Moving to the rear image labeled “ODM 1 Copy” by eBay, we at first thought that this was a Supermicro BigTwin (our BigTwin AMD EPYC review is almost done as well.) The power Supplies are too large to be a BigTwin and the rail mounting patterns mean that the servers are 2U systems, not 4U like the FatTwin. The other rear I/O patterns also do not match up well.
We took an annotated stock image of a current generation Supermicro TwinPro and compared it to what we see in the eBay photos. Admittedly, this was easy to find since we used Supermicro Twin servers for hosting about 6-7 years ago.
You can see that the servers are exactly the same save for one detail. The stock photo uses RJ-45 networking while the eBay servers have a DAC/ optics cage (we are going to guess SFP28 not SFP+ since it is 2018.) Networking in current-generation TwinPro servers is on what Supermicro calls its SIOM design. This is similar to what we see when we review servers from Dell EMC and others where they have LOMs providing networking. For the eBay ODM analysis, this means that changing the SIOM in the company’s servers can give them SFP28 (or SFP+) cages. Supermicro only offers Intel XXV710 in the dual SFP28 form factor widely, but a customer of eBay’s size can likely get a Mellanox, Broadcom, or other NIC vendors chips implemented in SIOM.
The “Gotcha” of this Analysis
The one part that may be missing here is that eBay never said that its pictures labeled “ODM” in the post are actually its new infrastructure. They may be a previous generation stock photo for all we know. I asked via LinkedIn if they were actually the new generation of servers, but we did not get a reply.
Frankly, in the Silicon Valley Supermicro makes a lot of boxes for a lot of companies. They are known for being very flexible and quick to implement changes. For eBay to turn to Supermicro and ask to take a TwinPro and customize for their scale and needs makes a lot of sense. Supermicro is a few minutes away by car.
Perhaps the Bigger Announcement: Software Platforms
eBay has been a well-known OpenStack supporter and user for some time. As the company looks to innovate on its platform, OpenStack is not found in that article. Instead, the company seems to be moving toward open source containerized technologies:
“Open source is fueling our transformation, with technologies like Kubernetes, Envoy Proxy, MongoDB and Apache Kafka, propelling eBay to new performance levels for our customers. In a sense, open source gives us control of our destiny, offering greater capabilities to customize for eBay’s needs.” (Source: eBay)
Just about every talk I give these days has the word “container” so this makes a lot of sense. At the same time, the fact that the company sees Kubernetes as its path forward, not referencing OpenStack we feel speaks volumes.
Final Words
We have a history of dissecting open source designs such as in Cavium ThunderX2 and OCP Platform Details and Second AMD Naples Server Pictures and Platform. After eBay posted its blog, we got over two dozen e-mails asking what servers those were. If the servers pictured are not of the next-gen eBay servers, then this is just fun analysis on previous systems.
Our bigger takeaway is the fact that eBay is joining a growing list of organizations like Facebook, Google, and (today’s) Microsoft that are committing to giving back to the open source communities. Since we are Silicon Valley-based, we work with startups who are now leveraging open source innovations to move technology forward. What eBay is committing to by open sourcing its innovations goes beyond building a unique infrastructure for their applications. Instead, it is one of the processions of small steps that allows new technologies to be rapidly deployed bringing new opportunities to billions worldwide.
We will bring you more as eBay announces further details.
Patrick wrote:”… “eBay processes 300 billion data queries each day …”
There are 7 billion people on the planet earth.
Dividing the daily 300 billing queries by 7 billion is 42 daily queries per person per day.
Not credible, to me. Most of the earth’s 7 billion people probably do NOT even access ebay.
data queries does NOT equal to visitors.
one page load may produce a tens/hundred of data queries.
one visitor usually request tens to hundreds of ebay pages.
so, those numbers look true to me.
iq100 I think Evgenij nailed it. eBay also has external API calls and such. Pages generate many queries.
Integrity of metrics, depends on documenting how the metric was obtained.
Where is the measuring code used? Did it count API calls?
I guess each dot on a page can be viewed as a query response to some “API”, LOL :-).
There are certainly less that a billion of the 7 billion earth’s population that ever access ebay.
A lot less.
Even 300 page queries per person per day is NOT credible.
Publish the code used to provide the metric.
I trust STH does not provide their own metrics, without documenting the environment and code being used that calculates the numbers. I am impressed that STH usually runs a test multiple times, and measures the variations, and repeatability before publishing same.
STH didn’t provide the metric, they merely repeated what ebay quoted in their own release that you can read in the hotlink next to the quote. Your request for more transparency around the stated 300B daily queries metrc should properly be directed at ebay. STH can’t validate every number put out in press releases by others.
iq100 – eBay provided the metric which we quoted and sourced. I am not sure why you think we can validate eBay’s internal system numbers?
Patrick wrote>”… I am not sure why you think we can validate eBay’s internal system numbers? …”
TX wrote>”… STH didn’t provide the metric, they merely repeated what ebay quoted in their own release ..”
Yes I knew that. My apologies, if my writings did not make that obvious.
I hold STH to high standards. Repeating possible falsehoods, are still possible falsehoods.
Not providing the environment, and access to the code used for testing, should IMHO cause STH to NOT publish the unverifiable metrics.
A link to ebay’s promotional URL should be used in such case. Clicking on a STH provided link to ebay, would make clear who is responsible for publishing such, to me, incredible metrics. As they say ‘buyer beware’. 🙂 LOL.
Adding,
Patrick, do you really think ebay gets 300 billion queries per day, when the earth’s population is 7 billion, most of which have never accessed ebay? You have a reputation for analysis to protect. I guess, if each pixel displayed on a page provided by ebay could be counted, then the 300 billion might be credible lol :-).
iq100 – “gets 300 billion queries per day” ebay.com is a top 50 worldwide website. Are you equating a query to a person searching the site? Each search likely can generate hundreds or thousands of queries. There are also automated API calls they are servicing. It is certainly plausible at that scale that they can claim this so I am not sure why this seems unreasonable?
uh… I don’t understand earth’s population is 7b and how that equates to a claim of 300b queries being false? Are you assuming that ebay is a small site? Are you assuming each page view is a single query or something?
I don’t understand why you’re attacking STH credibility on this when it sounds like you don’t know how this stuff works. Just take a look at requests when you load web pages. Go to Yahoo or a tech site like Tom’s Hardware and you’ll probably see 250-300 requests per page view. Each request is a query getting made, responded to and then a value returned. Some requests may spawn multiple queries.
I’m dumbfounded why anyone would doubt that number and address it here.
Patrick wrote>”… It is certainly plausible at that scale that they can claim this so I am not sure why this seems unreasonable? …”
Granting that Ebay uses 300 Billion queries each day to satisfy “175 million active users”.
To me, that is like designing a system to provide global transportaton needs of 7 Billion people, by using a network of turtles with connection points to transfer a person on their back 🙂 LOL.
Perhaps that is why Ebay is undertaking this “effort to replatform and modernize our our backend infrastructure”.
STH should analyze the algorithms used by Ebay (I do NOT think Ebay provides all the source code they use??), besides the hardware. The design of such algorithms effect speed more than hardware. My background is in CAP-like theorem concerns.
https://en.wikipedia.org/wiki/CAP_theorem
Great analysis STH.
As for comments at the end I’ve got no words for iq100. I’m dumbfounded.
brianmc and Hans Winkle both use the ‘dumbfounded’ word.
I suggest reading/re-reading the wiki CAP theorem and its references.
https://en.wikipedia.org/wiki/CAP_theorem
Ebay’s effort to “replatform and modernize our backend infrastructure”, “decentralizing”, “edge computinng”, and “leveraging the latest in open source” means to me that this is as much about new algorithms as faster hardware. Anyone know if Ebay publishes its source code and/or references to such ‘open source’ architecture it uses?
Dumbness is a state of ignorance. I trust the term is used here as self admissions, and not adhominem aspersions. We are all here to learn. Hopefully dumbness becomes enlightenment with knowledge.
LOL @ iq100, you have no idea what you are talking about, just stop and go away please, you are making a fool of yourself.
Navi,
No one is forced to read another poster’s words.
To demand another “go away” should IMHO violate a free exchange of collegial spirited ideas. Shame on you.
Patrick should banish you, if you continue such behaviors, here.
I do not know servers. I do know eBay. They don’t “do” open source/giving back to the community. I would look for any improvements they make to the code to be proprietary.
Tim Walter wrote>”… I would look for any improvements they make to the code to be proprietary. ..”
I agree, unfortunately.
Often references to “open source” is lip service, self service commercial efforts.
Thanks for an on target response.
oops, typo, should be “… self serving commercial efforts …”