Recently Cliff sent me a blog about Inspur Systems with the comment “Is this fast? This is fast right?” Inspur Systems is now the third largest server vendor in the world. One of their key target markets is hyperscale. In the blog, the company talks about deploying 10,000 servers at Baidu in 8 hours. The math works out to (8 hours * 3600 seconds per hour) = 28,800 seconds in 8 hours. 28,800 seconds for 10,000 servers is 2.88 seconds per server. After seeing that, we wanted to do some research and see what is behind that feat.
What is L11 Integration?
The easy answer is that Level 11 or L11 is between Level 10 (L10) and Level 12 (L12) manufacturing. For some perspective, Level 1 through Level 9 deal with manufacturing the actual server, and getting an individual server to a completed server with CPU, memory, add-in cards, drives, and so forth. Here is an idea of L10 through L12 manufacturing.
Level 10 Manufacturing
Level 10 integration includes the full server assembly with OS and software integration. There one also tends to see the documentation included for the complete system here. If you buy a Dell EMC, HPE, Inspur, Lenovo, Supermicro, or another server with VMware ESXi, Ubuntu, Windows Server, or RHEL installed, and that server arrives in a box with documentation, you have a Level 10 manufactured system.
Level 11 Manufacturing
Level 11 manufacturing includes all of the server node assembly steps. Nodes are then installed into a rack cabinet assembly. PDUs are installed in the cabinet and then wired to nodes. Networking including switches and cabling are also installed. The entire setup is then tested and ready for shipment. At L11, the on-site team will de-crate the rack, install the rack onto the data center floor, plug in power and networking uplinks. The servers are then ready to go.
Level 12 Manufacturing
Level 12 manufacturing goes beyond the rack-level and moves to multi-rack level manufacturing. This includes everything in Level 11 but then also usually has full multi-rack cluster testing and can include the installation, validation, and testing of cluster management software like OpenStack or a Kubernetes cluster.
Behind the Inspur Systems 10,000 Server Installation at Baidu
Using L11 manufacturing, Inspur Systems delivered 10,000 servers to Baidu and those servers were installed in 8 hours. The racks needed to be brought to the data center floor, secured, and hooked up so this is not a node-by-node on the floor integration into the racks themselves.
One other quote stuck out as fast: “Currently, most orders can be delivered within 7 days, large orders taking no more than 14; Baidu’s order delivery took 11.” (Source: Inspur Systems)
The reason that stuck out is that getting 10,000 servers ordered to installed and operational in the data center was 11 days. For some perspective here, In Q3, 2018, Dell EMC sold around 550,000 servers according to IDC’s Worldwide Server Tracker. Dell EMC, on a single CTO server order, has lead times of two weeks or more. Being able to go from order to installation in 11 days is very fast, especially for one this large. Given that this is likely a nine-figure order, and Baidu has access to off-roadmap products, one would assume that the company’s procurement and logistics team had some advance notice that the order was about to be placed.
Outside Perspective from STH’s Hyperscale Readers
I reached out to a few of our STH readers that work at US hyperscale companies about this post. We did not get permission to post on-the-record but directionally, I think the feedback is consistent:
One of our readers said that it was not as fast as they deploy container solutions, but fast for rack server deployment in traditional data centers. They expected that Baidu would deploy in this manner.
Another reader at a different US hyperscale company said that they have unofficially done more in a day, but the Inspur and Baidu installation was still fast because it means there were no hiccups in installation.
We did not get permission to publish the third even anonymized quote. The feedback is consistent. This is simply what hyperscalers do now. Sub 3 second node deployment, in quantity, is expected by vendors who want to compete in this space.
Final Words
If you think you are fast at deploying servers, especially if you are deploying them at Level 10 manufacturing levels, you likely are not. It will take longer to open the box a Level 10 manufactured server arrives in than the sub-3 seconds a hyperscale company takes to bring up a node.
With that said, using L11 manufacturing shifts the burden of racking and cabling to the manufacturer’s team rather than the data center ops team. The work is still done, just less onsite labor is used. This also means that companies who are competing for L11 and L12 work are investing heavily in rack integration capabilities to speed delivery further.
This gives new meaning to rack and stack. O.o
Nice paid plug for Inspur Systems. $10k? More?