At STH, we have had a Dell EMC PowerEdge R640 1U server in the lab. It has been poked and prodded for weeks and we wanted to show why this is an absolutely brilliant machine. We were able to review this system from a unique perspective. We were able to test the system with a pair of CPUs from all five major lines in the new Intel Xeon Scalable family. This is going to be a long and comprehensive review.
Dell EMC PowerEdge R640 Test Configuration
Dell EMC sent an absolutely awesome configuration. Our test system was configured with dual Intel Xeon Platinum 8180 CPUs. These are essentially the highest-end 28 core / 56 thread CPUs that Intel offers. Here is a quick summary of the table.
- Server: Dell EMC PowerEdge R640 1U 10x 2.5″ Chassis
- CPUs: 2x Intel Xeon Platinum 8180, 2x Intel Xeon Gold 6152, 2x Intel Xeon Gold 5119T, 2x Intel Xeon Silver 4116, 2x Intel Xeon Bronze 3106
- RAM: 384GB in 12x 32 DDR4-2666MHz RDIMMs
- Storage Controller: PERC H740P
- SAS3 Storage: 5x Samsung PM1635a 400GB Mixed Use SAS3 12gbps SSDs
- NVMe Storage: 2x Samsung PM1725 1.6TB U.2 NVMe SSDs
- OS Storage: BOSS controller card with 2x 120GB m.2 sticks
- Networking: Mellanox ConnectX-4 Lx dual-port 25GbE
The Intel Xeon Platinum 8180 CPUs are absolutely awesome, however, we felt that the $10,000+ list price each CPU was not representative of what most buyers are going to purchase. Intel sent us a quad Xeon Platinum 8180 system for the Xeon Scalable launch and told us that the Platinum 8180 was not a general-purpose SKU. Instead, it expects volume to come from the Gold and Silver lines.
Those CPUs push the price of the configuration Dell EMC sent, using standard discounts, to just over $44,000. We do want to note that this is well over the price of a base system at just over $2,700 so there is a lot of room to scale on this platform.
Since this is STH, and we have all AMD EPYC and most Intel Xeon Scalable SKUs available in-house, we went off script. We installed a variety of dual CPU configurations to show a broader gamut of what the Dell EMC PowerEdge R640 can do. Dell EMC did not know we were going to do this, but our readers demand to evaluate a gamut of options.
Dell EMC PowerEdge R640 Hardware Overview
The Dell EMC PowerEdge R640 is a 1U server with an attractive bezel. Behind that bezel is an extraordinarily customizable system. Our particular system came in a 10x 2.5″ hot-swap front bay configuration which we will cover more in-depth in our storage section.
Dell EMC PowerEdge R640 FrontOn the right side of the server is the server tag along with power buttons and front I/O ports (VGA and USB) for servicing the machine from a cold aisle. The VGA connector is hidden behind the locking bezel which provides some measure of physical security.
We will cover the other side of the front panel as we discuss mobile management options. Next, we wanted to show the rear I/O.
There is an RJ-45 Ethernet port for out-of-band iDRAC management. Many organizations have a separate physical network for OOB management interfaces and the left side orientation allows for easy cabling. There are legacy serial, VGA and a pair of USB 3.0 ports on the rear panel for connecting KVM carts in the data center.
Unlike many commodity servers, the new PowerEdge R640 does not have 1GbE ports or PCH provided 10GbE. Instead, it relies upon a custom mezzanine card to provide networking tailored for a customer’s environment.
At STH we work with servers from every major vendor, and many niche players, on a daily basis. Again, we need to take a pause and acknowledge the awesome mechanical design team that Dell EMC has. Here is a simple example of the how Ready Rails allow one to pull out a server, open the lid for servicing while still sitting in the rails. One may think that by 2018 this is a feature all servers would have, but we have tested $100K+ servers that do not have this simple and useful feature. The PowerEdge R640 was in one of the middle of aisle racks in our data center lab so simple mechanical design features like this make the machine immensely easier to service.
Looking at the inside of the server from this pulled out in-rack servicing position, we see a few major sections. In front of the fan bank is dedicated towards storage connectivity and hot-swap drive bays.
After the fan bank, we see two CPU heatsinks that keep our processors cool. The CPU sockets are each flanked by twelve DDR4 DIMM slots for a total of 24. Unlike previous generations, the Intel Xeon Scalable platform has six DDR4 memory channels so we suggest installing either 12 or 24 DIMMs total in dual CPU configurations. The upshot is that Dell EMC and Intel see a greater than 50% improvement in memory bandwidth and one does not get severe down-clocking penalties for using 24 DIMMs as they did in the Intel Xeon E5 era.
After the fan bank, there is storage connectivity to the motherboard in the form of a PERC controller slot, and motherboard PCIe lanes for U.2 NVMe storage connectivity. By using a custom PERC slot, the PowerEdge R640 does not use a valuable PCIe expansion slot for adding high-end RAID functionality.
The rear of the unit has redundant power supplies, the 25GbE mezzanine networking slot, the Dell EMC BOSS (more on this in the storage section,) and two PCIe 3.0 expansion slots via risers.
The design of the server is extremely well thought out. Cables are tidy and one can clearly see why the company calls this a 14th generation server. Everything is well labeled and the PowerEdge team did a great job designing the system for quick servicing.
If you spend a lot of time around servers from different vendors, like we do, you see the small things that make servicing the server in a data center easy. We did five processor swaps in the system and they took under five minutes to replace each pair. Part of the reason it was so quick was this attention to detail. That CPU heatsink retainer clip, shown above, is a prime example of things that make servicing so easy. Dell’s high-performance heatsinks have two Torx retainers, like the standard Intel design for LGA3647. Unlike the standard Intel design, the corners use these retention clips instead of two additional Torx screws. On a dual processor swap, that simple innovation means it took us around two minutes less than on a standard quad Torx heatsink. That little innovation directly reduces downtime if one is upgrading processors.
Cooling the heatsinks for the 205W TDP Intel Xeon Platinum 8180’s was a wall of redundant “High Performance” fans. Fans generally have low failure rates, but since there are many in the system, it was great to see that the PowerEdge team made these easy to swap and properly orient. Some minimalistic 1U server designs have fans that are either hard to swap or do not seat with confidence. The PowerEdge R640 design excellence carries into these details.
The rear expansion slot setup is fairly ample for a 1U server. You are not going to fit multiple full-length GPUs in this system but for standard load-outs, this is a great setup. Our test unit came with a configuration that had a 25GbE mezzanine card and BOSS installed which left two PCIe 3.0 x16 riser expansion slots. One area that we would have liked to have seen is a tool-less retention for the mezzanine card.
Underneath these risers, we can see more of the motherboard including the TPM slot. The R640 has multiple TPM modules that can be installed.
Even the nooks and crannies of the server have functionality that is surprisingly easily accessible. For example, here is a VFlash card reader and 16GB VFlash SD card that is simple to swap out if needed.
On the other side of the PCB, just above where the power supplies mate to the server, there is an internal USB 3.0 Type-A header. These internal headers are often used for embedded boot applications and some legacy USB license key implementations where you do not want the USB dongle with a costly software license activation hanging off the rear of a server.
While the PowerEdge R640 excels in mechanical design, customization, and packing features into the 1U form factor, it has another advantage: management software. Dell EMC is able to leverage its investment in iDRAC, Open Manage, and supporting technologies to minimize your need to ever touch the hardware.
Dell EMC PowerEdge R640 Management Overview
We are going to focus our management discussion on iDRAC and some of the tools to manage an individual server. The company has offerings to manage fleets of servers, but that can be several additional reviews.
The PowerEdge R640 utilizes iDRAC 9. One of the first things we noticed was the snappy responsiveness of the web UI. With this generation, Dell EMC upgraded the CPU that runs iDRAC. This means that the PowerEdge server is able to collect more data, send more data to fleet management controllers, and more notably, render pages faster.
We use a lot of these web management tools since our lab has racks of gear from dozens of vendors. Some are fast with far fewer features such as Supermicro’s IPMI. Some have a lot of features but are slow. For example, if you have used a Lenovo Xeon E5 generation system’s IMM, you have had time to contemplate whether a sundial is an appropriate tool for timing the page loads. With iDRAC 9, the system is responsive.
We even validated that that responsiveness continued on the road. With the server in the data center in the Silicon Valley California, we remotely operated the UI over an IPsec VPN from Singapore and Taipei and the page loads were still relatively on par with what we would expect given the geographic latency. If you have remote administration teams that sit on another continent, iDRAC is a pleasant experience while some other, slower solutions, are rough on the admin.
The dashboard provides a simple UI to see status at a glance and directly launch IPMI management. One can also place the system into lockdown mode from the More Actions menu in the event you need to increase security.
The iKVM feature is a must-have feature for any server today as it has one of the best ROI’s when it comes to troubleshooting. iDRAC 9 features several iKVM console modes including Java, ActiveX and HTML5.
Modern server management solutions such as iDRAC are essentially embedded IoT systems dedicated to managing bigger systems. As such, iDRAC has a number of configuration settings for the service module so you can setup proper networking as an example.
In terms of monitoring, iDRAC has a basic dashboard that gives stats. This can often be useful as a sanity check. For example, this system is running at 97% CPU utilization so we may want to see why it has such a high CPU load.
Although the individual metrics are interesting, the larger picture implication is that this monitoring can feed into larger monitoring and management solutions. Dell EMC has their own tools and is also active in the industry so if you use a 3rd party tool there is a high probability iDRAC is supported.
Part of that data collection involves simple tasks such as inventorying systems. It is common to have servers that were purchased at different times to have slightly different configurations. iDRAC can offer this data to management tools.
A standout feature we wanted to show was a BIOS configuration page. One can make BIOS changes via a web UI. That is astounding. If you (or your IT admin) has ever had to make a BIOS change using a legacy method, this is a huge benefit. Using iKVM was a major upgrade to the process but it was still onerous. An admin would remotely reboot a system and furiously attack the DEL, F2, or other keystroke to ensure they entered the BIOS. From there, some of the less advanced 2018 BIOS setups still look like their UI designers idolized early 1990’s DOS programs. Some of the more advanced BIOS setups look more like Windows 2000 era programs. By having BIOS configurable using the iDRAC, one can use a modern UI and avoid that unpleasant process.
From the screenshots provided, you can see there is a lot more to iDRAC 9. If you want to learn more, go try it.
Dell EMC PowerEdge R640 Quick Sync 2 Overview
Our Dell EMC PowerEdge R640 came with another slick feature called Quick Sync 2. This is not Intel “Quick Sync” that uses Intel iGPUs to accelerate video transcoding. Instead, it is an immensely useful tool for in-data center server management.
On the left side of our R640 chassis, there is a button that lights up when pressed. That lets you head to your mobile device (iOS or Android) and connect to your server, wirelessly.
You need the Dell Open Manage application which even tells you to press the Quick Sync 2 button if you forget. As you can see, once we did this, the server was discovered in the mobile app.
For those that are thinking that this solution is inherently ultra insecure in a multi-tenant data center, Dell EMC is one step ahead of you. The next step is to pull out the service tag and scan the QR code. That offers at least that someone has physical access to your server instead of simply scanning the airwaves.
We are going to make a comment on this implementation. In general we like it, however, there was another QR code atop the tab. That other QR code was the wrong code. If you only occasionally use Quick Sync 2, you may forget where to look. It would be great if the server could have one QR code instead of multiple.
Once you are connected, you have access to the server. For example, we can see the service tag information as well as get server inventory information.
There are facilities (e.g. inside AWS data center “Red Zones”) where you cannot bring mobile devices. For those types of applications, this is not an overly useful feature and those customers are unlikely to get servers with Quick Sync 2 anyway. If you do want to take advantage of the feature, Quick Sync 2 on the R640 is immensely useful.
Dell EMC PowerEdge R640 CPU Performance
As mentioned earlier in this article, we swapped sample pairs of processors from throughout the Intel Xeon Scalable range. We did not tell Dell EMC that we were doing this so they had no input on the process nor the CPUs used. At the same time, we wanted to provide a view of the different CPU levels. Intel has around 50 public domain Xeon Scalable (Skylake-SP) SKUs and the STH/ DemoEval lab has just over half of them on hand to use for testing. We did not have time to run through every set, so instead we picked a few samples to show how incrementing CPU levels impact performance.
Running through our standard test suite generated over 1000 data points for each set of CPUs. We are cherry picking a few to give some sense of CPU scaling.
Python Linux 4.4.2 Kernel Compile Benchmark
This is one of the most requested benchmarks for STH over the past few years. The task was simple, we have a standard configuration file, the Linux 4.4.2 kernel from kernel.org, and make the standard auto-generated configuration utilizing every thread in the system. We are expressing results in terms of compiles per hour to make the results easier to read.
Moving from Intel Xeon Bronze 3106 to the Intel Xeon Platinum 8180 provides significant benefits. We generally tell our readers that Bronze 3104 and 3106 parts are “light the platform” parts which should only be used in applications where CPU performance is nearly irrelevant.
c-ray 1.1 Performance
We have been using c-ray for our performance testing for years now. It is a ray tracing benchmark that is extremely popular to show differences in processors under multi-threaded workloads. We are going to use our new Linux-Bench2 8K render to show differences.
This benchmark scales well with threads and clock speeds. As a result, the dual Intel Xeon Platinum 8180 configuration pulls well ahead of even the high-end Intel Xeon Gold 6100 series parts.
7-zip Compression Performance
7-zip is a widely used compression/ decompression program that works cross-platform. We started using the program during our early days with Windows testing. It is now part of Linux-Bench.
Here we wanted to highlight that there is generally a big jump between the upper part of the Intel Xeon Gold 5100 series and Gold 6100 series. Dell EMC has dozens of CPU options so there is a lot of room to tailor machines to your specific needs.
OpenSSL Performance
OpenSSL is widely used to secure communications between servers. This is an important protocol in many server stacks. We first look at our sign tests:
Here are the verify results:
OpenSSL is used just about everywhere. This is a good example where one can configure higher-end processors and better leverage the per-system investment.
Chess Benchmarking
Chess is an interesting use case since it has almost unlimited complexity. Over the years, we have received a number of requests to bring back chess benchmarking. We have been profiling systems and are ready to start sharing results:
Here the extra cores and thermal headroom allow the dual Xeon Platinum 8180 configuration to stretch its legs and show a convincing value proposition.
GROMACS STH Small AVX2/ AVX-512 Enabled
We have a small GROMACS molecule simulation we previewed in the first AMD EPYC 7601 Linux benchmarks piece. In Linux-Bench2 we are using a “small” test for single and dual socket capable machines. Our medium test is more appropriate for higher-end dual and quad socket machines. Our GROMACS test will use the AVX-512 and AVX2 extensions if available.
With this generation, AVX-512 is the hot new feature. This is a decent illustration of why. The Intel Xeon Bronze, Silver and Gold 5100 series CPUs have only a single port FMA while the Intel Xeon Gold 6100 series and Xeon Platinum series have the fully enabled hardware stack. If you compare this chart to others above, you will notice a relatively huge chasm that highlights the impact of the new instructions. More applications are taking advantage of AVX-512 and this shows why you may want to move up the stack if yours does.
Dell EMC PowerEdge R640 Storage
We do not frequently start with OS storage, but one of the “cool” features of the Dell EMC PowerEdge R640 test system is the Dell BOSS. This card has mirrored 120GB M.2 drives using a Marvell chipset, which can be used to boot into an OS or hypervisor such as VMware ESXi.
This allows Dell EMC to use higher-quality NAND than is found in most SD cards/ USB drives, to present an OS boot disk. Lest you think that the BOSS name is simply to get smirks from IT procurement and that this is simply a normal card, it even shows up as the DELLBOSS in systems.
The impact of that cannot be underestimated. One can install an OS on mirrored SSDs without utilizing valuable 2.5″ drive bays.
For the mainstream storage, our test PowerEdge R640 had a PERC H740P with 8GB NV cache. This card sits on the motherboard instead of in a PCIe expansion slot and has the capacitor array onboard. One will also notice that the PERC is one of the few items that is screwed in that requires tools to change. We understand this since loose cables are a likely cause for server issues. The R640 even has custom cabling instead of industry standard SFF-8643 cables to ensure a reliable connection.
While the PERC will handle SAS3 connectivity, in modern servers NVMe is the hot storage technology. The PowerEdge R640 has cables that run from near the CPUs to the front of the chassis where the U.2 form factor 2.5″ drive bays are found. Many vendors use PCIe add-on cards from standard PCIe expansion slots to anchor cabling for NVMe drives. Dell EMC’s solution keeps the PCB traces minimized which helps ensure proper signaling for high-speed PCIe 3.0 communications.
If one were counting, they would see that the chassis could have 8x 2.5″ NVMe 2.5″ drives yet there is an 8-port PERC H740P as well. That brings us to the front storage backplane which is part magic. Here are the pieces where that magic happens:
Dell EMC is one of the few vendors, as of this writing, that has a working, hot-swap backplane that can be configured for either SAS3/ SATA or NVMe SSDs. That simplifies deployment of the PowerEdge R640 and is truly a cool feature. In a few years, we expect this type of feature to become more commonplace, but as of this writing, this is an extremely forward thinking capability.
The physical storage is, as you probably expect, tied into iDRAC 9. What that means is that one can see storage in the web management interface and that configuration can be passed to and from fleet management tools.
One can see the storage controllers and health of the PERC power backups from iDRAC.
One can also delve into disks. If you have ever experienced a need to upgrade firmware on drives and needed to find which disks in systems need the update, you will completely understand the power of this information.
As we mentioned earlier, BIOS settings changes via iDRAC 9 is a great feature. Unlike many of the low-cost servers available, one can configure RAID setups via the web UI. No need to go through posts spamming keyboard combinations, again this is here.
SSDs are extremely reliable, but they do infrequently fail. The storage management interface allows you to see provisioned global hot spares.
Dell EMC has a huge selection of drives intended for specific applications and has taken the steps necessary to make them easy to manage. More importantly, the PowerEdge R640 ensures that those valuable 2.5″ bays are used for high-value applications instead of as boot devices which it has solutions like the Dell BOSS to service efficiently.
Dell EMC PowerEdge R640 Networking
Our test system was equipped with a Mellanox ConnectX-4 Lx dual 25GbE adapter. With this generation, SFP28 25GbE is a cost-effective upgrade from 10Gb SFP+ offering effectively 2.5x the performance of the legacy standard.
We verified that the 25GbE adapter linked at both 25GbE and 10GbE speeds. For those who have legacy 10GbE networking, the 25GbE ports worked without issue even with SFP+ DACs and switches. We ran a quick test for throughput using iperf3 running out of the box on our network at 10GbE and 25GbE speeds:
With the Intel Xeon Scalable generation, one can achieve higher VM density and aggregate more storage I/O than with previous generations. Upgrading to 25GbE is a sensible choice. As one can imagine, while 25GbE is great, you can configure the PowerEdge R640 with a wide range of networking options depending on what you have deployed in your data centers.
Dell EMC PowerEdge R640 Power Consumption
Our Dell EMC PowerEdge R640 came equipped with dual 1100W power supplies. This is important for the dual Xeon Platinum 8180’s that can use over 700W combined under AVX-512 workloads.
Like just about everything in the Dell EMC PowerEdge R640, this data is fed back to iDRAC 9 for local monitoring and for aggregation to group management tools.
During AVX-512 workloads and the Intel Xeon Platinum 8180’s we hit over 700W consistently with the system. With the dual Intel Xeon Bronze 3106’s we stayed under 300W so there are a lot of variabilities here.
Dell EMC even computes the BTU/hr figures so one can aggregate that information and manage racks and data center cooling. That is an awesome figure.
We actually saw the input voltage for the rack drop to 204V from 208V and were able to pick it up and bring to the data center operator. This was picked up by our higher-end Schneider Electric APC PDUs as well, but if one did not have monitoring at the PDU level, this is a great feature.
Another cool tidbit, testing the Dell PowerEdge R640 for this review used over 1000kWh which was recorded by the iDRAC. At $0.30 per kWh of Silicon Valley California (expensive) data center power plus cooling overhead, we can directly attribute over $300 of electricity went into this review. If your operations team wants to get detailed in their power consumption tracking without swapping PDUs, the PowerEdge R640 can help.
Final Words
If you are looking for a highly customizable, rock solid, 1U server platform, the Dell EMC PowerEdge R640 is a great bet. Dell EMC’s commitment to the industry-leading remote management system using iDRAC 9 as its cornerstone is paying off in spades. Even the little details about the PowerEdge R640 are superb such as excellent rails which make servicing the machine that little bit easier. We noticed this as we swapped several sets of CPUs in the R640 and even that involved process was exceptionally easy due to the PowerEdge team’s attention to detail and refinement over generations. If you are set on the 1U form factor, the Dell EMC PowerEdge is a top choice.
Hi,
Using a Platinium CPU in a 2 CPU server just sounds stupid.
This CPU should be used in a 4 or 8 CPU Configuration to get everything you paid for ( including Intel® Ultra Path Interconnect – 3 UPI links ).
That CPU is for Servers like : https://www3.lenovo.com/us/en/data-center/servers/mission-critical/Lenovo-ThinkSystem-SR950/p/77XX7HSSR95
not for 2 CPU “webservers”…
regards,
Bang up review STH.
Gold 613X would be my sense at that sweet spot.
Would be helpful to see EPYC benchmarks in comparison.
iceman1 – these numbers are directly comparable to the volumes of EPYC numbers we have on the site.
Interesting review, iDRAC 9 looks like a decent step up.
Small error here:
Under – Dell EMC PowerEdge R640 Networking
Our test system was had Mellanox ConnectX
Hello, I have some of these servers but they were initially setup with a basic RAID 1 with 2 SAS SSD going through a PERC 740P.
To this day I still can’t figure out if and how U.2 NVMe drives are supported, while I don’t need them right now I check pricing from time to time and it seems we have reached a point where some U2 drives are both faster and cheaper than SAS ( and obviously SATA ) SSDs. The local Dell reps were pretty much clueless about U2 back in 2018/mid 2019 when I asked about that.
I can’t open/modify the R640s I’m using right now and have no U2 drive on hand either for testing, but in theory should I just be able to hot-swap one in the front and have it seen in idrac 9 ? And how do you handle RAID config etc if it doesn’t go through the PERC controller ?
Does board level component layout documentation exist for these systems? Does anyone know where to find it?
I’m troubleshooting an intermittent R640 “The system inlet temperature is greater than the upper warning threshold” problem, and I can’t even find where the temperature sensor is. I think it is somewhere in the front panel, or perhaps on the drive backplane. But where? There doesn’t seem to be any documentation to answer such questions.