Now Reading
Gigabyte R270-T61 2U Server Review – 96 core Cavium ThunderX
1
Review

Gigabyte R270-T61 2U Server Review – 96 core Cavium ThunderX

by August 2, 2016
Positives

Dual Cavium ThunderX 48 core CPUs (96 cores total). 24x 2.5" hot-swap SATA hard drive bays. 160Gbps (3x 40GbE, 4x 10GbE) networking all onboard. Standard web-based remote management with iKVM and Serial-over-LAN capabilities.

Negatives

Higher idle power consumption. Tool-less installation rails would increase installation speed. Non-redundant mid-plane fans.

Rating
Our Rating
Design & Aesthetics
9.0
Performance
9.5
Feature Set
9.0
Value For The Application
9.0
Bottom Line

If you are looking to deploy ARMv8 servers to a data center for development or production, these are the servers that will integrate well into existing 19" rackmount infrastructure.

9.1
Our Rating
You have rated this

Upon receiving the Gigabyte R270-T61 you may think it is an ordinary 2U server. That is exactly the point. With 24x 2.5″ hot swap bays adorning the front of the chassis and redundant power supplies in the rear, the server is meant to integrate into existing racks with little fanfare. What powers the server are two 48-core Cavium ThunderX chips. There are no HBAs in the system, no add-in NICs, just a genuine ARMv8 64-bit performance machine. If you are looking for a 64-bit ARM server today, this is the one you want.

Test Configuration

Cavium set us up with a test server that is very solid.

  • Server: Gigabyte R270-T61
  • CPU: 2x Cavium ThunderX 48 core CPUs onboard (96 cores total)
  • RAM: 256GB (8x 32GB) DDR4 RDIMM
  • SSD: 2x Samsung PM863 960GB
  • OS: Ubuntu 16.04 LTS

We did change the SSD from Cavium’s original configuration to Samsung PM863’s that we use in the lab. Notably, even with 24x 2.5″ bays in the front of the chassis, all available there is no need for add-on HBAs. Likewise, while there is a ton of network connectivity, there are no add-on network controllers. This leaves the PCIe slots free for additional expansion.

Gigabyte R270-T61 Overview

The front of the Gigabyte R270-T61 is a standard 24-bay in a 19″ 2U chassis. These form factors have become extremely popular since 2.5″ SSDs are easy to populate. Just by looking at the server, there is very little indication that anything is different than an industry standard server.

Gigabyte R270-T61 front view

Gigabyte R270-T61 front view

Inside the server, one can see Gigabyte’s motherboard design experience. Even the color scheme is what we would expect from Gigabyte branded motherboards.

Gigabyte Cavium 2P 2U

Gigabyte Cavium 2P 2U

There are 8 DDR4 DIMM slots per CPU for 16 DDR4 DIMM slots total. This allows for up to 2TB of RAM to be added to the system. Moving from single DIMM per channel (four per CPU) to two DIMMs per channel (eight per CPU) will drop the DDR4 RAM speeds from 2133MHz to 1866MHz.

Cooling the full 48-core Cavium ThunderX is a bit challenging in a 1U form factor (as with any similar TDP chip) but with the 2U form factor, cooling is much easier. The fan array is relatively simple with four 80mm fans that allow cabling to occur on either side of the fan partition. These fans have ducts that channel air over the 2U passive heatpipe coolers.

Gigabyte R270-T61 2U dual Cavium ThunderX platform open

Gigabyte R270-T61 2U dual Cavium ThunderX platform open

With dual Cavium ThunderX 48-core ARM SoCs one sees the clear platform advantage: integration. While this server has a combined 160Gbps of network ports, there are no PCIe add-in controllers found on the system. Likewise, controlling 24x SATA SSDs would normally require the addition of an add-on HBA such as the LSI SAS 9305-16i in addition to onboard motherboard ports. Instead we see six of the eight SFF-8087 ports being used, each cabling four drive bays. Between the networking and storage, these are the key value points of the Gigabyte-Cavium platform.

Gigabyte R270-T61 2U PCIe and SFF-8087

Gigabyte R270-T61 2U PCIe and SFF-8087

The PCIe 3.0 slots are actually PCIe 3.0 x8 in a PCIe x16 physical slot, two PCIe 3.0 x8 slots and one non-functional PCIe 3.0 x8 slot. We did get an e-mail from a reader who purchased one of these machines after our Cavium ThunderX Benchmarks Part II: Why enterprise ARM developers need these machines piece. The user was trying to figure out why four PCIe cards would not work in the system, however this is clearly listed on the Gigabyte spec page.

Moving to the rear of the server, one can see significant networking capabilities. The throughput of the machine can be absolutely awesome using the integrated networking. There are four SFP+ 10GbE ports along with three QSFP+ 40GbE ports. That is a total of 160Gbps in aggregate. The sole 1GbE port is the BMC management port.

Gigabyte R270-T61 rear IO being setup

Gigabyte R270-T61 rear IO being setup

 

The system has redundant 750w power supplies. This is a major upgrade to our 1U test system which featured a single PSU design.

Gigabyte R270-T61 Remote Management

A great feature with the Gigabyte R270-T61 remote management is the remote management interface. We did notice a maturation in the remote management interface from the single processor system we first tested months ago to the updated dual processor system. It is nice to see that Gigabyte is adding additional functionality with newer BMC firmware releases. Here is a quick view of the R270-T61’s IPMI management interface. We added the menu bar from the original Gigabyte R120-T30 1U server‘s interface as an easy visual indicator of areas added:

Gigabyte R270-T61 - Increased management functionality

Gigabyte R270-T61 – Increased management functionality

These features extend to standard functionality such as providing iKVM access with remote media mounting. During development cycles, having the ability to boot to an ISO image is handy.

Gigabyte R270-T61 iKVM functionality with BIOS setup

Gigabyte R270-T61 iKVM functionality with BIOS setup

For those looking at these systems for network appliances, the out of band management ports can also be used for Serial-over-LAN functionality:

Gigabyte R270-T61 ipmitool

Gigabyte R270-T61 ipmitool

Overall, these systems are much more advanced than the average ARM development motherboard. One can easily deploy them to a data center and they will function similarly to mainstream enterprise servers.

Gigabyte R270-T61 Networking Performance

With 160Gbps of networking promised on the system, we did want to update test figures from our Cavium ThunderX Benchmarks Part II piece with the dual processor model.

Gigabyte R270-T61 network speed

Gigabyte R270-T61 network speed

As one can see, there is a lot of potential networking bandwidth in this platform. We have recommended several companies that build networking appliances to get these servers for development as the potential bandwidth is excellent.

Power Consumption

We did take an opportunity to measure power consumption of the system in our lab using our calibrated APC metered-by-outlet PDUs. We test data center power consumption on our 208V 30A power circuits.

  • BMC Only: 6.1
  • Ubuntu 16.04 Idle: 319W
  • Maximum Recorded: 374W

Cavium ThunderX power consumption at OS idle is higher than one may expect from an Intel system. Cavium is utilizing OS power management while Intel relies heavily on hardware power management. Our sense is that with future software updates, we may see the idle power consumption decline on the ThunderX platform.

Final Words

One can certainly see the Cavium ThunderX ecosystem maturing. The dual processor server had a number of updates to the BMC firmware that bring additional features to the platform. You can see our 96 core micro-benchmarks for some perspective on compute performance. Overall, we believe the Gigabyte R270-T61 with dual Cavium ThunderX 48 core processors (96 cores total) to be the best arm64 development platform available to date for anyone looking to deploy into data centers. There are still a few areas we would like to see improved. One area is the rails. Most servers in this price range have tool-less rail systems. While they do add cost to the overall BOM, they greatly reduce installation time. This is doubly so if you are installing these machines into a mid-aisle rack. We also would like to see a redundant mid-plane fan array added. Fans are generally reliable but we prefer higher levels of redundancy with fans either through ducting or through in-line redundant fans. Finally, the pricing and availability of these servers has improved to the point that they can be ordered fairly easily. We have received feedback that they were hard to find a few months ago. If you are having issues finding them, feel free to send me a note at patrick at this domain and I will be happy to send you to someone who can help.

About The Author
Patrick Kennedy
Patrick has been running ServeTheHome since 2009 and covers a wide variety of home and small business IT topics. For his day job, Patrick is a management consultant focused in the technology industry and has worked with numerous large hardware and storage vendors in the Silicon Valley. The goal of STH is simply to help users find some information about basic server building blocks. If you have any helpful information please feel free to post on the forums.
1 Comments
  • Nils
    October 22, 2016 at 11:06 pm

    The problem with these is of course since they aren’t what I would call commodity hardware information on price is hard to come by.

Leave a Response