STH Colocation – May 2014 Update

STH Colocation - May 2014 Update
STH Colocation - May 2014 Update

After a quick hop out to Las Vegas, Nevada the STH colocation architecture has received a significant upgrade in May 2014. We originally posted our plans almost two months ago. A few iterations later and everything is now up and running. In total the node count has gone from 6 to 9 while only adding 1 amp to the setup (running right now at 6amps with most of the architecture idle.) This is a very impressive result as there have been major upgrades in terms of potential CPU power, drive speed and capacity, memory capacity and even interconnect capabilities. In keeping with tradition we take a look at some of the hardware that went into the colocation facility.

For those that did not read the forum thread, we got a great deal on a Supermicro Twin SYS-6027TR-D71FRF which is a monstrous system. It has 2x Intel Xeon E5 dual socket systems in a single 2U enclosure. Both systems are fed by a pair of redundant power supplies. Each node comes standard with a LSI SAS 2108 RAID controller and Mellanox ConnectX-3 56gbps FDR Inifiniband/ 40GbE onboard. Compared to the previous Dell PowerEdge C6100′s these are significantly more powerful. Here is how the internals are configured. 

Supermicro SYS-6027TR-D71FRF Initial Test
Supermicro SYS-6027TR-D71FRF Initial Test

We did an initial test of the storage and networking subsystems with only one processor. The next step was adding RAM. This was somewhat tricky as there are only 4 DIMM slots per CPU for a total of 8 slots per node and 16 per chassis. Using dual Intel Xeon E5-2665′s in each node and filling each DIMM slot with 16GB Samsung ECC RDIMMs yielded 16 cores, 32 threads and 128GB of RAM per node.

Supermicro SYS-6027TR-D71FRF CPU and Memory
Supermicro SYS-6027TR-D71FRF CPU and Memory

One other excellent feature of the Supermciro SYS-6027TR-D71FRF is that it has three PCIe expansion slots to go along with its onboard Infiniband and SAS RAID controllers. Two are full height and one is half height but there is enough room for many types of cards.

Supermicro SYS-6027TR-D71FRF Motherboard Tray
Supermicro SYS-6027TR-D71FRF Motherboard Tray

In terms of drives we ended up using a stack of Crucial M500′s. With lots of RAM in the colocation cabinet there is simply not much disk writing going on for a website like STH. Each of these Supermicro nodes got 2x Seagate 600 Pro 240GB “OS” drives in RAID 1, 2x Crucial M500 480GB “storage” drives in RAID 1 and 2x 4TB Hitachi CoolSpin 4TB drives in RAID 1 simply to act as backup targets. Crucial M500 240GB drives also found their way into the pfsense nodes.

Crucial M500 480GB SSD for the May 2014 Colo
Crucial M500 480GB SSD for the May 2014 Colo
We used the Supermicro A1SAi-2750F and A1SRi-2758F inside Supermicro CSE-505-203B cases. These were in many ways similar to the all-in-one units we discussed recently. pfsense 2.2 is still in Alpha and pfsense 2.1.3 is still best run bare metal. We initially tested a configuration with 10GbE NICs but then decided to simplify that configuration significantly when we swapped to bare metal pfsense.
Supermicro SC505-203B Rangeley 10GbE SSD HDD Fans
Supermicro SC505-203B Rangeley 10GbE SSD HDD Fans

Colocation cabinet redo

We started with the colocation cabinet just the way we left it after CES 2014 earlier this year. There were a total of four compute nodes and two pfsense nodes. A lot of work had to be done.

STH Colo Update Jan 2014

The basic steps we followed were:
  1. Prep everything ahead of time, including backing up and restoring the pfsense configuration on the new Atom C2758 based pfsense node
  2. Head to Las Vegas and the colocation facility
  3. Remove spare Dell C6100 chassis from the rack + rails for it in the middle of the cabinet
  4. Install the pfsense nodes and verify they were working
  5. Cutover to the new pfsense nodes
  6. Remove the top active Dell C6100
  7. Install the Supermicro 2U Twin with one node installed
  8. While allowing the new 2U Twin nodes to boot, replace two L5520′s in Dell C6100 nodes with Xeon X5650′s and add RAM
  9. Consolidate all Dell C6100 compute nodes to the bottom chassis
  10. Install the Atom based monitoring node in the front of the rack (not pictured)
  11. Pack up all of the remaining parts
  12. Head back to the airport
That left us with the following:
STH Colocation - May 2014 Update
STH Colocation – May 2014 Update
Please excuse the messy wiring, this picture was taken before we did cable management.
The major “drama” was that at first the right node (pictured) in the Supermicro Twin had fan speeds that greatly exceeded that of the left node. A quick look at IPMI confirmed one CPU was still “OK” temp wise but was 20 degrees C hotter than the other CPU. On the other node the differential was under 2 degrees. We pulled the node out, re-seated the heatsink and processor and the differential was down to 2 degrees, just like the other node. Fan speed and power consumption likewise went down so the cabinet now sits at 6A.
Total downtime was under 20 seconds and total time at the facility was 2 hours 43 minutes. Surprisingly close to my 190 minute guess.

The impacts were that we increased density significantly. We replaced 24 Nahelem EP cores (L5520) with 24 Westmere-EP cores (X5650 and L5639) while keeping the same 216GB of RAM total we had in the two Dell C6100 chassis earlier. We added 32x Xeon E5-2665 cores and 256GB of RAM, over 4TB of SSD and 8TB of disk capacity and got a fast Mellanox FDR Hyper-V replication link between the two main Hyper-V nodes:

Supermicro SYS-6027TR-D71FRF Right Node
Supermicro SYS-6027TR-D71FRF Right Node
We also added 16x Silvermont generation cores in the two pfsense nodes each with 8GB of RAM and 240GB SSDs and two older generation Intel Atom S1260 cores.
Overall, absolutely enormous changes. 3 added nodes total and 276GB of memory, many TB more storage, faster networking and all while only have a single amp impact on overall power consumption while mostly idle running the site.
Here are the configurations for those interested:
[tabgroup][tab title=”Summary of Configurations”]

Supermicro Twin SYS-6027TR-D71FRF with dual compute nodes:

  • 2x Intel Xeon E5-2665 – 2.4GHz -> 3.1GHz Turbo 20MB L3 and 8C/ 16 thread
  • 128GB DDR3 (8x 16GB Samsung per node)
  • 2x Seagate 600 Pro 240GB
  • 2x Crucial M500 480GB
  • 2x 4TB Hitachi CoolSpin drives (backup)
  • 1x gigabit port to each switch
  • 1x FDR infiniband port between the two nodes for replication

New pfsense node 1:

  • Rangeley C2758
  • 8GB ECC memory
  • Crucial M500 240GB
New pfsense node 2:
  • Avoton C2750
  • 8GB ECC memory
  • Crucial M500 240GB
Dell C6100 configuration
  • 2x dual L5520 nodes with 48GB RAM, Intel 320 160GB SSD, Kingston E100 400GB
  • 1x dual L5639 node with 48GB of RAM and Intel 320 160GB SSD, 2x WD Red 3TB
  • 1x dual X5650 node with 72GB of ram and Intel 320 160GB SSD, 2x Crucial M500 480GB

The mini node (just in case/ for monitoring)

  • Supermicro X9SBAA-F
  • Intel Atom S1260
  • 4GB RAM


Please enter your comment!
Please enter your name here

This site uses Akismet to reduce spam. Learn how your comment data is processed.