Dirt Cheap Data Warehouse – Low Cost Database and Storage Servers

1
Posted April 24, 2013 by Patrick Kennedy in Client Tips
Dirt Cheap Data Warehouse Processing

The Dirt Cheap Data Warehouse is a project to build low cost database and storage servers for development. Jeff Shukis both author on STH and active forum moderator recently detailed his very inexpensive data warehouse project. As a Silicon Valley start-up costs must remain in check while at the same time providing a lot of performance. The Dirt Cheap Data Warehouse is built using mostly recycled HP and Dell gear, including several 8 CPUs per 2U Dell C6100 XS23-TY3 (Intel) and the Dell C6145 (AMD) cloud servers. In the forum post, Jeff highlights the design philosophy of the Dirt Cheap Data Warehouse:

The secret to the Dirt Cheap Data Warehouse is this: Used generation-old server technology from eBay, plus a large number of consumer-grade SSD drives, carefully selected, configured, and deployed to avoid resource-wasting bottlenecks and to maximize query throughput and query throughput per dollar.

From the pictures, primary storage is 2.5″ consumer grade SSDs. The main storage uses a HP DL585 G7 quad Opteron 6000 series platform and 10 disk controllers (over 100 SAS channels) and handles over 16GB/s in Oracle database query throughput.

Dirt Cheap Data Warehouse

Dirt Cheap Data Warehouse

Further, both gigabit networking plus a 40gbps QDR Infiniband is in play. There are a total of 48 gigabit ports using a Dell switch. Further, a 40gbps QDR Infiniband network is handled by its own switch.

Dirt Cheap Data Warehouse Networking

Dirt Cheap Data Warehouse Networking

This storage and networking backend feeds a server farm. One can see that there are at least 16 compute nodes in the below picture, 12 dual CPU and 4 quad CPU.

Dirt Cheap Data Warehouse Processing

Dirt Cheap Data Warehouse Dell Cloud Servers

One key to setting this all up on a budget is carefully selecting previously used parts. The STH Forums includes a Great Deals forum where users trade finds on ebay and elsewhere on great server hardware. Many of these components were purchased at a small fraction of the cost new due in large part to that forum.

Bottom line, this is an awesome project that one needs to check out. Feel free to submit your own home labs in the forum and they may get featured on the main site like this and the 50TB VMware lab.


,

About the Author

Patrick Kennedy

Patrick has been running ServeTheHome since 2009 and covers a wide variety of home and small business IT topics. For his day job, Patrick is a management consultant focused in the technology industry and has worked with numerous large hardware and storage vendors in the Silicon Valley. The goal of STH is simply to help users find some information about basic server building blocks. If you have any helpful information please feel free to post on the forums.

One Comment


  1.  
    Lost-Benji

    Now I am not the kind of person who is into art, can’t justify artists…..

    But, that little rack holds some beautiful art-work.





Leave a Response

(required)


Newly Reviewed
 
  • Intel 730 SSD
  • Intel DC P3700 400GB Internal View
  • AppliedMicro X-Gene
  • Ubuntu RAID 1 - Step 18 finish partitioning and write changes
  • XeonPhiDie_03
  • Mellanox ConnectX-4