Dirt Cheap Data Warehouse – Low Cost Database and Storage Servers
The Dirt Cheap Data Warehouse is a project to build low cost database and storage servers for development. Jeff Shukis both author on STH and active forum moderator recently detailed his very inexpensive data warehouse project. As a Silicon Valley start-up costs must remain in check while at the same time providing a lot of performance. The Dirt Cheap Data Warehouse is built using mostly recycled HP and Dell gear, including several 8 CPUs per 2U Dell C6100 XS23-TY3 (Intel) and the Dell C6145 (AMD) cloud servers. In the forum post, Jeff highlights the design philosophy of the Dirt Cheap Data Warehouse:
The secret to the Dirt Cheap Data Warehouse is this: Used generation-old server technology from eBay, plus a large number of consumer-grade SSD drives, carefully selected, configured, and deployed to avoid resource-wasting bottlenecks and to maximize query throughput and query throughput per dollar.
From the pictures, primary storage is 2.5″ consumer grade SSDs. The main storage uses a HP DL585 G7 quad Opteron 6000 series platform and 10 disk controllers (over 100 SAS channels) and handles over 16GB/s in Oracle database query throughput.
Further, both gigabit networking plus a 40gbps QDR Infiniband is in play. There are a total of 48 gigabit ports using a Dell switch. Further, a 40gbps QDR Infiniband network is handled by its own switch.
This storage and networking backend feeds a server farm. One can see that there are at least 16 compute nodes in the below picture, 12 dual CPU and 4 quad CPU.
One key to setting this all up on a budget is carefully selecting previously used parts. The STH Forums includes a Great Deals forum where users trade finds on ebay and elsewhere on great server hardware. Many of these components were purchased at a small fraction of the cost new due in large part to that forum.