SAS Expanders, Build Your Own JBOD DAS Enclosure and Save – Iteration 2 – A Better Solution

16
Posted May 27, 2010 by Patrick Kennedy in Servers
Supermicro X8SIL-F next to the HP-SAS Expander and PCMIG setup with both power supplies

After completing the first DAS/ SAS Expander JBOD enclosure project I realized that there was a major area of improvement. Using less than 30% of a large 4U case’s volume for useful purposes seemed like the key area to improve upon. As I was completing that build I soon realized that I wanted a secondary server to be able to access some of the drives for EXSi or Hyper-V virtual machines. Further, NAS operating systems that run poorly in virtual machines, such as unRaid require dedicated server for testing. I could have built another server in another enclosure, but I decided that I could improve upon the original design and access drives that are housed in the Big WHS ecosystem through a simple cable swap. This eliminates the need to physically move drives from enclosure to enclosure. The following is a slightly (approximately $20) more expensive version of the original Build Your Own JBOD DAS Enclosure with a HP SAS Expander iteration.

As one can see from the original build your own project, the low cost motherboard takes up lots of useful space, yet provides little value as it only powering the HP SAS Expander.

Install Cables to the HP SAS Expander and motherboard in the Norco RPC-4220 DAS/ SAS Expander Enclosure

DIY JBOD DAS Iteration 1 – the HP SAS Expander and motherboard in the Norco RPC-4220 DAS/ SAS Expander Enclosure

The Norco RPC-4220 and RPC-4020 can both house large ATX motherboards that use all expansion slots to the rear of the case. From the above shot I realized that there was room to fit a mATX based motherboard (or mini ITX of course) and also the HP SAS Expander.

Setting up the PCMIG and HP SAS Expander solution

After searching for solutions, I saw a post by odditory at [H]ard|Forum where he used a PICMG based board to power the expander. After ordering the PICMG 1U backplane with 1x PCIe slot (you can search for PE-2SD1-R10 on Google to find a local retailer. It is made by a company called IEI).  I waited… two weeks for the PICMG board to ship and another few days for it to get here using ground shipping. It seems like these parts are not regularly stocked everywhere. Certainly this is not as convenient as using a low cost mATX board.

PE-2SD1-R10-1 PCMIG 1U Backplane with PCIe Slot

PE-2SD1-R10-1 PICMG 1U Backplane with PCIe Slot

One large consideration with this solution is that the PICMG connectors (bottom of the board) are very large. Looking at the bottom edge of the above pictures lets one see the back-side of where the pin outs for those slots connect to the board. The Norco RPC-470 does not have a raised motherboard tray and the PICMG board fit no problem. In the RPC-4220 the PICMG connectors were too tall and therefore I could not seat the PICMG board in the case and still have the HP SAS Expander seat properly in the chassis expansion slot. The simple solution, 100% guaranteed to void warranties on the PICMG board, is to dremel off the PICMG 1.3 slots. There was quite a bit of room between the PICMG slots and the PCB so this was very simple.

Bottom of modified PE-2SD1-R10-1 PCMIG 1U Backplane with PCIe Slot

Bottom of modified PE-2SD1-R10-1 PCMIG 1U Backplane with PCIe Slot

With the PCMIG board modified, I was able to use bumpers and drill new PCB mount holes for the PCB. It should be noted that the board is VERY long (I haven’t measured it but it is likely over 30cm long just fitting between the mid-case fan mounting point and the rear of the Norco case.) Practically speaking, this means that using motherboard bumpers to keep the board well supported is important. Securing the installed HP SAS Expander to the chassis helps a lot with the setups stability.

Next, I used a jumper on the power button pins. This allows me to power on and off the enclosure using the primary (large) power supply.

Jumper red connector to cycle power via PSU

Jumper red connector to cycle power via PSU

This is a very important point. Without separate power supplies, a powered down secondary server in the chassis would cause the HP SAS Expander to lose power and attached drives would drop from raid arrays. Using a jumper on the power button header makes the on/ off switch the power supply switch at the rear of the case. If one only wanted to implement the PICMG solution as an alternative to the low cost motherboard (and not have a secondary server), then one might try connecting the power button normally.

Pictures of the full server setup

The big question at this point is, of course, what does this look like installed? Here are some pictures.

First, I installed the HP SAS Expander in the second to last expansion slot. The HP SAS Expander card has a passive heatsink as one can see from the below picture. If it is installed in the last expansion slot, airflow and cooling become issues.

Gap between PCMIG-HP SAS Expander and Norco RPC-4220 Case

Gap between PCMIG-HP SAS Expander and Norco RPC-4220 Case

Here is a shot of the other side of the installed PICMG-HP SAS Expander next to the Supermicro X8SIL-F.

Gap between X8SIL-F and PCMIG with cards installed

Gap between X8SIL-F and PCMIG with cards installed

Clearly this is a tight fit solution. Expansion slots 1 through 4 can be used on the mATX board, expansion slots 5 and 7 are unused, and expansion slot 6 has the HP SAS Expander. In the above picture there is an Adaptec 5805 installed in slot 1, Intel Pro/1000 PT Quad in slot 2, and an Intel Pro/1000 GT (PCI) installed in slot 4. One can also see the OCZ Rally USB drive on the internal Supermicro X8SIL-F USB header. The setup affords plenty of room to work on the bottom edge of mATX server boards which typically have many cable connections and jumpers.

Just to give an overall view of what the case looks like with the SAS Expander infrastructure installed alongside the independently powered CPU and motherboard here is a picture of the entire setup.

Supermicro X8SIL-F next to the HP-SAS Expander and PCMIG setup with both power supplies

Supermicro X8SIL-F next to the HP-SAS Expander and PCMIG setup with both power supplies

I routed the PicoPSU 150XT wire to the external power supply through an unused LAN port slot. Overall cable management is quite challenging because the 24 pin ATX power cable needs to cross the entire enclosure. Also, there are cables for 22 drives, fans, fan controllers, and two power supplies so spending a few minutes to tidy everything up is worth it once you have done a test fitting. Here’s a quick picture of the PicoPSU peaking through the unused LAN cutout:

PicoPSU Powering X8SIL-F Server Motherboard

PicoPSU Powering X8SIL-F Server Motherboard

Not an elaborate setup, however it does work because the PicoPSU takes up very little space.

I am sure some readers will be confused why I do not just connect the drives to the server board installed inside the same 4U. The simple reason is that the drives are being used by a server in a second chassis. The Supermicro X8SIL-F was only installed because I wanted to utilize otherwise wasted space. One could easily build one of these DAS enclosures only with a HP SAS Expander and PICMG board not bothering with the secondary mATX server.

Conclusion

This DIY solution is a significant improvement over the original design. I was able to reclaim otherwise underutilized space using a modified PICMG board. Since it does require modifying a more expensive (I paid just over $55 US) piece of PCB versus the original low-cost and off-the-shelf mATX power board solution (less than $35 US), this is not going to be for everyone. On the other hand, I would offer that if someone is using a second 4U chassis (and therefore over 20 drives) for DAS storage, there is a good chance they will have use for a secondary server.

Feel free to discuss this topic on the ServeTheHome.com forums!


About the Author

Patrick Kennedy

Patrick has been running ServeTheHome since 2009 and covers a wide variety of home and small business IT topics. For his day job, Patrick is a management consultant focused in the technology industry and has worked with numerous large hardware and storage vendors in the Silicon Valley. The goal of STH is simply to help users find some information about basic server building blocks. If you have any helpful information please feel free to post on the forums.

16 Comments


  1.  
    Matt

    Cool I was looking to do something similar!




  2.  
    Mitch W

    thx. will be trying this soon.




  3.  
    Josh

    Hey Patrick,

    I just found this…. http://usa.chenbro.com/corporatesite/products_detail.php?sku=75

    Would that work in place of your PCMIG device? Not sure how HP compares though.




  4.  

    It would. The negatives of the Chenbro SAS expander are a smaller RAID card compatibility list, no SAS 6.0gbps support, lower availability in retail channels (some say it may be discontinued), and finally many users have issues with drives dropping/ drive compatibility.

    The HP SAS Expander is a solution that fixes most of these issues. My guess is Chenbro will come out with a new expander line soon, but until they do, the HP SAS expander is one of the better options out there.

    Finally, if you want to look at alternative expanders, both Areca and Supermicro have SAS Expanders that are pretty solid. For some the Chenbro is either very good, or very bad which is why I am not using them. Do homework before using the Chenbro expanders and make sure that you have a supported raid controller and drives.




  5.  
    Josh

    Patrick,

    One more quick question… you seem to be good with the mods, I’m looking at a Chenbro case (I hear they are a bit higher quality) and I just spoke with a tech over there and he said the 5 80mm fans are 3pin power and I’m wondering if you have any idea what I’d use to control all of these fans on 1 controller? I’m sure there are some sort of POTs out there that I could use for this purpose, just not sure how hard it is going to be.

    On a side note, I like the 3x 120mm fan mod that cavediver (from [h]) is selling. Maybe a Norco would be better for noise, I just wish the Norco supported SGPIO for failed drive identification purposes. Can you comment on this feature? Have you had a drive fail and if so, is it hard to find it?




  6.  

    On the fan controller, you can actually see one fan controller I use on the pic above at: “Bottom of modified PE-2SD1-R10-1 PCMIG 1U Backplane with PCIe Slot”. It works OK since in reality I set between 70%-75% and close the case. It isn’t the most elegant solution but for $12 and 3 minutes I won’t complain.

    I actually had a 2TB drive fail last night in the RPC-4020. I just logged into the Areca raid console, figured out the port/ S/N of the drive (to confirm) and then just pulled the drive. If you want, you can just label each tray with the S/N of the drive or the controller port, and that lets you know which one failed. It takes a few minutes but is not a big deal.




  7.  
    Thomas Vranas

    Great infomration on this and you other builds.
    I just got my chenbro expander on Friday. Hooked everything up and …. flames came out of the SAS card!
    Yes flames, not just smoke. Because I did a special order from CDW now I have to deal with Chenbro to get it replaced. They did respond Friday evening with RMA form which I sent back. Now I am just waiting for my response.
    Good thing the rest of my equipment was not damaged.




  8.  

    Yikes! Good luck with the RMA. Was it one of the older 3.0gbps expanders?




  9.  
    Stefan

    Maybe this is interesting an all-in-one kit 24-48 port (250-500 USD) from Chenbro:
    http://usa.chenbro.com/corporatesite/products_detail.php?sku=76




    •  

      Stefan: That is very similar to the HP SAS Expander which can be had for a bit less. A big driver is the SAS card you are using to connect the expanders. Some work well with the HP units some work well with the Chenbro making one or the other a better option for an application based on the controller used.




  10.  
    Rob

    IBM 59Y3825

    Does anyone know anything about this.

    It appears to do the same as the HP Expander but claims 6GB/s support




  11.  
    Paul

    Are you installing an OS on this machine? And if so, which one?




  12.  
    clax

    Hello everyone, I would like to know how you solved the lighting of Norco cabinet. Are present on the PE-2SD1-R10-1 PCMIG 1U Power Switch Connector ?




  13.  
    Jeff Shukis

    Great article – hopefully very usefor for the many many people who are building their own budget storage servers.

    One comment: The article mentions “PCMIG” many times. It’s actually “PICMG”, which stands for PCI Industrial Computer Manufacturers Group (PICMG). I make this comment only because I want folks who search for “PICMG” to find your article.




  14.  
    dan

    Hey Gang,

    way late to the party but this is a great project. I have a possibly stupid question but cant seem to find an answer anywhere.

    I love that you made a secondary mobo for the Expander. Is it possible to plug the expander into a mobo, use the expander for an array accessed by a different computer and then use the mobo as an independent computer? or would that cause all sorts of issues?

    Thanks, and great build!





Leave a Response

(required)


Newly Reviewed
 
  • WD Red and RE 4TB Thermal Imaging
  • 9.0
    iStarUSA BPN-124K-SA front
  • Western Digital Red 5TB 6TB Launch
  • Amazon T2 Instances and Intel C2550 time comparison
  • 9.6
    ASUS P9A-I-C2550-SAS-4L-Overview
  • 9.3
    Supermicro X9SKV-1125 Overview