External SAS/ SATA Disk Chassis Wiring – Part 2

10
Posted December 12, 2012 by Jeff Shukis in Client Tips
SFF-8087 Port on Disk Backplane

In External SAS/SATA Disk Chassis Wiring – Part 1 we looked at wiring a server with the cables and adapters needed to connect to an external SAS/SATA disk chassis. Now we can move on to wiring up the JBOD disk chassis itself. This article is for those who plan to assemble their own disk chassis, JBOD or RAID, or those who just want to better understand the cabling involved.

Our external SAS/SATA disk chassis starts with disks. The disks themselves are not the subject of the article, so we’ll gloss over your options: 2.5″ versus 3.5″, SAS or SATA, single or dual ported. For your home storage server you’ll probably select 3.5″ SATA drives. 2.5″ drives and both single and dual ported SAS drives are much more expensive. Let’s assume that you have done so unless otherwise noted. Whatever disks you select will be mounted into drive sleds and inserted into hot-swap backplanes. SAS/SATA backplanes are preferable, but you might find a good deal on SATA backplanes so we’ll talk about those as well.

Wiring for SAS Backplanes

If your backplane is a standard non-expander SAS backplane then it will have one SFF-8087 port for every four disks. We’ll talk about other backplanes a bit later in this article. We remember SFF-8087 ports and cables from Part 1 of the article and the SAS/ SATA cable guide. As a reminder, the photo below is of one of six SFF-8087 ports on the back of a Supermicro 24-bay backplane. The port shown has a SFF-8087 cable already attached. A second SFF-8087 connector is visible at the right of the photo, partially hidden under the large blue SFF-8087 cable.

SAS/SATA Backplane with SFF08087 Port

SAS/SATA Backplane with SFF08087 Port

We need to convert this SFF-8087 backplane port to a SFF-8088 female port at the rear of the disk chassis. As with our server in External SAS/SATA Disk Chassis Wiring – Part 1, we accomplish this with a PCI SAS adapter bracket and a SFF-8087 cable.  The adapter bracket mounts in the back of the disk chassis in a PCI opening. Since we have done exactly this in our server already, we can skip the details and show the results. The photo below shows a 24-bay external disk chassis with six SFF-8087 ports (one port per four drives of course.) It is wired using three Supermicro pre-wired dual-port PCI SAS adapter brackets model CBL-0168L. The small green board is the Supermicro power board from the earlier article on that component. This chassis is ready to connect to a server with six SFF-8088 ports of its own – or to connect to up to six different servers, since each disk port is an entirely separate connection to a group of four drives. A database server in my environment uses three dual-port LSI SAS cards to connect to this chassis. Running six SFF-8088 cables from a server to a disk chassis is a lot of wires, but the resulting system provides real-world throughput of over 8 Gigabytes per second using 24 SSD drives. Theoretical bandwidth for this setup is an enormous 144 Gigabits per second.

Fully Wired 6-port SFF-8088 Disk Chassis

Fully Wired 6-port SFF-8088 Disk Chassis

At the rear of the chassis you can see the SFF-8088 ports. The photo below is from a similar chassis, but one with eight ports instead of six.

Rear View of Disk Chassis with SFF-8088 Ports

Rear View of Disk Chassis with SFF-8088 Ports

Wiring for SATA Disk Backplanes

If your JBOD disk backplane is a SATA backplane instead, or if you are using SATA drives without a backplane at all, then the wiring is just a little bit different. In that scenario we need to convert your SATA disk connectors to SFF-8088. We use a PCI SFF-8088 to SFF-8087 adapter bracket as before, but instead of a SFF-8087 cable, we use a SFF-8087 to SATA breakout cable, also known as a SAS to SATA “Forward” breakout cable. Sometimes these connectors are called 7-pin SATA or SAS connectors. One such cable is shown below, attached to a PCI SAS adapter bracket. You’ll also see “Reverse” breakout cables sold. These are used to convert four motherboard SATA ports to a SFF-8087 connector – something I advised you not do to in Part 1 due to cable length and signal strength limitations. Other than the use of the forward breakout cables, the wiring is identical to our earlier chassis.

SAS to SATA Forward Breakout Cable and PCI Adapter

SAS to SATA Forward Breakout Cable and PCI Adapter

Basic Wiring for SAS Expanders

The external disk chassis above, with multiple SFF-8088 ports, uses the least complex wiring setup and is perfect if you are using SSD drives and need maximum bandwidth from your disk chassis. Surprisingly, it can also be the lowest-cost solution for more general-purpose use if you have an inexpensive way of outfitting your server with multiple HBAs. Often, however, a different solution is desired – a chassis-mounted expander. SAS expanders allow you to connect many more than four drives to each SFF-8088 port on your server. Expanders add another tier to the wiring setup, and we’ll explore that now.

A SAS expander mounts within your external disk chassis and connects to your server with one or two SFF-8088 ports and then connects to multiple disk backplanes with other SFF-8087 ports. The most commonly available expanders come in two physical formats: PCI and what we’ll call “Rectangular” for the lack of a better word. The PCI form expanders are designed to mount in PCI slot. While they can draw power from the slot, they do not send any data at all via that port. In other words, this format of SAS expander cards look like SAS adapter cards, but they aren’t. The photo below shows a Chenbro SAS expander. HP offers a similar PCI-based SAS expander with a single external port. Patrick used this HP SAS Expander in his early DIY SAS Expander chassis articles years ago.

Chenbro PCI SAS Expander

Chenbro PCI SAS Expander

A PCI format SAS expander like the one above simply replaces the PCI adapter bracket in our earlier wiring architecture. We mount the expander in a PCI slot within our disks chassis and connect the SFF-8087 cables from our disk backplanes to the SFF-8087 ports on the expander. The result looks like the diagram below.

Basic SAS Expander Wiring

Basic SAS Expander Wiring

The other form of SAS expander is the “Rectangular” form like the Intel expander below. These are designed to mount entirely within a server chassis and use SFF-8087 connectors for all connections.

Intel RES2CV360 Rectangular SAS Expander

Intel RES2CV360 Rectangular SAS Expander

In our setup we’ll need to convert one of these SFF-8087 ports to a SFF-8088 connector for connection to our server. We’ll do this with the same kind of PCI SAS adapter bracket as before. Interestingly, Intel also sells a SAS expander that uses the PCI format and yet does not have external SFF-8088 ports. We’ll treat this like the rectangular format, using a PCI adapter bracket to convert one of the ports to SFF-8088. The image below shows the wiring for the rectangular type of SAS expander.

Internal Style SAS Expander

Internal Style SAS Expander

More Speed? Better Fault Tolerance?

The SAS expander setups above feature very simple single-cable wiring between server and disk chassis, and with the appropriate RAID card in the server they allow us to create a RAID volumes containing up to all of the disks in our disk chassis. Daisy chaining, a key feature of SAS expanders, won’t be discussed in this article, but enables even larger numbers of disks per RAID or HBA card. The downside of this simple expander wiring is that we have limited bandwidth – around 2GB/Second with current generation hardware – and several possible “single points of failure”. We can address the bandwidth limitations by using a feature of some SAS expanders sometimes known as x8 wide porting – combining two x4 ports. With a compatible RAID card and SAS expander, you can have dual SFF-8088 connections between the server and disk chassis, doubling the theoretical bandwidth. It is also possible, with the right hardware and the right operating system drivers, to achieve both increased bandwidth and failure tolerance by using wide porting and multi pathing. Most such setups also utilize dual-ported SAS drives for additional speed and redundancy. The details of such a setup are outside of our scope, except that I’ll show one possible wiring diagram for reference. See below.

Fully Redundant Expander Wiring

Fully Redundant SAS Expander Wiring

Other Notes

This article covers the most common disk chassis cabling issues, but is not comprehensive. Fire off a message if you have a corner-case and would like an opinion from me and/or the talented STH community. Since someone will probably ask, I will share these additional tips now:

  • Some JBOD disk chassis lack PCI bracket openings and have smallish rectangular openings instead. Look for Centronics SAS adapter brackets for these instead of PCI SAS adapter brackets – the guts are the same but the brackets are shaped differently.
  • As in Part 1, we are purposely ignoring the new and still rare new style SAS connectors for the sake of simplicity. Wherever you see “SFF-8088″ in this article, you can read it as “SFF-8088 or SFF-8644″. You can also translate “SFF-8087″ to “SFF-8087 or SFF-8643″. The new connectors are smaller than the old ones, which is good news for high density servers.
  • SAS expander cabling is quite simple, but SAS expanders quickly get complex when you factor in compatibility, performance, and features beyond the basics. Do your homework before buying an expander.
  • You do not need to mount a PCI SAS expander in a PCI port. These cards include a Molex port that can also be used to power them. The HP SAS Expander does not have a 4-pin Molex power connector. Instead it must be placed in a powered PCIe slot. Patrick showed an option in his JBOD SAS Expander chassis v2.
  • SAS expander chips are sometimes built in to the disk backplane. Many Supermicro chassis are available with either standard SAS backplanes or SAS expander backplanes, and you can replace one with the other if your budget allows. Wiring an expander backplane is easy – SFF-8087 cable from the backplane to yet another SFF-8087 to SFF-8088 PCI adapter bracket.

About the Author

Jeff Shukis

Jeff is a "Software Guy" who seems to like Hardware quite a lot. Formerly VP Engineering at Oracle, formerly VP/CTO for Bridgestream (acquired by Oracle), and formerly VP Engineering and SaaS Operations for Promptu, Jeff is currently launching a new software company, OpenIDA, focused on analytics and analytical applications for IT Security and Identity Management. Jeff sometimes posts as "dba" on the ServeTheHome forums.

10 Comments


  1.  
    Teg Bains

    Thanks for the great SAS wiring and Expander article.




  2.  
    MaxAMil

    Another informative one Jeff.

    Have any SAS expander recommendations? Is there a cable length restriction using SAS or SATA with these?




  3.  
    dba

    MaxAMil,

    For SAS, total cable length is limited to 10 meters. For SATA it’s one meter. If your RAID card/HBA is SAS but your disks are SATA, you get to use the whole 10 meters – or so I have heard. I’ve never needed more than 3 meters in such a scenario and it worked without any problems.




  4.  
    Victor

    Jeff,

    Thanks for the great articles. I read your 3 and Patricks 2. These articles gave me a lot of ideas. The info on the bandwidth really made me think. Very good writing, great pictures and illustrations. Did I say great job.

    Thank you,




  5.  
    Ben

    Thank you. Great article and very helpful as I’m beginning to work with SAS backplane system at home.




  6.  
    M2

    Great articles,

    But SAS seems to be directional, are there any SFF-8087 to SFF-8088 cables? Do these even exist? I can only find SFF-8088 to SFF-8087.

    Matt




  7.  
    dba

    Matt,

    The good news is that an SFF-8088 to SFF-8087 cable is the same as an SFF-8087 to SFF-8088 cable. I’ve done it.




  8.  
    M2

    Hey Dba,

    I guess I’ll just have to give it a shot, but I’m just a little weary of dropping $40+ on a decent length of SAS cable if in the description it says its directional.




  9.  
    TheNumber

    Hello,

    Great job,
    A configuration with 2 server chassis attached to one single disks chasis. will provide a better availability.

    http://i748.photobucket.com/albums/xx124/The_Number/Fully-Redundant_zps00962f0b.jpg

    What kind of operating system can handle this kind of dual pathing ?
    The hard drives and the controllers in the disk chassis will they support fail over from one node to another ?

    Thank you,

    TheNumber




  10.  
    Ray

    Thanks Jeff, best guide on the web no kidding. Newbie to this subject. Using M1015 or M5015 with RES2CV360 can I just use fan out cables directly to disk from expander instead of using a back plane board? Also had some issues getting the expander card not being recognized when attached to M1015. I have not explored the configuration interfaces on the raid card but it does recognize JOBD attached to raid card after card BIOS. Again great article. And thank you.





Leave a Response

(required)


Newly Reviewed
 
  • Purch Anandtech
  • Seagate 1200 v Toshiba PX02SMF040 400GB ATTO Write Benchmark
  • Intel S3700 and 710
  • pfSense Hyper-V 2.2-RC
  • Toshiba PX02SMF080 800GB
  • Amazon AWS Logo2