This AIC SB407-TU 60-bay Storage Server Houses over 1PB of Capacity


AIC SB407-TU Topology

With all of the slots and connectors, it is important to see how the dual-socket server is connected. Here is that view:

SB407 TU_System Block Diagram
AIC SB407 TU Block Diagram

CPU0 controls the OCP NIC 3.0 slot and two PCIe Gen4 x16 slots along with the Lewisburg PCH. On the other hand, CPU0 has a Gen4 x16 slot and two x8 slots but handles 6 of the eight rear SSD bays.

To be transparent, something that would have been nice in the system is to have had two x16 slots on both CPUs. One of the challenges with systems of this class is balancing I/O and avoiding UPI links. Having two x16 links would allow for 200Gbps networking from each CPU plus enough I/O for higher-end RAID controllers/ HBAs or other devices. One of the biggest challenges we had in terms of performance was simply getting everything aligned to a single NUMA node so we could have 32 cores, 256GB of memory, a 100GbE NIC, and 30x drives connected to each NUMA node.

AIC SB407-TU Performance

In terms of performance, we initially tried with TrueNAS Scale. This is always fun to see the ZFS solution scale to many drives.

TrueNAS Scale With Up To Sdai Shown
TrueNAS Scale With Up To Sdai Shown

After 15 minutes or so, we had a configuration that could hit 25GbE speeds from either bank of 30 drives.

AIC SB407 TU Performance
AIC SB407 TU Performance

We did not have several of these chassis to be able to come up with a larger-scale Ceph cluster, so instead, once we balanced the configuration on each NUMA node, we were able to push over 40Gbps of traffic from either NUMA node. Applications will vary, but by far this NUMA alignment was the most important aspect of getting performance from this system.

AIC SB407 TU CPU Performance
AIC SB407 TU CPU Performance

In terms of CPU performance, we saw fairly close to baseline performance using the Intel Xeon Gold 6330. The Intel Xeon Platinum 8368 is a significantly hotter CPU. That is one where we might suggest using lower power and lower cost CPUs for a storage server like this.

We also managed to run our NVIDIA T4  in this system. Our results mirrored our NVIDIA Tesla T4 AI Inferencing GPU Review results. The NVIDIA T4 is easy to power and cool, but cards like the T4 are important in storage servers because many of these top-loading storage servers are now being deployed into video analytics solutions. Having a local GPU minimizes network transfers of large video footage. We just had one T4 for this system, but it worked flawlessly.

Next, let us get to power consumption.


  1. Very cool, but if we’re going to talk about “cost optimizations” and being “much less expensive”, it would be nice to have a ballpark as to what the chassis cost. I didn’t see that anywhere (unless I missed it).

  2. It’s fun to see the backblaze legacy still thriving. Bulk storage will always be needed. I’m more curious when chip density will eventually overtake spinning rust in TB/$. It seems every time the gap gets close, the mad scientists create a better spinner. Who knows, maybe we’ll get rewritable crystal storage before that happens.

  3. There has been a tendency to take liberties with the “unregulated 3rd dimension” of 19″ rackable storage gear over the past few years.

    I used many Sun “Thumpers” (X4500, 4U 48-drive, 2x CPUs) in a previous life…A bit of a challenge to rack (per Sun, place said beast in the bottom of a rack).

    With the more recent Western Digital UltraStar102 (4U JBOD, 102 drives, 42″ depth iirc and 260 lbs loaded with drives) things got a bit dicey/scary with that much weight & depth hanging out of the rack when swapping a drive from the top of the chassis.

  4. I know SSDs don’t yet have the capacity of hard disks, but would converting this thing to SAS SSDs make the energy consumption and total system weight figures a little more tenable? Also, could one plug a DPU or GPU into any of those PCIe x16 slots? Y’know, to have more compute options, CUDA, etc.?

  5. Also, why don’t they do one of these up for 2.5″ drives? I’m assuming that either that’s been done before or nobody’s actually thought to do that, which in that case, why not? Smaller drives = more drives = more storage = more better, right?

  6. Where’s the required airflow ? I worked with a similar unit once, seeing our drive access pattern with an thermal camera was fun, but scary…

  7. 2.5″ HDDs here top out at 5TB…

    they are SMR (shingled magnetic recording) disks, the tracks overlap a bit. The controllers do a good job of mitigating write amplification to some degree.

    For my home lab, 2.5″ HDD was probably a mistake; better performance, reliability, density with 3.5″ disks. On the other hand, some applications prefer more small drives, I suppose for read bandwidth. Latency on my 5400 RPM 2.5″ drives isn’t going to beat 3.5″ enterprise drives.


Please enter your comment!
Please enter your name here

This site uses Akismet to reduce spam. Learn how your comment data is processed.