This weekend I was doing some work on the new ServeTheHome storage test bed. It is a very fast machine. Once I got Windows Server 2012 working on the Supermicro X9DR7-LN4F, I ran into some troubling issues. I first added five different SSD combinations onto the onboard SAS 2308 controller. Write performance was simply abysmal. Benchmarking on an Intel PCH is very easy, everything just works out of the box. Benchmarking on the LSI SAS 2308 controller is a bit harder as I have been spending more time tuning the system. One area was to change the write cache setting from the default on to the off setting.
Drives were all 240GB or 256GB in capacity. This is a decent cross section and I did a round-up this summer of several SSD’s and their Anvil’s Storage Utilities Performance. What we found was some controllers respond differently to the default settings and changing write cache settings in Windows Server 2012 may help performance. Also, using Anvil’s Storage Utilities (which is a great benchmark) turning on the write through setting can help performance. Let’s take a look at those situations.
For this piece, I am using the new SSD test configuration.
- CPUs: 2x Intel Xeon E5-2690 CPUs
- Motherboard: Supermicro X9DR7-LN4F
- Memory: 8x 4GB Kingston unbuffered ECC 1333MHz DIMMs
- SSD: Corsair Force3 120GB
- Test SSDs: OCZ Vertex 4 256GB, Kingston SSDNow V+200 240GB, SanDisk Extreme 240GB, Samsung 840 Pro 256GB
- Power Supply: Corsair AX850 850w 80 Plus Gold
- Chassis: Norco RPC-4220
- Operating System: Windows Server 2012 Datacenter
The strange thing here is that there has not been a lot by way of driver updates since the end of May. Certainly not around the release of Windows Server 2012 nor Windows 8.
Write Cache On Versus Off with a SSD on a LSI SAS 2308 and Write Through Mode in Anvil’s Storage Utilities
By default, Windows Server 2012 turned write cache on for new drives. The impact of this cannot be overstated. Write speeds plummeted across every drive. Here are a few examples of write cache on versus write cache off in Windows Server 2012 with a few popular SSD offerings. Just for those that are not aware, here is the setting we are changing:
Leading practice here is to change the setting then reboot the system.
The other setting that we are looking at is enabling Write Through on Anvil’s Storage Utilities. You can find this by going to Settings.
Next we are going to look at a quick sample of SSDs. The first picture will be how the setup worked out of the box on the LSI SAS 2308. The second picture for each drive will show performance when write cache was disabled. The third picture for each drive will show what happens when the write-through option is selected. You will notice that some of these were taken on Windows 7 and others on Windows Server 2012. This was simply due to the fact we did 5x SSDs with On/ Off Write Cache, On/ Off Write-Through, two driver versions, two operating systems, two firmware modes (IR and IT) and etc. The reason we are highlighting the write cache and write through settings are that they did show a marked impact on the results.
OCZ Vertex 4 Write Cache On v. Off and Write Through
The OCZ Vertex 4 is known for its speed on shorter tests. Using the 256GB variant and not so large test sizes means that the drive will be extremely fast if all is working.
Interestingly enough, using the Intel PCH the Vertex 4 achieves over 4,600 total and 2,800 on write. Here is the view with the write cache off.
As you can see, not much of an improvement by changing cache settings. Let’s try write-through.
Here we see a slight improvement. Nothing overly great but the best benchmark yet. Let’s see how other drives fare.
Kingston SSDNow V+200 240GB
The Kingston SSDNow V+200 240GB hit the scenes and is a decent performing drive given it uses asynchronous NAND. Let’s take a look at the Anvil’s Storage Utilities results.
Yikes! What is going on there? If you look at the 4K writes, performance is much lower than we saw in the previous Kingston SSDNow V+200 240GB review. For those that don’t want to click, we would expect to see around 76MB/s for the 4K and about 220MB/s for the 4K QD16 tests on an Intel PCH. Let’s see what happens when write cache is turned off.
Numbers here are still a bit low. With that being said, much better than we saw with Write Cache On. What happens when we turn on write through?
Not much of an improvement on the Kingston SSDNow V+200 with write through, but still around 2.5x the stock out-of-the-box speeds. Let’s see if there is a trend with SandForce controllers.
SanDisk Extreme 240GB
SanDisk made a splash with a SandForce SF-2281 based drive with Toggle NAND. I am keeping firmware consistent here to keep results somewhat standardized with the old ones. Plus, adding TRIM does not do much good on an LSI controller at the moment. Let’s again start with write cache on.
Again, yikes! Write cache in Windows plus the SandForce controller seems to absolutely kill write performance. Even sequential speeds are extremely low. Let’s see if the same pattern emerges as with the Kingston SSDNow V+200 with the Write Cache turned off.
Much better performance again for the SandForce based SanDisk Extreme 240GB. Turning off write cache in Windows Server 2012 improves write performance a ton. Performance still lags behind the Windows 7/ Intel Z77 PCH results but is vastly improved. Let’s try turning on write-through.
Here we see significantly better performance.
Samsung 840 Pro 256GB
When looking around the web, and all of the Intel PCH based benchmarks, the Samsung 840 Pro is fast. Yet, the Samsung 840 Pro out of the box was a drive that showed not so great performance with Windows Server 2012, a LSI SAS 2308 HBA and write cache on.
Again, write cache on in Windows Server 2012 takes the hottest consumer SSD around today and brings it to sub 10,000 4K IOPS. This looks very much in-line with the SandForce drives right? Here’s what happens when the write cache box is de-selected:
Performance dips by a small amount, more or less test variance. Similar behavior as to the OCZ Vertex 4. The Samsung 830 behaves in a manner similar to the Samsung 840 Pro on these test so those results are omitted since they are duplicative. Let’s see what happens when we toggle write-through.
Bottom line here, if you have a Samsung drive, on a LSI SAS 2308 controller be very careful with benchmark settings. I did want to point out that IOMeter managed to provide almost 17K on QD1 and over 40K on QD4 sequential 4K write tests so the above seems correct.
This is one of those times that we see totally unexpected results. When I first started testing, I expected performance to be +/- 10% of an Intel PCH chipset using Windows Server 2012 default settings. I also expected everything to be plug-and-play easy. It turns out, for SandForce drives, changing the write cache settings makes a huge impact. For Samsung drives, with Anvil’s Storage Utilities, you will want to enable write through mode. It was really interesting to see these results. The drives did not hit their Intel PCH speeds. Also, when using a LSI controller, one does need to pay attention to various settings. It seems like some of these benchmarks are optimized for the Intel + SSD platform. That makes sense as SSDs are commonly found in workstations. Bottom line, if you do run a similar setup, do spend the time to ensure everything is configured properly.