We have another update on the WD Red SMR drive story. To be more precise, we have several updates. Since WD has remained silent on this subject, and we, as a community, need to get the word out to keep customers informed, it was time for a follow-up piece. In this edition:
- Jim Salter at Ars Technica confirms what we saw in ZFS
- We found evidence that WD-HGST has said DM-SMR is not good for reliable performance, even discussing ZFS issues for years. Effectively, they had reason to know the drives would not work well in, at least ZFS NAS units, if not other NAS applications.
- We cover a great user story of actual experience with 6TB WD Red DM-SMR drives in a non-ZFS Synology NAS. WD did a great job doing the right thing and replacing his drives.
- We discuss a theory on how these drives were qualified by WD and drive vendors, but now de-certified/ de-qualified.
- We discuss a theory on the conspicuous silence with no updates from WD.
There is certainly a lot to cover here, so let us get to it.
If you prefer to listen along, here is the accompanying video:
As always, more detail is below for those who are interested, but we recognize some prefer different media.
Ars Technica Confirms What We Saw in ZFS
When I set the scheduled publishing time for our first hard drive SMR piece, I felt like we needed to get some data. I was very worried that the time it would take would mean others would beat us to the story. Once the first set of results came in, we saw terrible WD Red DM-SMR performance in a ZFS NAS array rebuild scenario. One of the great parts of the tech community is that others will often try for themselves. Jim Salter at Ars Technica did just that.
Jim tested a RAID 6 array using 12TB drives partitioned to 4TB to see rebuild times and various performance angles with the WD Red DM-SMR drives. He certainly found a difference in testing. In these tests, while he saw a difference, he did not see the dramatic difference we saw on the ZFS side.
Some of the charts in the article look not too bad, but when you click through to some of the next-level charts, some start to look not great. Jim mentions them in the article, but if you are skimming, you may miss this. Our advice, read Jim’s article thoroughly because he explains some less great findings that are not highlighted via charts in the default page loads.
In this example, one can see the first (default) thumbnail. If you are skimming the article, we suggest at least clicking on the other charts in these blocks because there are some that paint the WD Red DM-SMR drives in a less favorable light. Being clear, in the text, Jim discusses these in his text so it is not that they are hidden. They can just be hard to see when scrolling through and skimming.
Jim also profiled a ZFS rebuild more similar to what we did. He got similar results. Combined with what we have heard from ZFS NAS companies and heard from users, this is a fairly clear story that ZFS and DM-SMR are not a good match. We are going to talk more about that later, as well as some stories of real-users on non-ZFS platforms.
Frankly, I think it is great that Ars did this testing so there is more data out there. From experience, this requires a lot to produce. Please check out that Ars Technica article here. We may have differing views of the impact, however, I always encourage our readers to get more data to make better decisions for themselves.
As a quick note here, some folks have pointed to the “NASCompares” testing. Something I learned only in the last few days (via a major NAS vendor) is that the NASCompares folks are not an independent site like Ars or STH. Instead, according to the NAS vendor, they are affiliated with a NAS reseller who may have sold NAS units using these drives. While those results are meaningful, we are more focused on independent site results along with user experiences for this article. We encourage our readers to get any information they can, but also filter based on the context of the testing. Our testing was done without vendor support. We were simply interested in ourselves as consumers of the drives and for our readers. This is why we used ZFS as that is very common for us to use as well as for our readers.
WD-HGST Said DM-SMR Drives Are Not Suitable at an OpenZFS Conference in 2015
HGST Engineer Manfred Berger speaking at OpenZFS European Conference in 2015 gave an awesome overview of SMR technology, what it is, and why we need it. You can see the video here. He did a great job explaining why the industry is moving to SMR for hard drives since it increases storage density and the amount of storage one can get per dollar. It is one of the better talks I have seen talking about technical and practical reasons behind a SMR transition. This is a great video worth watching.
Towards the middle of the video (timestamp here), the discussion is on how HGST sees DM-SMR as not a good fit with performance issues.
Manfred is discussing with the ZFS folks development efforts to support SMR, significant for ZFS. Later in the video, he was discussing with the OpenZFS developers just how much work it would take to make SMR drives work with ZFS (mostly focusing on host managed SMR. You can see that discussion during the Q&A starting around here.
You may ask why are we talking about this excellent presentation. It turns out Manfred works at HGST, a Western Digital company. Again, given his presentation at OpenZFS in 2015, this is a knowledgeable gentleman and from that video alone has the ability to articulate storage concepts in a way that any storage company would be better for having someone like this on their team. Of note, he is not on the WD Red team per his LinkedIn profile, but HGST and WD have been active in OpenZFS. This is just an example.
In essence, we have a WD employee saying five years ago DM-SMR is no good. He is saying there are performance issues. He engaged in a discussion on the implications for ZFS. Of course, that may not represent current thinking since everyone is entitled to change opinions as they get more information, but the idea that WD-HGST had no idea that ZFS may be an option, and people may see performance issues over time, is seemingly far fetched at this point. This is what we saw in our testing data so it seems what we found is what would be expected.
ZFS is not just FreeNAS, TrueNAS, or Oracle/ Sun. Instead, it is in products as simple as Ubuntu as a desktop installation option.
We see it in products such as Proxmox VE, and we use ZFS all over the STH hosting cluster from our backup NAS to web hosting nodes.
Other vendors are getting in on this too. There are a number of smaller players that use ZFS now for their NAS system and QNAP has a line that is dedicated to using ZFS and they are increasingly looking toward ZFS. In our recent “SwitchNAServer” or QNAP QGD-1600P Review, we were greeted by a QTS hero ZFS integration announcement.
It may not be the most popular, but it is growing in popularity, even among major Linux distributions and traditional SMB NAS vendors.
The Problem Goes Beyond ZFS Into Other NASes
After publishing our test results and video, as well as our original Surreptitiously Swapping SMR into Hard Drive Lines Must Stop piece, we had several readers and folks who watched the previous video and wanted to share their experiences. One that I wanted to highlight was a gentleman by the name of Rob Janssen from the Netherlands.
Rob reached out via e-mail and was extremely open and helpful sharing his experiences of using the WD Red SMR drives in his new Synology NAS. Rob did a great job in his personal blog (note currently HTTP) documenting his experiences with the WD Red DM-SMR drives in his Synology NAS. He talked about how he, like many users, reflexively purchased WD Red because he thought they were the same high-quality drives we have had for generations. He also discussed how after installing his drives, he saw some less than great performance. Here is one example:
Even more than that, he saw strange behavior such as the NAS array working overtime to shuffle data around on the drives even when he would expect it to be idle.
Ultimately, WD did the right thing and exchanged his WD Red DM-SMR 6TB drives for WD Red Pro drives. That is a good hint that if you have these drives and are seeing poor performance, you may want to request a swap.
ZFS Issues Surfaced in Mid-2019
Rob was one of a number of folks who reached out on non-ZFS, but there were other sources pointing to issues with DM-SMR and ZFS. Chris Moore, Super Moderator at the iXsystems forums, even reached out to say how the issue has been thoroughly discussed on the iXsystems FreeNAS forms through a YouTube comment.
We double-checked and specifically searching, we confirmed there were threads in June-August 2019 where the moderators on iXsystems’ own forms were advising that the WD Red DM-SMR drives were no good (also before they shipped in FreeNAS systems in September 2019.)
One of the hard things is that this data on personal blogs, and dispersed in forums, is very difficult to corroborate and aggregate. We have a collection of user blogs (Rob’s above, as well as others), and even posts in impacted vendor’s forums not being seen by the general tech community. We had not even seen them for months due to the nature of the issue. That is a big reason consumers rely upon NAS drive vendors, as well as NAS vendors to test and certify drives.
Next, we are going to talk about how all of this discussion could be missed, and try getting some insight into what WD is doing about the situation.