Recently it was discovered that many hard drives being sold are indeed shingled magnetic recording (SMR) instead of more conventional technology. Since the story broke, principally by Chris Mellor at Blocks and Files, there has been a lot of discussions online. Indeed, when the first reports were coming out, we had just finished buying Western Digital drives for, and starting a series comparing clustered versus traditional RAID storage for higher-end home and SMB setups. In the meantime, it came out that it was not just Western Digital using SMR surreptitiously in drives, but Seagate and Toshiba as well. In this piece, we wanted to go through the central issue, perspectives of the stakeholders, and how this gets fixed going forward.
A Video Version
Most of our readers prefer our web articles, but this is a piece with less hardware and more discussion. We have a video version on YouTube if you prefer to listen instead.
Feel free to listen to that one instead of reading this article during your busy day. This article has fewer amazing high-end hardware photos than our normal review content.
The 2-minute SMR and Industry Background
SMR technology has been around for years. The “MR” or magnetic recording label tells us that this is the low-level technology that is used to encode bits onto a spinning disk for data storage. Effectively write patterns are changed which allows denser data recording.
There are a lot of resources out there to explain exactly how this is done. Personally, I usually tell people to look at a great 2015 paper by Toshiba: Shingled Magnetic Recording Technologies for Large-Capacity Hard Disk Drives. In this four-page paper, you can go from knowing nothing about SMR to understanding all of the basics with relatively simple language.
The reason that people consider SMR to be a concern is that the different recording leads to very different write characteristics, especially on random writes. Something drive manufacturers use to mitigate this is to have a cache of sorts with a more conventional recording area that can ingest writes for later migration to the slower SMR section during otherwise idle periods. In the SSD world, this is very similar to how modern QLC SSDs often buffer writes in SLC then transition to slower or higher density storage later. That is a similar concept, just at faster transfer rates than in the hard drive world.
As a computer industry, storage paradigms such as legacy RAID arrays take years to develop and become mainstream, then can be widely used for decades. The vast majority of storage that consumers and SMBs have access to assume CMR drives to the point that many do not look at whether a drive is CMR or SMR, even though SMR can have a massive impact on the operation of an array.
Again, the data storage industry stepped up here. We have drive managed SMR (DMSMR) drives which effectively use the CMR “cache” and handle data placement and optimization for SMR needs. Host-aware SMR is where a host system is aware that a drive is SMR and can choose to optimize commands specifically for SMR. Host-managed SMR is where hosts and software are designed specifically to optimize and send commands for SMR drives. As a result, host-managed SMR are generally used in storage systems from big vendors and hyper-scale customers where this logic can be built into the entire storage paradigm at scale.
The main benefit of SMR, and why the technology is used, is because these large customers demand higher data densities and lower cost per PB numbers, even if individual drive performance is not great. When you are buying high-capacity drives based on the cost per pallet, or simply pound rather than even using devices, you do not care as much about the individual devices versus how you can optimize fleets of tens or hundreds of thousands, or even millions of devices.
One of the key reasons that this whole SMR in the consumer market is problematic is twofold. First, the drive manufacturers are using DMSMR which effectively hides the SMR label from host systems and customers. Second, the drive manufacturers are not telling the market they are or may be selling DMSMR drives. The net result is that the way a consumer finds it is indeed a DMSMR is that they see evidence of the performance differences which can take the form of RAID arrays failing rebuilds as a scary case.
Looking Through Lenses of the Consumer SMR Fiasco
Next, we wanted to talk a bit about the different lenses that both drive manufacturers and consumers are coming from.
Drive Manufacturer SMR Perspective
First, we wanted to discuss the perspective of why a hard drive manufacturer would put a DMSMR drive into a consumer platform. Here, the case is not performance nor reliability. Instead, it is cost.
The reason hard drive manufacturers went through brutal decades of consolidation is that the business is largely dependent on volume manufacturing. We have hit the point now where the last few deals that led to the current oligopoly were heavily scrutinized by antitrust regulators. As these firms grow, through absorbing competitors, they can use the best technology and standardize on a fewer number of models, thereby driving up volumes. Saving $1 per drive adds up when you are selling tens of millions of drives per quarter.
Hard drive manufacturers typically see volumes driven from hyper-scale customers, large PC/ server/ storage OEMs/ ODMs, and then large distribution partners. Generally, think of this as tens of millions of drives per quarter, but mostly to only a few customers.
When one looks at where Western Digital is selling its previously undisclosed SMR drives, it is in the low-capacity space. Low capacity also means a lower dollar per drive metric. Saving $0.50 or $1 here in manufacturing costs can make or break a product in the market. If using a SMR platter means you can create a drive with a lower cost or fewer number of platters, then that yields big savings.
WD stated its position, effectively saying it uses an annual workload metric, and that it does not need to disclose what technology is in its drives.
You can read the full statement here. One very strange note on this one is that it is authored “by Western Digital.” That is a bit surprising as that kind of customer care statement would normally have an executive sponsor attached. This is the official company position, however, no executive seemed to want to take ownership of it which seemed a bit strange.
We have hit the point where 1TB hard drives have street pricing of about half of a 1TB NAND SSD so hard drive manufacturers are hitting points where they are having to compete on price. To many, a $60 premium to move from a 1TB hard drive to a 1TB SSD is offset by better performance and reliability. As SSDs get closer to hard drives in terms of price per TB (and in some ways performance) selling a hard drive becomes a challenge.
Likely the use of SMR technology was seen as a benefit to delivering less expensive drives in the market when many users simply would have no idea a switch from CMR to SMR was made.
The drive manufacturers also know that SMR has a certain stigma to it. While the industry recognizes its use as a tool, especially with larger storage pools, they also undertook, as an industry, a campaign to educate the market that SMR was not for most buyers. Over the past few years, the industry has had a mantra of SMR is for large storage systems with sequential I/O. Even that Toshiba paper mentioned earlier discusses that the technology is for higher-capacity drives, not low-capacity drives such as in the sub 6TB segment. No vendor wanted to disclose the switch to SMR at low capacities because they had spent years educating the market that SMR was for a different segment.
To a drive vendor, if you want CMR, the answer is simple. One can always buy a bigger drive. That brings us to the consumer perspective.
Consumer SMR Perspective
Let us start with why consumers and small businesses buy hard drives, especially NAS drives. Today, digital content can consume a significant portion of one’s income. It can generate income for many around the world. We also have moved to an era where our records, assets, and things of real value are digital. Having a backup of that data is important whether it is in a PC, NAS, or the cloud. Soon, virtually all primary OS and application disks will transition to SSDs. In many ways, they have if you look at the mobile markets. To a consumer or a SMB using hard drives to back up and store these vital assets is putting livelihoods on the line. With that knowledge, customers need to trust storage. A big reason the storage market is littered with failed startups is just how long it takes to garner that trust because data is important, and it is vital.
There are effectively two types of consumers here. The first are ones who do not know, nor will they ever know what a SMR drive is, and why one does not want it. If you are someone who backs up a 100MB of data a day from your Excel and Word documents to a NAS or an external drive, you simply will not notice a difference. Plus, let us be realistic here, there are a lot of people who just do not care or know how any of this stuff works.
On the other hand, there are consumers who know about SMR because the industry education efforts have reached them. They understand that SMR was designed primarily for high-capacity scenarios and work best with customized systems that can handle SMR.
Let us be clear, in 2020, there are a few reasons one buys a 2TB to 6TB hard drive. The biggest, by far, is cost. There are consumers and businesses who have tight budgets and cannot afford higher capacity hard drives for their storage. Storage density is a big deal and there are significant costs to just connect a hard drive to a NAS or PC. Also, fewer drives mean less vibration, noise, power consumption, and heat. For example, at STH it has been almost three years since we have purchased under 10TB hard drives, simply to reap the benefits of higher storage density.
The 2TB to 6TB market is not for buyers like STH, or even me personally. It is for the lower-cost segments. While many of our readers in the US may think of these segments as very small or struggling businesses or consumers without exorbitant means, that is a very narrow view of the world. When we look beyond the US, smaller customers have to face distribution challenges, currency, and local market gaps, and well as tariffs that can make 2TB to 6TB drives relatively very expensive. If you have never traveled and spent time seeing this dynamic first-hand, it is actually difficult to empathize with.
Personally, it is something that I try to pay particular attention to whenever I am editing our team’s work. I am not perfect in my editing, but I am hoping that at STH we are doing a better job and continue to do so. It is hard to come from the perspective of having access to every piece of hardware at relatively inexpensive prices and remember that everyone is not as fortunate.
Effectively, buyers in the 2TB to 6TB market are often ones who do not know about SMR v. CMR or have no other choice. At this point, even if they did have the choice to use a different drive, there is a good chance that is SMR as well. To make matters more challenging, even if a buyer knows the difference between SMR and CMR, that is not enough. The fact that the market is having to pressure hard drive vendors into disclosing this information means that an educated consumer could not get the technology they want.
We are taking special note of Western Digital here. Not only is WD using SMR in their consumer drives where buyers may have no reasonable alternatives, but they are also going a step further. They are using SMR in their “premium” WD Red NAS drives. These same NAS drives are likely to be used in RAID arrays that could see specific problems with SMR access patterns. Seagate, for its part, is still using CMR in IronWolf alternatives. WD is going a step further as a disservice to end customers and hiding the fact it is selling an inferior product until it was recently caught.
Western Digital updated its response and included which drive models are SMR. At the same time, the company’s response made a glaring omission: there was no commitment to continue doing so. A fairly standard line for dealing with PR issues like this is something along the lines of:
- We screwed up
- Here is what we should have disclosed
- In the future, we will ensure this is disclosed
Usually, that message is delivered by an executive to put a face behind the pledge to do better. Western Digital’s response summary was more along the lines of:
- We did not tell you, but it should not matter to you
- Here is what we should have disclosed
- Silent on the future
WD’s blog post was not even sponsored by a named executive. If the WD60EFAX was replaced tomorrow by a WD60EFAX-V2, the company has not committed to disclosing what technology is used. The company simply stated, “We will update our marketing materials, as well as provide more information about SMR technology, including benchmarks and ideal use cases.” (Source: WD) That is not a pledge to clearly and accurately disclose recording technology on all drives going forward. It can be satisfied by updating marketing materials for the nine drives it already has. There is a subtle distinction but is one that was not worded with a pledge for future transparency.
Where Are We Now
Summing up the above perspective, we are at a pretty classical case, but one that is not attractive. The oligopoly of hard drive manufacturers, at some point, decided that moving to DMSMR for these lower capacity customers would help support the market and pricing. What is interesting, is that all of the manufacturers seem to have moved to DMSMR at around the same time, to price drives about the same. They also all decided to not disclose the use of DMSMR until presented with evidence.
The consumers who were hurt by this were the ones that trusted WD, Seagate, and Toshiba to disclose the use of SMR. All of those companies do, indeed, disclose the use of SMR technology openly for their higher-end customers.
It is almost impossible for a customer to navigate this purposeful lack of disclosure at the lower-end. These are customers who lack other options or purchasing power and the hard drive manufacturers are treating them worse because of it.
Effectively where we are is that the oligopoly of hard drive manufacturers is using inferior technology, without disclosing it, on classes of customers who have no realistic alternatives to protect their vital assets and livelihoods. That almost sounds like a plot to a movie that does not paint corporations in high-esteem.
Final Words
While at STH we may not be impacted by this today, indeed most of the systems we test all have SSDs these days perhaps with the exception of the HPE ProLiant MicroServer Gen10 Plus (we even recommended WD drives in our HPE ProLiant MicroServer Gen10 Plus Ultimate Customization Guide.) We typically buy larger drives.
Still, it seems like this is the type of information that should be disclosed to customers clearly on datasheets of all hard drive products, even external hard drives sold in the channel. That is whether they are what Western Digital, Seagate, and Toshiba think are high-value customers where this information is disclosed, or those it considers such low-value customers that the companies actively hide this information. While the dollar value of a customer may not be high to a manufacturer, the value of those drives in storing precious data can be high. This is doubly so for the WD Red drives that it says are destined for NASes, but can cause issues if they are SMR.
The big question is whether the large hard drive manufacturers are able to take this as a learning opportunity and be honest and open about their specs. If they cannot do that, and given the importance of drives to this market, as well as the structural barriers to increased competition in the market, one has to wonder if this is something we need to see a set of regulations mandating disclosures. The cost to disclose what is being sold is virtually nothing but that information can be vital to those storing their most critical assets.
For those who think it is OK to obfuscate vital product information to a lower-cost segment of customers, I offer this: just because someone or some business does not have the opportunity or means to purchase a higher-end drive, does not mean we should deprive them of an opportunity to make informed purchasing decisions to protect their livelihoods and memories.
Thank you Patrick. I’ve been reading and hearing about this SMR mess the past few weeks and was wondering if the MFRs would change and begin disclosing to all of their customers, not just disclosing on the more expensive enterprise drives. They should listen to you.
Your explanations are spot on too.
I just want to know why it took you so long to comment on this.
Thanks. Well explained and fair perspective. Hope they learn from it.
Now what can I do with my RAIDS full of SMR drives that I bought as non SMR? HDD’s were not best overall option but i ended with HDD due the lower upfront cost, now I need to replace almost new drives. Now my data is at risk, well it was before but now I know it and I can’t do a lot because of the pandemic… Ideal
Tomas H. – If you watch the video, you may notice this was written about a week ago. Then new bits came out, the video was recorded and handed off to Chris for editing. Our publishing chain is a lot longer these days, and the pipeline is full until June 2020 already. That means some of these pieces have more latency. Not ideal, but it is where we are.
@Tomas H.: I too wondered, but than was extremly satisfied by the most comprehensive, complete, neutral, clear but strong analysis and stand against this semi-cartel’s actions (which boarder on fraud, IMHO).
@Patrik: Thank you for this article, the best as i said above IMHO on the topic far and wide.
I also applaud that you took a firm principaled and honest stance, deconstructing the WD PR speak.
This is tech journalism at its best. At times when many of the decades long leaders in this space have abandoned real journalism, and replaced it with rewording PR statements, “Just buy it” pseudo reviews, and SEO-optimzed crap content factorys.
I’m with Steven. I’d call this balanced but firm. We’ve got a satellite operation in Argentina. Their menu of IT products you can get easily is astonishingly different than what we’ve got here. We’re now blacklisting WD Red drives and going to Seagate IronWolf’s.
I am curious how much this is saving WD per drive? It seems like the ill will has to be costing them much more. Not long ago, 4TB hard rives were hovering around $100, now I see Newegg is selling the WD 4TB drives for $65. It seems like any cost saving is being eaten up by lost selling price. Plus even those on the sidelines not current on technology, will hear, Stay away from Western Digital. It sure seems like to me a bad idea that blew up in their faces. And the PR from them only is making it worse.
ServerGarbage
I would run your array only as long as necessary for a full tape backup image not merely a file system level backup. Then I would send my bill including for the replacement performant array, to the vendor.
This is slam dunk tort for imposition of costs in contract and the deception involved is directly punitive multiplication at least in England I’d be on my way to high court over this already. The reputational risk of data loss or just fear that you might are greater than the enterprise in most small professional companies
If you haven’t watched the video you can just watch starting here https://youtu.be/gSionmmunMs?t=1328
That’s like how my dad used to tell me my friends and I were being dumb@$$es. LOL memories of 30 years ago
Disappointing article.
Shows CMR representation but not SMR.
Complains of “different write characteristics” but then only shows a random write benchmark with no CMR chart for comparison.
HDD manufacturers or incentivized to make a lot of drives as cheaply as possible. Sounds like capitalism, if they all do it to hit a given price point, well that’s the trade-off.
That said of course consumers deserve to know what tech is in their drives.
Uh, Andrew. Those are excerpts from the Toshiba paper they linked and showed in the video. You’re asking for the next-level information that the drive manufacturers already published. That chart they’re using is straight from a drive manufacturer showing SMR’s performance freefall on sustained RWs.
As some have commented elsewhere on this article, these low capacity drives aren’t falling in price with the introduction of SMR either.
Do the WD100EMAZ 10TB shucks described in your article “Building a TrueNAS Core 8-bay mATX ZFS NAS” employ SMR techniques? Do they have TRIM support? I have read these drives can be very slow when used for mixed read and write workloads.
Does ZFS have settings to optimise for SMR drives? It will be very interesting to see some realistic performance numbers (not just pure read or pure write benchmarks) for that newly assembled NAS.
Thanks for the reporting. It seems the newest server we built here (due to mixing of drives from different vendors in the pool) has an unpleasant mix of SMR and non SMR drives. Would it be better to divide it into two separate pools?
Nitpicking:
1: 2TB drives were $80 in 2011 with 5 year warranties. then there was a flood in Thailand and prices more than doubled
2: They didn’t return to that price until 2018 and warranties still haven’t recovered
3: These SMR drives are frequently entering the market at higher prices than their CMR predecessors
4: In the case of REDs – they are MARKETED as RAID drives, so telling people that “We only mean some kinds of raid” when previous iterations worked perfectly is gaslighting
5: These are sold as home/soho devices – so this kind of thing falls under Consumer Protection laws.
“Does ZFS have settings to optimise for SMR drives? ”
No. It’s been under discussion for at least 5 years but one of the problems is that DM-SMR drives seldom admit that they are SMR drives (they’re supposed to return a “zoned media” response. many don’t even say “trim”)
As for how bad these things are:
pool: ZFSMEDIA
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Mon Apr 20 20:16:49 2020
11.3T scanned at 23.8M/s, 11.1T issued at 23.3M/s, 32.2T total
380G resilvered, 34.44% done, 10 days 23:40:50 to go
This normally takes 27 hours AT MOST on an idle array
Putting it in even more perspective:
# zpool iostat -Td -w scsi-SATA_WDC_WD40EFAX-68J_WD-WX12D10xxxxxx
Mon 27 Apr 18:38:49 BST 2020
scsi-SATA_WDC_WD40EFAX-68J_WD-WX12D10xxxxx total_wait disk_wait syncq_wait asyncq_wait
latency read write read write read write read write scrub trim
————————————————– —– —– —– —– —– —– —– —– —– —–
1ns 0 0 0 0 0 0 0 0 0 0
3ns 0 0 0 0 0 0 0 0 0 0
7ns 0 0 0 0 0 0 0 0 0 0
15ns 0 0 0 0 0 0 0 0 0 0
31ns 0 0 0 0 0 0 0 0 0 0
63ns 0 0 0 0 0 0 0 0 0 0
127ns 0 0 0 0 0 0 0 0 0 0
255ns 0 0 0 0 0 0 0 0 0 0
511ns 0 0 0 0 0 0 0 0 0 0
1us 0 0 0 0 12 7 99 47 35.0K 0
2us 0 0 0 0 20.9K 30.4K 141K 123K 12.7M 0
4us 0 0 0 0 494K 41.3K 1.72M 4.11M 8.39M 26
8us 0 0 0 0 230K 10.5K 428K 4.50M 777K 2
16us 0 0 0 0 21.7K 1.68K 106K 399K 574K 0
32us 0 0 0 0 9.26K 181 44.5K 194K 147K 0
65us 8.69M 6 9.76M 8 4.65K 83 29.0K 75.0K 87.9K 2
131us 11.5M 152K 10.8M 170K 3.33K 47 28.4K 56.6K 80.2K 0
262us 1.54M 587K 1.38M 590K 2.36K 29 29.6K 74.6K 76.5K 0
524us 442K 420K 434K 441K 1.08K 9 26.5K 102K 78.5K 3
1ms 300K 489K 331K 683K 530 4 18.0K 135K 93.2K 1
2ms 269K 507K 363K 604K 298 1 7.18K 161K 81.9K 12
4ms 193K 450K 230K 550K 62 0 3.06K 200K 51.0K 23
8ms 162K 558K 197K 1.03M 85 0 4.24K 250K 49.5K 1.67K
16ms 163K 1000K 225K 2.17M 195 0 7.35K 294K 53.5K 20
33ms 184K 1.26M 294K 2.23M 462 0 12.4K 326K 52.5K 12
67ms 290K 833K 355K 938K 1.53K 0 19.4K 412K 77.5K 13
134ms 408K 890K 425K 1.11M 1.84K 0 32.1K 480K 80.0K 8
268ms 893K 1.19M 869K 1.52M 3.81K 0 51.2K 622K 116K 0
536ms 1.10M 2.13M 1.02M 3.04M 6.87K 0 58.3K 764K 104K 0
1s 784K 2.12M 670K 2.55M 8.43K 0 46.0K 853K 86.2K 0
2s 418K 1.68M 316K 1.41M 8.12K 0 27.9K 1.06M 60.0K 0
4s 158K 1.74M 75.9K 694K 4.44K 0 14.3K 1.31M 46.2K 0
8s 83.2K 1.88M 25.0K 211K 873 0 6.14K 1.49M 46.6K 0
17s 43.5K 1.33M 402 47.9K 33 0 454 1.19M 41.8K 0
34s 37.9K 597K 2 6.05K 0 0 110 552K 37.7K 0
68s 19.2K 159K 0 103 0 0 0 150K 19.1K 0
137s 6.59K 52.1K 0 0 0 0 0 51.1K 6.58K 0
————————————————————————————————————————
VS an Ironwolf:
# zpool iostat -Td -w scsi-SATA_ST4000VN008-2DR1_ZGY6SM46
Mon 27 Apr 18:40:07 BST 2020
scsi-SATA_ST4000VN008-2DR1_ZGY6SM46 total_wait disk_wait syncq_wait asyncq_wait
latency read write read write read write read write scrub trim
————————————————– —– —– —– —– —– —– —– —– —– —–
1ns 0 0 0 0 0 0 0 0 0 0
3ns 0 0 0 0 0 0 0 0 0 0
7ns 0 0 0 0 0 0 0 0 0 0
15ns 0 0 0 0 0 0 0 0 0 0
31ns 0 0 0 0 0 0 0 0 0 0
63ns 0 0 0 0 0 0 0 0 0 0
127ns 0 0 0 0 0 0 0 0 0 0
255ns 0 0 0 0 0 0 0 0 0 0
511ns 0 0 0 0 0 0 0 0 0 0
1us 0 0 0 0 850 1 15.8K 1.63K 86.2K 0
2us 0 0 0 0 289K 38.1K 2.20M 2.05M 28.4M 0
4us 0 0 0 0 589K 33.1K 2.37M 7.82M 16.4M 0
8us 0 0 0 0 29.7K 2.17K 269K 1.32M 1.01M 0
16us 0 0 0 0 9.23K 367 115K 340K 979K 0
32us 0 0 0 0 4.37K 36 54.0K 157K 275K 0
65us 16.4M 0 17.1M 0 1.89K 11 27.2K 56.4K 174K 0
131us 22.9M 3.87K 22.9M 5.75K 686 11 13.4K 35.1K 177K 0
262us 8.80M 1.15M 8.52M 1.19M 402 0 11.4K 55.4K 182K 0
524us 1.27M 2.59M 1.26M 2.61M 415 0 7.69K 93.7K 160K 0
1ms 669K 3.00M 703K 3.08M 372 0 5.87K 149K 149K 0
2ms 500K 3.30M 587K 3.74M 529 0 6.89K 180K 140K 0
4ms 515K 1.13M 587K 1.15M 1.11K 0 10.3K 217K 134K 0
8ms 599K 489K 639K 342K 2.88K 0 16.7K 247K 165K 0
16ms 898K 386K 938K 434K 4.96K 0 24.1K 245K 142K 0
33ms 1017K 439K 965K 1015K 7.55K 0 30.3K 279K 98.2K 0
67ms 770K 706K 682K 1.16M 10.4K 0 32.9K 394K 89.0K 0
134ms 436K 750K 336K 207K 9.63K 0 28.2K 495K 90.6K 0
268ms 248K 642K 147K 37.3K 5.48K 0 22.5K 487K 71.9K 0
536ms 111K 354K 34.4K 2.93K 2.12K 0 13.7K 276K 47.8K 0
1s 61.5K 65.4K 4.47K 884 139 0 4.87K 49.9K 46.0K 0
2s 52.1K 2.80K 220 156 0 0 194 2.21K 51.1K 0
4s 61.1K 17 93 12 0 0 0 5 61.0K 0
8s 48.6K 11 50 11 0 0 0 0 48.4K 0
17s 4.44K 0 13 0 0 0 0 0 4.39K 0
34s 5 0 0 0 0 0 0 0 5 0
68s 0 0 0 0 0 0 0 0 0 0
137s 0 0 0 0 0 0 0 0 0 0
————————————————————————————————————————
The interesting thing is that the 1/2/4/8/17s stats are virtually the same across all the CMR and SSD units – pointing to OS/ZFS or other issues – but the SMR drives are in a league of their own
I’m the one who collated most of the information and gave it to Chris. He then dug in and found the Toshiba stuff (I thought Tosh were more ethical than that)
I’m in the UK. These things are marketed as home units, so are the desktop drives. That means consumer protection laws apply.
Changing a previously working device’s internals for something that works “similarly for a few seconds”, then suffers severe degradation – 90-98% on raid rebuilds and 40-60% when treated just right, whilst keeping packaging and marketing identical, is the kind of thing that brings authorities down on a company like a ton of bricks.
To then refuse to admit they’ve done it when directly questioned by customers, then gaslight and impune customers as WD has done when this moved into the public eye moves into the realm of exemplary damages according to a magistrate friend.
WD’s April 22nd turnaround happened about 12 hours after I informed them of the last 2 paragraphs and that the only hope they had of mitigating penalties was to come clean and documente all their DM-SMR drives plus their roadmap for the future.
The problem isn’t just the NAS drives.
Think about desktop drives. Think about the worst possible case scenario for SMR drives – lots of concurrrent reads and writes – and now think about how most filesystems work – can anyone say “access time metadata updates” ?
There’s a reason a SMR desktop drive has utterly rotten performance if used as a boot drive (remember WD Blacks are sold as performance drives, so are Barracudas), or if used to load up games containing loats and lots of files.
I’ve spent the best past of the last 12 months chasing ghostly “intermittent network hangs” around the workplace. We now know these are down to the desktop drives in the linux desktop computers we were using as NFS cache units…. (Seagate, Tosh and WD drives, depending on whatever HP was putting in their £1300 Eltedesk800s that day)
The problem is far from over: On the WD blog page, there’s a link to support with a specific number to call for WD REDs – which from the UK was answered by someone speaking spanish, who then knew nothing about WD RED issues, but pulled up a canned statement that “we have been informed there is a minor problem in 3rd party NAS system where they may not perform properly in some RAID configurations which we do not recommend”
It seems WD are STILL attempting to minimise the issues and as the call cut off about that point – about 30 minutes in – I’m not really in a mood to play their silly games. It’s now in the hands of the UK courts.
@stoatwblr Hat off sir! Thank you for doing this on behalf of everyone that values their data. As far as I’m concerned, SMR driver are glorified tape drives. I’ve had a problem with a 4TB 2.5″ (15mm height) which looks so good on paper but resilvering to it takes inordinate amounts of time. I’m almost certain it’s an SMR drive but there’s no way for me to find out. This kind of thing should be the first thing after the HDD designator. CMR drivers and SMR drives are completely different devices and I have absolutely no use for the latter.
@Nikolay: It’s a virtual certainty anything larger than 2TB in 2.5″ is SMR
Even what’s come out so far is interesting by omission.
Toshiba’s “voluntary” disclosure to Chris Mellor misses out the status of their NAS, Survellance, Videostreaming and enterprise storage ranges. (they only offered data for drives showing on the skinflint lookup pages additional to the drives he specifically asked about)
Seagate are – of course – Seagate… So far we know that the Barracuda, Barracuda compute (sold to us as “perfomance storage”) and storage drives are SMR but they’re staying quiet and are likely to do so until forced to open up.
If ANYONE believes that three makers actually did this in the same sizing and product ranges independently, I have a string of bridges over the Thames I’d like to sell them, cheap and can do an extra special deal on the one next to the Tower of London.
I’ll make a final observation on this and it’s related to SSD pricing
The traditional “jumping off point” for switching from rotating to solidstate media is 4-5 times the price.
Samsing parked a howitzer across town with the 4TB 860QVO and their 16/31TB SM886 drives
Micron have done one better than that: Their Enterprise archival/coldstorage 5210 ION drives – intended to take on 10krpm SMR drives – were recently reduced by about 40% and are now only double the price of spinning Enterprise SMR media
3.96TB – £308+VAT(380+tax), 7.96TB – £580+VAT($680+tax)
This is very much “tanks parked on the hdd factory lawn”
These are only rated at 0.2-0.8DWPD, but as Micron point out, that’s still more rewrites than comparable storage gets – and I’ll point out that this kind of drive at this kind of endurance will match 80-90% of home workloads – which tend to be write once anyway
“the boresights are squarely aimed at SMR drives”
Those prices are a smidgen over 3 times the cost of a WD RED – but you get a 5 year warranty + power loss protection – which samsung’s (more expensive) 860QVO can’t match.
Micron’s 5100ECO/5100MAX/5300 enterprise SATA lines start at double those figures, but they’re Nearline drives and will take on CMR workloads.
Unless HAMR/MAMR and bit patterned media hit the market in a remarkably short period of time – and remarkably cheaply – I’d say the days of spinning media are very much numbered.
I have stayed away from the WD Red and have avoided them like the plague – but they are also not the drives for my SGI TP16000 – I use 14 and 16TB Seagate Exos drives – I have 96 14TB and 96 16TB drives. Initial failure rate was 2% – 2 drives out each pool of 96 failed within the stress period – and were swapped out with the spares I bought at the same time.
I see people shucking externals, without realizing that they are NOT drives meant for RAID duty and are cold storage drives – write lightly and then put somewhere safe. Will be funny to see those arrays start to fail because the shuckers think they found a loophole – but apparently their data is not very important to them.
It is beyond shady to sell SMR in part numbers that should be CMR. I prefer Toshiba and HGST (WD now unfortunately) over plain WD any day. I have so many dead WD, it’s starting to rival the amount of dead Razer gear. Within 10 years I don’t see HDs being viable vs SSDs.
-In Slack We Trust-
I suspected as much but they are not the only ones.
What about all those “megafast NVMe” drives that are only fast until their SLC cache runs out then
they degrade to spinner level performance for bulk transfers (70-120 MB/s). Yes Crucial P1 I’m looking at you.
I felt betrayed and personally ashamed when I figured that out when peeking IO performance with iotop with a client breathing down my neck.
I bought a 6TB Red drive last year and I noticed the performance initially appeared great (the drive’s CMR portion) which then plummeted since. At the time it appeared just odd but now I know. I think it’s in WD’s interest to label SMR vs CMR also because it gives people an incentive to buy higher-capacity CMR drives. If I had known 6TB Reds were SMR I wouldn’t have minded shelling out a little more for an 8TB drive.
Also, I knew Seagate’s Archive drives were SMR, and I recognise some people would accept that given the price/GB. Considering how unclear Seagate is with their datasheets and their somewhat unclear statements regarding what drives have undocumented SMR it makes it far more annoying at this point because I wouldn’t be surprised if they have more SMR drives which they haven’t listed as such. I think SMR vs CMR should be required on the spec sheet just like how cameras basically need to list sensor size. Considering they market SMR and non-SMR drives similarly without clarifying that certain products within the same product lines use completely different technologies, I’m sure lawsuits will find this misleading advertising.
Having looked at what drives use SMR, I suspect WD/Seagate/Toshiba will take advantage of this situation to justify stopping manufacturing of lower-capacity drives because SSDs are gradually taking over those segments and the low-capacity drives can’t compete on price much because there’s a base cost for components like a 3.5″ shell, the controller, the head and motor.
It’s telling that the manufacturers don’t utilise SMR in high-capacity 8-16TB drives for the most part, despite SMR’s purpose being to get more density (=higher capacity drives). If the manufacturers really wanted to use SMR for the original purpose they’d be making 18TB SMR drives and not use SMR at all for capacities under about 8 TB.
Despite being a good thing that we now know at least some of the drives with undocumented SMR, I’ve been tracking a Swiss retailer’s prices for HDDs and even just between last week and this week they increased the price on all Ironwolf/Ironwolf Pro drives by about 10 CHF ($10) while the SMR drives stayed at the same price. Considering WD Reds are marketed for NAS use, I think WD will have a nasty shock in lost revenue at the end of the year, because if you’re buying Reds for a NAS, you probably know a thing or two about HDDs, and this SMR mess leaves a really bad taste for the Reds.
I agree, WD has acted in a shady way by slipping SMR drives into the supply chain. I can see their argument now: “But look! The part number is different.” Indeed it is, but Joe Consumer and Jane Small Business Owner don’t know how to tell the difference. If they did “seek the knowledge” they would not have found it. I hope WD receives some very stiff fines for their actions at the least, or maybe worse YJMV (Your Jurisdiction May Vary).
I have checked 75 percent of my WD RED drives so far and they are all CMR drives; their part number does not match what WD says are the WD RED SMR drives. Fortunately those drives have been relegated to rarely & barely used “cold storage” RAID systems.
I think it is possible that the HDD makers will see this debacle as a chance to further segment the marketplace. As Patrick has pointed out, some customers pay much higher prices compared to similar goods in other countries, so expect that consumer segment to get stuck between the proverbial rock & hard place.
As for SSDs and QLC tech, no thanks, not even for my “cold storage” systems; QLC or QVO tech still seems terribly unproven to me and mostly unexplored by tech web sites. My best performing SSDs use MLC; the next level down are the 3D and TLC models. Most of my failed SSDs come from poor firmware that has no updates or vendors that do not make those updates widely known or easily obtained. I have had 1 Intel SSD “go marginal” (reports 80 percent life remaining, maybe less now) simply due to be written so much; I think it has Micron TLC tech inside, but it’s on the shelf now.
I would seriously consider using more SSDs if they used 3D or TLC at most, improved the storage density per cubic centimeter of space required, added power-fail final write support (almost all consumer SSDs lack this), and published further studies on long-term extended read-write performance. The last major survey of consumer SSD reliability that I read was published by the legacy “Tech Report” web site; the current “Tech Report” web site is sad by comparison. So if I have describe a target SSD that is larger, so be it. It sounds like I have described a consumer-grade SSD (lower TBW) based on Enterprise tech; so be it. And what I have described might not be a SSD worth producing due to low production volume and/or low demand (too small of a consumer group).
I am interested in higher capacity and higher density SSDs because the alternative is not having enough SATA connectors to hook it all up in a reasonable consumer-sized case. The more I see SSD makers evolve new potentially unstable, unreliable, and/or low-lifetime tech so they can cram more TB into the same 2.5 inch form factor, the more I cringe at the thought of the massive amounts of data loss in a medium once known for high reliability. In other words, where HDD makers are now finding themselves.
You made a lot of good points about the drive companies and their failure to come clean. In fact, if you look at the ATA specs written several years ago to accommodate SMR, it was designed to deliberately hide the fact that a drive is DM-SMR from the host. There’s no way in the ATA or SMR specs for the host to know the way they can with host-managed and host-aware SMR.
However, this does leave the people in the 2TB to 6TB range with a problem, namely cost. If the drive companies provide CMR drives in this storage range, it will cost more. Many consumers may be priced out of the market.
Ultimately the problem is that magnetic storage isn’t getting cheaper anymore. Like silicon, we’ve reached the point where it’s extremely hard to increase storage density without major compromises. In the past the 2TB to 6TB market would get drives riding down the cost curve, now that’s no longer possible.
The best solution would be for drive manufacturers to provide a way for the host to know it’s dealing with a DM-SMR drive, and for OS and NAS providers to write code that takes it into account.
Albert they already charged the same price as a Cmr drive in the red product line but made the switch to SMR and put the money in the pocket. That’s the point.
Really interesting article, shame i didn’t see it when it came out, right when i bought a 4tb wd red, not to save money but because it was the right capacity for my expected usage (should be many years before it gets even 50% full).
Still feeling annoyed that it wasn’t what i thought i was buying, and might affect ability to add drives and use RAID.
I think his was a deliberate action by WD to improve it’s bottom line at the expense of it’s customers. In the US (N.Y.), I would report this to the NYS Atty. Gen. as an attempted case of fraud by a company, making the decision to NOT DISCLOSE to it’s customer base the DANGER in using this product to Store/Keep their important/pers./sensitive data that might be needed in the future for proof of compliance in a matter of legality. ( think Income tax return, insurance, license application, proof of citizenship, etc. etc.). This company should be sited and fined (HUGE) for what they did and tried to get away with. hoping no one would notice ( Thank you STH and others for bringing this to our attention, I just found this while looking at other unrelated info, Just prior to putting in an order for their GARBAGE, think low cost home built NAS.). If these corps. want to sell product and advertise their safety/reliability they need to be honest about what they are really selling. Selling a customer/family/smb, an advertised Safe/reliable House that blows away at the first sign of a heavy wind does NO ONE ( customer, sales rep, company and it’s customer “TRUST”) any favors. I used to trust/ swear by WD. and have SPENT A LOT OF $$$$$$$$ $$$$$$$ on their products over the years. THEY HAVE JUST LOST ME ( and all those I will tell this story of Lies of Omission to), as a loyal customer for at least the foreseeable future.
thanks for the post, but one thing i still didn’t get is :for an home user where accessing the drive could be 4-5 times a day, what is the life expectancy out of these SMR drives(4TB seagate). It is possible to estimate in terms of years
Hello Patrick, thank you for excelent article. I have been wondering why no one else talks about other implications on SMR drives. Last year I bought a SMR (Without knowing it) for my yearly backup. What I do is copy my entire files to that drive and then storage it offline. For starter I notice that copying 3 TB of data to the 4 TB SMR was tremendously slow it took like 1.5 Days to do this. (First Thing no one talks about on “standard” home users). Second thing is that I needed some files of the backup , and I did place the HDD on My PC, and noticed that I can read every file, but I cant erase anything. I had to format it to make it work again “normally”. Thank you!!!
That Toshiba paper link is dead ( Shingled Magnetic Recording Technologies for Large-Capacity Hard Disk Drives). Does someone have a working link for it?