Recently it was discovered that many hard drives being sold are indeed shingled magnetic recording (SMR) instead of more conventional technology. Since the story broke, principally by Chris Mellor at Blocks and Files, there has been a lot of discussions online. Indeed, when the first reports were coming out, we had just finished buying Western Digital drives for, and starting a series comparing clustered versus traditional RAID storage for higher-end home and SMB setups. In the meantime, it came out that it was not just Western Digital using SMR surreptitiously in drives, but Seagate and Toshiba as well. In this piece, we wanted to go through the central issue, perspectives of the stakeholders, and how this gets fixed going forward.
A Video Version
Most of our readers prefer our web articles, but this is a piece with less hardware and more discussion. We have a video version on YouTube if you prefer to listen instead.
Feel free to listen to that one instead of reading this article during your busy day. This article has fewer amazing high-end hardware photos than our normal review content.
The 2-minute SMR and Industry Background
SMR technology has been around for years. The “MR” or magnetic recording label tells us that this is the low-level technology that is used to encode bits onto a spinning disk for data storage. Effectively write patterns are changed which allows denser data recording.
There are a lot of resources out there to explain exactly how this is done. Personally, I usually tell people to look at a great 2015 paper by Toshiba: Shingled Magnetic Recording Technologies for Large-Capacity Hard Disk Drives. In this four-page paper, you can go from knowing nothing about SMR to understanding all of the basics with relatively simple language.
The reason that people consider SMR to be a concern is that the different recording leads to very different write characteristics, especially on random writes. Something drive manufacturers use to mitigate this is to have a cache of sorts with a more conventional recording area that can ingest writes for later migration to the slower SMR section during otherwise idle periods. In the SSD world, this is very similar to how modern QLC SSDs often buffer writes in SLC then transition to slower or higher density storage later. That is a similar concept, just at faster transfer rates than in the hard drive world.
As a computer industry, storage paradigms such as legacy RAID arrays take years to develop and become mainstream, then can be widely used for decades. The vast majority of storage that consumers and SMBs have access to assume CMR drives to the point that many do not look at whether a drive is CMR or SMR, even though SMR can have a massive impact on the operation of an array.
Again, the data storage industry stepped up here. We have drive managed SMR (DMSMR) drives which effectively use the CMR “cache” and handle data placement and optimization for SMR needs. Host-aware SMR is where a host system is aware that a drive is SMR and can choose to optimize commands specifically for SMR. Host-managed SMR is where hosts and software are designed specifically to optimize and send commands for SMR drives. As a result, host-managed SMR are generally used in storage systems from big vendors and hyper-scale customers where this logic can be built into the entire storage paradigm at scale.
The main benefit of SMR, and why the technology is used, is because these large customers demand higher data densities and lower cost per PB numbers, even if individual drive performance is not great. When you are buying high-capacity drives based on the cost per pallet, or simply pound rather than even using devices, you do not care as much about the individual devices versus how you can optimize fleets of tens or hundreds of thousands, or even millions of devices.
One of the key reasons that this whole SMR in the consumer market is problematic is twofold. First, the drive manufacturers are using DMSMR which effectively hides the SMR label from host systems and customers. Second, the drive manufacturers are not telling the market they are or may be selling DMSMR drives. The net result is that the way a consumer finds it is indeed a DMSMR is that they see evidence of the performance differences which can take the form of RAID arrays failing rebuilds as a scary case.
Looking Through Lenses of the Consumer SMR Fiasco
Next, we wanted to talk a bit about the different lenses that both drive manufacturers and consumers are coming from.
Drive Manufacturer SMR Perspective
First, we wanted to discuss the perspective of why a hard drive manufacturer would put a DMSMR drive into a consumer platform. Here, the case is not performance nor reliability. Instead, it is cost.
The reason hard drive manufacturers went through brutal decades of consolidation is that the business is largely dependent on volume manufacturing. We have hit the point now where the last few deals that led to the current oligopoly were heavily scrutinized by antitrust regulators. As these firms grow, through absorbing competitors, they can use the best technology and standardize on a fewer number of models, thereby driving up volumes. Saving $1 per drive adds up when you are selling tens of millions of drives per quarter.
Hard drive manufacturers typically see volumes driven from hyper-scale customers, large PC/ server/ storage OEMs/ ODMs, and then large distribution partners. Generally, think of this as tens of millions of drives per quarter, but mostly to only a few customers.
When one looks at where Western Digital is selling its previously undisclosed SMR drives, it is in the low-capacity space. Low capacity also means a lower dollar per drive metric. Saving $0.50 or $1 here in manufacturing costs can make or break a product in the market. If using a SMR platter means you can create a drive with a lower cost or fewer number of platters, then that yields big savings.
WD stated its position, effectively saying it uses an annual workload metric, and that it does not need to disclose what technology is in its drives.
You can read the full statement here. One very strange note on this one is that it is authored “by Western Digital.” That is a bit surprising as that kind of customer care statement would normally have an executive sponsor attached. This is the official company position, however, no executive seemed to want to take ownership of it which seemed a bit strange.
We have hit the point where 1TB hard drives have street pricing of about half of a 1TB NAND SSD so hard drive manufacturers are hitting points where they are having to compete on price. To many, a $60 premium to move from a 1TB hard drive to a 1TB SSD is offset by better performance and reliability. As SSDs get closer to hard drives in terms of price per TB (and in some ways performance) selling a hard drive becomes a challenge.
Likely the use of SMR technology was seen as a benefit to delivering less expensive drives in the market when many users simply would have no idea a switch from CMR to SMR was made.
The drive manufacturers also know that SMR has a certain stigma to it. While the industry recognizes its use as a tool, especially with larger storage pools, they also undertook, as an industry, a campaign to educate the market that SMR was not for most buyers. Over the past few years, the industry has had a mantra of SMR is for large storage systems with sequential I/O. Even that Toshiba paper mentioned earlier discusses that the technology is for higher-capacity drives, not low-capacity drives such as in the sub 6TB segment. No vendor wanted to disclose the switch to SMR at low capacities because they had spent years educating the market that SMR was for a different segment.
To a drive vendor, if you want CMR, the answer is simple. One can always buy a bigger drive. That brings us to the consumer perspective.
Consumer SMR Perspective
Let us start with why consumers and small businesses buy hard drives, especially NAS drives. Today, digital content can consume a significant portion of one’s income. It can generate income for many around the world. We also have moved to an era where our records, assets, and things of real value are digital. Having a backup of that data is important whether it is in a PC, NAS, or the cloud. Soon, virtually all primary OS and application disks will transition to SSDs. In many ways, they have if you look at the mobile markets. To a consumer or a SMB using hard drives to back up and store these vital assets is putting livelihoods on the line. With that knowledge, customers need to trust storage. A big reason the storage market is littered with failed startups is just how long it takes to garner that trust because data is important, and it is vital.
There are effectively two types of consumers here. The first are ones who do not know, nor will they ever know what a SMR drive is, and why one does not want it. If you are someone who backs up a 100MB of data a day from your Excel and Word documents to a NAS or an external drive, you simply will not notice a difference. Plus, let us be realistic here, there are a lot of people who just do not care or know how any of this stuff works.
On the other hand, there are consumers who know about SMR because the industry education efforts have reached them. They understand that SMR was designed primarily for high-capacity scenarios and work best with customized systems that can handle SMR.
Let us be clear, in 2020, there are a few reasons one buys a 2TB to 6TB hard drive. The biggest, by far, is cost. There are consumers and businesses who have tight budgets and cannot afford higher capacity hard drives for their storage. Storage density is a big deal and there are significant costs to just connect a hard drive to a NAS or PC. Also, fewer drives mean less vibration, noise, power consumption, and heat. For example, at STH it has been almost three years since we have purchased under 10TB hard drives, simply to reap the benefits of higher storage density.
The 2TB to 6TB market is not for buyers like STH, or even me personally. It is for the lower-cost segments. While many of our readers in the US may think of these segments as very small or struggling businesses or consumers without exorbitant means, that is a very narrow view of the world. When we look beyond the US, smaller customers have to face distribution challenges, currency, and local market gaps, and well as tariffs that can make 2TB to 6TB drives relatively very expensive. If you have never traveled and spent time seeing this dynamic first-hand, it is actually difficult to empathize with.
Personally, it is something that I try to pay particular attention to whenever I am editing our team’s work. I am not perfect in my editing, but I am hoping that at STH we are doing a better job and continue to do so. It is hard to come from the perspective of having access to every piece of hardware at relatively inexpensive prices and remember that everyone is not as fortunate.
Effectively, buyers in the 2TB to 6TB market are often ones who do not know about SMR v. CMR or have no other choice. At this point, even if they did have the choice to use a different drive, there is a good chance that is SMR as well. To make matters more challenging, even if a buyer knows the difference between SMR and CMR, that is not enough. The fact that the market is having to pressure hard drive vendors into disclosing this information means that an educated consumer could not get the technology they want.
We are taking special note of Western Digital here. Not only is WD using SMR in their consumer drives where buyers may have no reasonable alternatives, but they are also going a step further. They are using SMR in their “premium” WD Red NAS drives. These same NAS drives are likely to be used in RAID arrays that could see specific problems with SMR access patterns. Seagate, for its part, is still using CMR in IronWolf alternatives. WD is going a step further as a disservice to end customers and hiding the fact it is selling an inferior product until it was recently caught.
Western Digital updated its response and included which drive models are SMR. At the same time, the company’s response made a glaring omission: there was no commitment to continue doing so. A fairly standard line for dealing with PR issues like this is something along the lines of:
- We screwed up
- Here is what we should have disclosed
- In the future, we will ensure this is disclosed
Usually, that message is delivered by an executive to put a face behind the pledge to do better. Western Digital’s response summary was more along the lines of:
- We did not tell you, but it should not matter to you
- Here is what we should have disclosed
- Silent on the future
WD’s blog post was not even sponsored by a named executive. If the WD60EFAX was replaced tomorrow by a WD60EFAX-V2, the company has not committed to disclosing what technology is used. The company simply stated, “We will update our marketing materials, as well as provide more information about SMR technology, including benchmarks and ideal use cases.” (Source: WD) That is not a pledge to clearly and accurately disclose recording technology on all drives going forward. It can be satisfied by updating marketing materials for the nine drives it already has. There is a subtle distinction but is one that was not worded with a pledge for future transparency.
Where Are We Now
Summing up the above perspective, we are at a pretty classical case, but one that is not attractive. The oligopoly of hard drive manufacturers, at some point, decided that moving to DMSMR for these lower capacity customers would help support the market and pricing. What is interesting, is that all of the manufacturers seem to have moved to DMSMR at around the same time, to price drives about the same. They also all decided to not disclose the use of DMSMR until presented with evidence.
The consumers who were hurt by this were the ones that trusted WD, Seagate, and Toshiba to disclose the use of SMR. All of those companies do, indeed, disclose the use of SMR technology openly for their higher-end customers.
It is almost impossible for a customer to navigate this purposeful lack of disclosure at the lower-end. These are customers who lack other options or purchasing power and the hard drive manufacturers are treating them worse because of it.
Effectively where we are is that the oligopoly of hard drive manufacturers is using inferior technology, without disclosing it, on classes of customers who have no realistic alternatives to protect their vital assets and livelihoods. That almost sounds like a plot to a movie that does not paint corporations in high-esteem.
While at STH we may not be impacted by this today, indeed most of the systems we test all have SSDs these days perhaps with the exception of the HPE ProLiant MicroServer Gen10 Plus (we even recommended WD drives in our HPE ProLiant MicroServer Gen10 Plus Ultimate Customization Guide.) We typically buy larger drives.
Still, it seems like this is the type of information that should be disclosed to customers clearly on datasheets of all hard drive products, even external hard drives sold in the channel. That is whether they are what Western Digital, Seagate, and Toshiba think are high-value customers where this information is disclosed, or those it considers such low-value customers that the companies actively hide this information. While the dollar value of a customer may not be high to a manufacturer, the value of those drives in storing precious data can be high. This is doubly so for the WD Red drives that it says are destined for NASes, but can cause issues if they are SMR.
The big question is whether the large hard drive manufacturers are able to take this as a learning opportunity and be honest and open about their specs. If they cannot do that, and given the importance of drives to this market, as well as the structural barriers to increased competition in the market, one has to wonder if this is something we need to see a set of regulations mandating disclosures. The cost to disclose what is being sold is virtually nothing but that information can be vital to those storing their most critical assets.
For those who think it is OK to obfuscate vital product information to a lower-cost segment of customers, I offer this: just because someone or some business does not have the opportunity or means to purchase a higher-end drive, does not mean we should deprive them of an opportunity to make informed purchasing decisions to protect their livelihoods and memories.