Question WD P50 Problems

[orbisvicis]

I was considering purchasing this drive until I encountered this review: The World’s FASTEST USB Drive Has a Problem. Inconsistent speeds and most disturbing, data corruption reproducible across P50s, cables, and motherboards. The problem is that even though this drive was released in January there are hardly any reliable reviews. For example the TweakTown review only tests the drive with a USB 3.2 Gen 2x1 system. So has anyone encountered data corruption with this drive? Were you using USB 3.2 Gen 2x2? Are there any reported problems with the ASM3242 controller?

 

[WarthogARJ]

I've just seen this post, and I don't know about it in particular, however I AM interested is SSD behaviour in general, so this is my feedback.

The Tweaktown Review, in my opinion, is not very thorough. And that CanucksReview was also not very good.

If you look at HardwareCanucks Review, you can see that actual SSD inside the enclosure is a WD SN750, which is PCIe 3.0x4 NVMe 1.3, with Toshiba 64L TLC NAND. And uses Sandisk's own 20-82-007011 controller.

There are LOTS of reviews on the SN750, in 1TB and 2TB size, and except for it being confusing about which generation SN750 it is (there are three), I have not seen ANY issues with it anywhere.

And note that the PCIe Version 4 is rated at 16GT/sec transfer rate, which means that if it's not bottlenecked, with 4 chanels it's 8Gb/sec, and with 8 channels it's 16Gb/sec. The Sandisk controller has 8 channels, so the MAXIMUM it will pass is 18Gb/sec. And that's assuming that the NAND connections let it use all 8 channels. BUT, I think that the SN750 1TB and 2TB are one-sided, and thus use 2 NAND packages for the 1TB, and 4 for the 2TB.

So I think, and perhaps someone will say I'm wrong, but that means the MAX you can get is 8 Gb/sec out of the SN750 for the 2TB, and 4 Gb/sec for the 1TB size.

And the review that Tweaktown gives on the 1rst gen P50, is consistent, with the output being limited by the USB 3.2 connection max limit of 10Gb/sec (1250MB/sec) and actually giving around 950MB/sec.

So how can you get the enclosure outlet suddenly given MORE than the internal SN750 can actually provide even hooked up directly to a PCIe connction??

You cannot, at least not in a realistic and consistent manner.

So what's going on?

I think that there is some kind of acceleration software in the interface firmware acting to temporarily boost transfer speed, before dropping back to the base rate. Something like Samsung Magician's Rapid Mode that uses the mothernoard DRAM as a buffer. Or Primo Cache.

You can see the effects of enabling either of these on the results of the AS-SSD benchmark by looking at this forum:
AS-SSD Benchmark Fun and Games

There are examples of where a Samsung 840 EVO tested at 1,431 without RapidMode, and was boosted to 15,987 using RapidMode.

And if you are trying to transmit more info than your actual SSD can send, there's bound to be some glitches. Not in what the SSD sends out, but in what the ENCLOSURE firmware is passing onto the motherboard.

My conclusion is that you should get a straighfoward external drive where the SSD's native bandwidth matches what you are plugging it into. If you have a laptop with a USB connection, then get an external with a USB.

Otherwise, get an SSD with a PCIe, and get a SIMPLE enclosure that converts PCIe to USB. Without any attempt to speed things up. You can get the SN750 1TB at $150, and a simple enclosure to connect PCIe to USB from Amazon at $20.

 

[Billy Tallis]

And note that the PCIe Version 4 is rated at 16GT/sec transfer rate, which means that if it's not bottlenecked, with 4 chanels it's 8Gb/sec, and with 8 channels it's 16Gb/sec. The Sandisk controller has 8 channels, so the MAXIMUM it will pass is 18Gb/sec. And that's assuming that the NAND connections let it use all 8 channels. BUT, I think that the SN750 1TB and 2TB are one-sided, and thus use 2 NAND packages for the 1TB, and 4 for the 2TB.

I think you're deeply confused about the differences between PCIe lane count, PCIe link speed, NAND channel count and NAND channel speed. The PCIe lanes are the host interface between the SSD controller and the computer's CPU, though in this case there's a USB to NVMe bridge chip in the way. The NAND channels are between the SSD controller and the flash memory. The SSD controller itself is infinitely more complex than an analog telephone switchboard, and it does not simply connect NAND channels to PCIe lanes.

 

[orbisvicis]

So how can you get the enclosure outlet suddenly given MORE than the internal SN750 can actually provide even hooked up directly to a PCIe connction??

All my information comes from The Western Digital WD Black SN750 SSD Review: Why Fix What Isn't Broken? Isn't the SN750 the 3rd generation of WD Black? Furthermore there are two specific cases where the SN750 PCIe can saturate or exceed theoretical USB 3.2 2x2 speeds (sequential 128kB read with QD > 8 and burst 128kB sequential write) and many cases where it exceeds USB 3.2 2x1 speeds.

 

[WarthogARJ]

I think you're deeply confused about the differences between PCIe lane count, PCIe link speed, NAND channel count and NAND channel speed. The PCIe lanes are the host interface between the SSD controller and the computer's CPU, though in this case there's a USB to NVMe bridge chip in the way. The NAND channels are between the SSD controller and the flash memory. The SSD controller itself is infinitely more complex than an analog telephone switchboard, and it does not simply connect NAND channels to PCIe lanes.

"Deeply confused" am I?
Heh heh heh.....

Well, seeing as you guys are one of the reviewers distributing information about SSD's, then I think that means something as well....:-}

So yes, I understand that the relationship between the controller and the NAND chips is quite complex, but at the end of the day, there is an upper limit on what any component in the chain can transmit. And the fundamentals are pretty simple.

If you haven't seen this review, it's worth a read, because they have a nice SIMPLE model that describes the upper limit to transfer rate:
Grupp, L. M., Davis, J. D. & Swanson, S. The Bleak Future of NAND Flash Memory. 8.

Look at Table 3, where they summarise their Model. And don't be confused by the main paper discussing limits to planar NAND, the model is talking about the general relationship been NAND and channel speed, and the concept is applicable to 3D NAND.

Otherwise, you can read up on it in Micheloni's book: "Inside NAND Flash Memories". I trust you have a copy?

Or else there's a good conference paper by Giovanni of Micron, given at ISSC 2010. If you look at his analysis of max bandwidth of that device, you can see how it is applied elsewhere:
Marotta et al. - 2010 - A 3bitcell 32Gb NAND flash memory at 34nm with 6MBs program throughput and with dynamic 2bcell blocks...

So no offence, but maybe you've missed the point here, and are thinking it's too complex to understand it at a basic level.

My point was that even though a controller CAN transit at a high rate, it needs all its channels to do so, and if the SSD card is only using a fraction of the total available, then it's NOT going to be as fast as it might be.

As a Reviewer, I think that when you review an SSD, you should give the details of NAND, in terms of die size, number of dies per package, and total packages. And comment on how that affects throughput. And ideally add in specifics about the NAND itself, like latency, page size etc. It's not easy to get some of that, although you can if you look hard.

I see that some Reviews done by your precursor Anand La Shimpi had more in depth analysis, that's what I'm refering to as a more thorough review. I'm not saing the ones you write are bad per se, but it would be nice to have more substance than just regurgitating the results from canned benchmarks.

Your're new here, but this type of post is not allowed.
Don't sign up and then tell people who are pretty knowledgeable
concerning SSDs to "read a book" or "it's too complex to understand".

AT Mod Usandthem

 

[NewMaxx]

Otherwise, you can read up on it in Micheloni's book: "Inside NAND Flash Memories". I trust you have a copy?

...

As a Reviewer, I think that when you review an SSD, you should give the details of NAND, in terms of die size, number of dies per package, and total packages. And comment on how that affects throughput. And ideally add in specifics about the NAND itself, like latency, page size etc. It's not easy to get some of that, although you can if you look hard.

I can't speak for him, but I certainly do, as well as the rest of those sources. Plus a few hundred more that I share - the textbooks are copyrighted.

I personally do cover all those details in my posts (mostly on Reddit) but I'm not entirely sure how relevant they are to reviews, especially if it's information that's being retreaded, although I certainly respect the desire to see them.

Nevertheless, let me address some of your statements above.

except for it being confusing about which generation SN750 it is (there are three)

There's only one SN750 which is based on the SN720 (there's also a SN730 with 96L flash). The 2TB version does use denser flash, though. In any case, there are three Blacks, although the SN750 and Black from 2018 share the same basic hardware (as explained in Tallis's review).

Sandisk controller has 8 channels, so the MAXIMUM it will pass is 18Gb/sec.

The controller will ultimately be limited by the PHY, that is x4 PCIe 3.0, since an 8-channel controller that can handle, say, 667 MT/s 8-bit flash, is capable of far more than the 3550 MB/s or so maximum of that PCIe link. As in, 667 * 8 * 8 = ~42Gb/s. Also, don't confuse GB and Gb.

I think that the SN750 1TB and 2TB are one-sided, and thus use 2 NAND packages for the 1TB, and 4 for the 2TB.

No, it uses 512Gb/die in 16DP/HDP with just two packages at 2TB. 16 * 2 * 64 = 2048GiB

the MAX you can get is 8 Gb/sec out of the SN750 for the 2TB, and 4 Gb/sec for the 1TB size.

You're limited by the bridge chip either way. A lot of the 10 Gbps (2x1) bridge chips are x2 PCIe on one side but the ASM2364 is not. So you're limited to 20 Gbps at 128b/132b encoding (double payload and header, unlike just payload with 128b/130b PCIe 3.0+) which after overhead (~15%) is around 2100 MB/s. About half that with 10 Gbps.

My point was that even though a controller CAN transit at a high rate, it needs all its channels to do so, and if the SSD card is only using a fraction of the total available, then it's NOT going to be as fast as it might be.

If you're just saying "the drive is bottlenecked by the slowest part in the chain, that is the USB side of the bridge chip" then, yeah, obviously. Not really sure what your arguments are beyond that as you don't seem to realize that pretty much all portable drives and enclosures on the market use a bridge chip.

 

[WarthogARJ]

I can't speak for him, but I certainly do, as well as the rest of those sources. Plus a few hundred more that I share - the textbooks are copyrighted.

I personally do cover all those details in my posts (mostly on Reddit) but I'm not entirely sure how relevant they are to reviews, especially if it's information that's being retreaded, although I certainly respect the desire to see them.

Nevertheless, let me address some of your statements above.



There's only one SN750 which is based on the SN720 (there's also a SN730 with 96L flash). The 2TB version does use denser flash, though. In any case, there are three Blacks, although the SN750 and Black from 2018 share the same basic hardware (as explained in Tallis's review).



The controller will ultimately be limited by the PHY, that is x4 PCIe 3.0, since an 8-channel controller that can handle, say, 667 MT/s 8-bit flash, is capable of far more than the 3550 MB/s or so maximum of that PCIe link. As in, 667 * 8 * 8 = ~42Gb/s. Also, don't confuse GB and Gb.



No, it uses 512Gb/die in 16DP/HDP with just two packages at 2TB. 16 * 2 * 64 = 2048GiB



You're limited by the bridge chip either way. A lot of the 10 Gbps (2x1) bridge chips are x2 PCIe on one side but the ASM2364 is not. So you're limited to 20 Gbps at 128b/132b encoding (double payload and header, unlike just payload with 128b/130b PCIe 3.0+) which after overhead (~15%) is around 2100 MB/s. About half that with 10 Gbps.



If you're just saying "the drive is bottlenecked by the slowest part in the chain, that is the USB side of the bridge chip" then, yeah, obviously. Not really sure what your arguments are beyond that as you don't seem to realize that pretty much all portable drives and enclosures on the market use a bridge chip.

Thanks for the reply.

And as I said at the start, I'm interested in SSD behaviour, and I'm still learning.

My MAIN point is that I don't think that this external SSD can actually transfer at the claimed maximum speed. I might well not have identified the actual bottle neck very well, but yiou seem very well equipped to do so.

So rather than find fault in the DETAILS that I present, why not address the actual qustion: about how fast one should expect an External USB SSD that contains a WD SN750?

And perhaps look at the issues of thermal throttling, which in an enclosed External SSD must be very likely, seeing as any heat transfer shields are going to be pretty ineffective.

So far, I've just seen your reply, and that of Billy Tallish as a bit of one-upmanship. I think a more constructive response would be to see if my comment about the restrictions due to using an existing M.2 SSD are useful to explain poor behaviour. And then to ADD to it, and work out the bugs in what I have observed.

So far, neither of you have managed to address the actual question in the posting.

 

[NewMaxx]

So rather than find fault in the DETAILS that I present, why not address the actual qustion: about how fast one should expect an External USB SSD that contains a WD SN750?

The WD P50, which has a SN750 internally, is rated ("top speed") for a maximum of 2000 MB/s. Again, if you look at my math - sequential transfer speed will be 20 Gbit/s divided by 8 and times 1024 to get it in MB/s, 128b/132b encoding, and ~15% latency/overhead from USB - this is about right, keeping in mind queue depth/threading plays a role. This is a raw estimate and further you have to be careful comparing decimal and binary values especially since PCIe is given in GT/s which is decimal.

If you want "proof" that it can hit that speed, just look here at a drive using the same bridge chip over an appropriate connection. Imagine that, 2100 MB/s.

And perhaps look at the issues of thermal throttling, which in an enclosed External SSD must be very likely, seeing as any heat transfer shields are going to be pretty ineffective.

All of my "quick looks" do include thermal monitoring. I was unable to get any portable drive to throttle even with full-drive writes, although I have not tested this specific bridge chip/drive. However I had someone take apart a P50 and it has thermal interfacing directly to the metal case - I don't consider it a massive concern. Keep in mind that the SN750 runs relatively cool (see TPU's WD Black 2018 review) because it has a small, static SLC cache, prolonged writes will be in TLC, plus the controller is incredibly efficient under load (see AnandTech's SN750 review).

So far, neither of you have managed to address the actual question in the posting.

Bridge controllers are notoriously fickle. My ASM2362 will drop files on my X570 board, but it's fine on my Z170 - this means the host controller plays a role (something AnandTech tests/compares for its portable drive reviews quite often). Also, check the SLC cache results on Tallis's SN750 review and consider that a file transfer is effectively one thread and QD1 plus has the USB limitation; write speeds are not going to be consistent. The linked review also states ASM3242 but I've seen the inside and the ASUS enclosure they mention uses the ASM2364.

I only replied to back up Tallis who I can assure you knows what he's talking about. It's true that he and my reply did not address the OP directly, but now I have.

 

[WarthogARJ]

So Billy, and NewMaxx, could we please hear your input with respect to the actual question that was posted? That would be very interesting.

Especially with regards to my comment that the internal drive is an SN750. Which the Tweaktown Review did not mention, and quite possibly never know, and the Canucks one noted in a photo, but didn't consider its implications.

And I'd be interested to hear what you both say about the effect of using a one or two sided M.2 card in a large size. In terms of allowing optimal performance.

An external USB card does not really have a thickness requirement, so one could use a double sided PCB, if there was an advantage to it. And that means for a 1TB, and even a 2TB, you could use 256 or 512 Gb NAND flash if it helped you.

In addition, I haven't seen any comments at all in Anandtech reviews (or by other Reviewers) for External SSD about thermal throttling, or at least it's not something they seem to address on a regular basis. Why not?

I understand that it's less likely to occur on a drive that is already throttled by using a slow USB/PCIe interface, but it MUST occur. I know that a Twerktown Review on an Extenal SSD mentioned "it got very warm to touch" in the comments section, but didnt actually look into what that implied.

 

[NewMaxx]

Check my reply immediately above, particularly the last section, for details. Yes, the P50 is a SN750 internally, I had a user take one apart. The amount of sides a drive has does not in any way impact performance directly, only indirectly.

 

[WarthogARJ]

Hi Newmaxx,
Excellent and very helpful reply.
Not only to the initial Poster, but to me too.

I will go over my math for how I got to my conclusion that the potential speeds were wrong: I'm sure what you've just posted is correct.
I've probably made a mistake with Gb and GB somewhere, and bits vs bytes.
I'm a materials engineer, and except for magnetism, our nomenclature is much clearer.

As far as my comment about one sided or two sided, my point is that it is indded an INDIRECT constraint. Obviously the SSD does'nt care if it's got a 1 or 2 sided PCB, but the point is due to die sizes, and numbers per package, and the physica size of a package, it's pretty obvious that you've added a potential artificial constraint to the SSD's behaviour by limiting it to one side. Which is just saying two packages.

As far as thermal throttling is concerned, I do think it should be monitored in a more rigorous manner, and reported on in a review.

Looking at the development of SSD's, from a heat transfer perspective, it's been extremely hit and miss. And I don't see the Reviewers to have been particularily effective in monitoring it, with the exception of TechPowerUp, and he usually has a section on it, with test data, and even an IR scan from a heat gun.

Perhaps this is more obvious to a meterials engineer than to an IT guy, buy when you are trying to carry out operations at greater and greater speeds, in smaller and smaller sizes, constrained not only in 2 dimensions, but in three, and then you start adding in the requirements of monitoring changes in voltage from 2 states to sixteen.....well, obviously heat transfer and its effect on temperature has a very large effect.

There's even a hint there when IT people start to use the term "entropy" to describe the relative order in files that are transferred. Well, to a physical chemist or engineer, if you start to talk about entropy, and rates of change (i.e reaction), you immediately bring in enthalpy.....which is HEAT. Or more precisely, ENERGY.

So I think anyone who is going to try to review SSD's properly, needs to be able to look at thermal effects. Otherwise you are wasting your time.

If I was to review an External SSD, I'd want to be able to monitor TEMPERATURES. Ideally from any internal sensors there were, and I'd use an external monitor like an IR heat gun as well. They are not expensive.

 

[NewMaxx]

Sequential speed is a factor of n-way interleaving via c-channels, d-dies, and p-planes, such that n = c * d * p. A NAND package can have anywhere from 1 to 16 dies typically but this means the 1TB and 2TB SN750 have the same amount of dies and interleaving, which is in any case irrelevant for reads and SLC writes as it hits the limits of x4 PCIe 3.0. TLC write speed may vary depending on the amount of planes/LUNs, the controller, flash speed. The 970 EVO and EVO Plus work similarly to the SN750. Being single-sided makes them more flexible for laptop/client use. However, they both hit maximum speed just fine at 16DP with 256Gb flash which is 1TB with two packages, so irrelevant. For that matter, the 2TB SM2262EN (double-sided) drives also use 512Gb flash, which tends to be cheaper per GB (higher wafer yield) and the use of 8DP with higher yields can ostensibly reduce cost but performance there is only a factor because the controller is limited to 32 CE - 4 per 8 channels - which normally engages two-plane flash for only 64 LUNs when it can scale further; the 970 EVO Plus and SN750 do not have this issue.

Heat and power is a different discussion although JEDEC has means of testing stress temperature, swing temperature (program/read), etc. as part of the spec for the NAND. The controllers themselves are ARM Cortex-R5 usually (with some exceptions) with their own limits but the composite temperature is generally set to match a throttling scenario in the >70C range. However, if you're power- or performance-limited as you are over USB and further know there's a thermal interface, chances are heat isn't necessarily a huge factor. TechPowerUp does test SSD temperatures which is nice, though.

 

[orbisvicis]

Thanks for the response. I wasn't sure if the silent data corruption reported by HardwareCanucks was just another incompatibility per *STABLE* NVMe - USB Adapter? (which never mentions data corruption) or an actual enclosure issue. Data corruption is frightening.

The SN750 does run cool. I briefly considered the Asus ROG STRIX Arion enclosure but didn't want to mess with the thermal characteristics of the drive especially after I saw photos of the WD P50 teardown for NewMaxx on Reddit. Also I worried the enclosure's RGB lighting could cause the drive to exceed its power budget - probably not so much an issue for USB-C at 7.5W minimum. And what if the lights couldn't be permanently disabled...

 

[WarthogARJ]

Sequential speed is a factor of n-way interleaving via c-channels, d-dies, and p-planes, such that n = c * d * p. A NAND package can have anywhere from 1 to 16 dies typically but this means the 1TB and 2TB SN750 have the same amount of dies and interleaving, which is in any case irrelevant for reads and SLC writes as it hits the limits of x4 PCIe 3.0. TLC write speed may vary depending on the amount of planes/LUNs, the controller, flash speed. The 970 EVO and EVO Plus work similarly to the SN750. Being single-sided makes them more flexible for laptop/client use. However, they both hit maximum speed just fine at 16DP with 256Gb flash which is 1TB with two packages, so irrelevant. For that matter, the 2TB SM2262EN (double-sided) drives also use 512Gb flash, which tends to be cheaper per GB (higher wafer yield) and the use of 8DP with higher yields can ostensibly reduce cost but performance there is only a factor because the controller is limited to 32 CE - 4 per 8 channels - which normally engages two-plane flash for only 64 LUNs when it can scale further; the 970 EVO Plus and SN750 do not have this issue.

Heat and power is a different discussion although JEDEC has means of testing stress temperature, swing temperature (program/read), etc. as part of the spec for the NAND. The controllers themselves are ARM Cortex-R5 usually (with some exceptions) with their own limits but the composite temperature is generally set to match a throttling scenario in the >70C range. However, if you're power- or performance-limited as you are over USB and further know there's a thermal interface, chances are heat isn't necessarily a huge factor. TechPowerUp does test SSD temperatures which is nice, though.

Thanks: a very helpful reply.

First of all, my comment about being single or double sided is based on what I've seen with some SSD's, especially 2TB, where it can add a significant constraint in terms of the package/die variants, and DRAM that the manufacturer can use, for the desired cost. It's hard squeezing onto just one side the DRAM and two NAND packages plus assorted other bits and pieces.

And even in 1TB it can be a constraint, because the newest NAND is not always available in the denser variants when the model cycle starts, and so a manufacturer might use an older NAND generation to allow them to fit it all on one side.

I'm not talking about the high end SSD's, like Samsung Pro etc, but the middle of the range ones. You can see where its affected performance
because of the flash the manufacturer was forced to use because of it.

As far as sequential speed being related to NAND package architecture: sure. A good SSD Review should report the NAND package STRUCTURE, as in number of dies, page and block sizes and quantity, and read/erase/program latencies.

Otherwise it's a bit like reading a review about a car, where the reviewer doesn't know how many cylinders it has, or if it's got twin turbos etc etc.

I know of the relatonships between dies, planes, LUN etc. However the issue is that it's very hard to get datasheets for the newer NAND Flash: it's all tied up in NDA-only release. I've got data sheets for the earlier generations: that's easy. But for 3D it's much harder. But the reviewers themselves don't seem to be able to get this information from the manufacturers, so they are pretty much in the dark.

And I'm interested in SSD behaviour under all loads, so thermal effects are indeed an issue of interest to me. And in any case, any big write load you put on an SSD that lasts for longer than a few minutes can easily get the controller up to 70 C if it has no heat dissipation measures. That's been shown by the reviewers who do measure temperatures. And from the manufacturers as well as some research on it.

So I think it's poor for a Reviewer to run an intensive write or mixed load task and not at least monitor (and report) the SSD temperature. The SSD itself usually has at least two sensors.

And ideally monitor it with an IR sensor. You don't need anything extremely fancy, as long as you can bring it close enough so that the resolution of the sensor gives you sufficient detail over the controller package to get an accurate spot temperature, and not an average over a larger area.

There's an FLIR ready-made thermal camera that uses the FLIR Lepton 3.5 sensor that should work. It's got a radiometric accuracy of +/- 2 C / 5%, spatial sensitivity of 12um per pixel with sensor resolution of 160x120 pixels.

If you do a dry run before, and note where the hot spots are, you can ensure you get proper coverage of them if you use this.

In addition, the temperature of the NAND flash itself is important, because that affects the NAND microstructure: specifically the oxide layers. And THAT has a direct effect on endurance. It's roughly an Arrenhius relationship, so even temperatures of 40 C start to affect the flash over a long period.

 

[WarthogARJ]

Sequential speed is a factor of n-way interleaving via c-channels, d-dies, and p-planes, such that n = c * d * p. A NAND package can have

Heat and power is a different discussion although JEDEC has means of testing stress temperature, swing temperature (program/read), etc.

In any case, I appreciate your input. I've looked at pretty much all of the review sites, but not really looked at Reddit. I've never really been on it much. But if you post on that, I'll have a look at your stuff.

I wish that the SSD reviewers had a better archive of information, such as the one on TPU for GPU's. I think that's the gold standard for archiving review information on computer components.

I'll post more on another thread, but the reason why this question was asked, was because the reviews done on SSD's are not very thorough or consistent. There are perhaps too many SSD models released every year, so nobody covers a very broad range of them. So if someone wants information on a particular model, you really have to hunt. And if you want to compare it to another one of your choice, you need to hunt again.