Do you think there will ever be 3d xpoint M2 SSD and how long that will take ?

[William Gaatjes]

At what price point ?
Just wondering what the current state is of 3d xpoint memory.
I see ssd and pcie -x4 cards.
But not m2 format yet.
Is 3d xpoint still in its infancy ? Running too hot or not integrated enough ?


edit:
typo too to

 

[whm1974]

Intel does has M.2 sticks with 3D Xpoint memory and for awhile they bundling them with some NUCs. However right now those are being used for caching. I have no idea what Crucial/Micron is doing with it.

 

[IntelUser2000]

Yes.

https://www.anandtech.com/show/12899/intel-previews-m2-optane-ssd-905p-380gb

Said to be identical in performance to the regular 905P. Price if anything will be higher. Well, maybe the 380GB will cost same as the 480GB 900P.

 

[cbn]

Intel 800p.

 

[whm1974]

Yes.

https://www.anandtech.com/show/12899/intel-previews-m2-optane-ssd-905p-380gb

Said to be identical in performance to the regular 905P. Price if anything will be higher. Well, maybe the 380GB will cost same as the 480GB 900P.

It's also 110mm enterprise length instead of the more common consumer 80mm one. So you will to wait awhile yet.

 

[cbn]

Carson Beach will probably be the first PCIe 3.0 x 4 with 2280 form factor.

 

[IntelUser2000]

Carson Beach will probably be the first PCIe 3.0 x 4 with 2280 form factor.

I'm not sure if that is still on their roadmap anymore. The Optane M10, for example has a single review, the Anandtech one. I cannot find the M10 out of that single review, and some older rumor news articles. Its technically sold but zero user reviews out there. There's also the tidbit about 800P deliberately using x2 connection because it offers lower latency(presumably compared to x4).

It's also 110mm enterprise length instead of the more common consumer 80mm one.

While that's true, most if not all ATX boards support 110 mm length. I don't think the mITX and SFF computer builders are rushing out to buy the 900/905P anyway.

If you really want the M.2, then there's the 800P. Yes, I think the price needs to be about 30% cheaper just to go from ripoff to expensive. But there ya go. The 900/905P is actually not that pricey if you consider that it has much better endurance statistics and max sequential ratings, especially writes. The 900P is cheaper per GB than the 800P.

However right now those are being used for caching.

800P is officially not for caching. Although it doesn't matter as you can use the official Optane Memory application for every Optane drive. 800P, Optane Memory, M10, 900/905P can all become a cache drive.

 

[cbn]

If you really want the M.2, then there's the 800P. Yes, I think the price needs to be about 30% cheaper just to go from ripoff to expensive

Software that would allow caching NVMe NAND would help a lot IMO. (I know you mentioned this in the past as well)

Then a person doesn't need so much Optane if only the small files are being cached (PCIe 3.0 x 2 is just as good as PCIe 3.0 x 4 for 4K QD1 read).

Hopefully Intel is not that far away from getting this to happen? (I am thinking having the option for write through rather than only write back would allow even the smallest optane to work well even with the highest capacity and fastest PCIe 3.0 x 4 3D TLC drive)

P.S. Would like to see caching of NVMe NAND RAID arrays as well. (Maybe with Optane DIMMs on the horizon this will come fairly soon.)

 

[IntelUser2000]

Software that would allow caching NVMe NAND would help a lot IMO. (I know you mentioned this in the past as well)

Then a person doesn't need so much Optane if only the small files are being cached (PCIe 3.0 x 2 is just as good as PCIe 3.0 x 4 for 4K QD1 read).

It's much harder to make it work in practice because caching itself has an overhead. This is why ideally the gap between the cache and the cached drive has to be big.

The best is eventually if they decide to use Optane dies to replace DRAM dies on future NAND SSDs. While it'll be slower, the capacity can be 5-10x larger. That allows eliminating software overhead and purpose tuning the firmware which you can't do with software schemes. I thought something like a future 770p Intel TLC SSD using Optane would be a great idea.

 

[IntelUser2000]

My initial expectations are being more and more met as time passes by

Out of 1) Caching 2) SSD 3) DIMM, only #1 and #3 makes sense in the long run. The reason is because they are on the extremes.

Caching fulfills its purpose of speeding up in cost-constrained scenarios. Optane cache is also superior to NAND based cache due to its nature(Consistency, increased reliability, very low latency). Absolute cost is low.

DIMM does the same on the opposite side despite costs being high. That's because the DDR4 interface and abandoning the 4K addressing lowers latency immensely. High cost is matched by high performance. It also allows for very cool future computers to happen. Which you can't say for anything else.

SSDs are the worst, and a temporary solution for rich enthusiasts. On client workloads, sequential throughput plays a part too. SSDs are also not used in compute which significantly reduces the relevance of bandwidth and latency in performance, unlike DIMMs. Load times are limited by mostly software and ecosystem limitations. High storage requirements mean absolute costs are high.

 

[cbn]

Software that would allow caching NVMe NAND would help a lot IMO. (I know you mentioned this in the past as well)

Then a person doesn't need so much Optane if only the small files are being cached (PCIe 3.0 x 2 is just as good as PCIe 3.0 x 4 for 4K QD1 read).

It's much harder to make it work in practice because caching itself has an overhead. This is why ideally the gap between the cache and the cached drive has to be big.

Okay, so maybe not 3D TLC NVMe....but 3D QLC NVMe? (Which should have slower 4K QD1 Read than 3D TLC NVMe).

 

[IntelUser2000]

Okay, so maybe not 3D TLC NVMe....but 3D QLC NVMe? (Which should have slower 4K QD1 Read than 3D TLC NVMe).

It could work, but we don't have conclusive evidence, nor the ability to test it out until they allow NVMe caching. While the Optane Memory application is more integrated than the SRT version, the limitation is still software running on background, and NVMe protocol. If its on the drive the interface can be optimized for its purpose from the ground up.

Sequential speeds are higher on the NVMe drives than on the Optane Memory, and that cuts into performance advantage. It certainly plays a non-negligible part looking at 800P reviews. There's also the matter of Optane's latency being too low to be really used in reality due to limitations, like in software. Compared to a NAND SSD drive, NVME drives' QD1 IOPS and latency is 50-60% better due to the protocol changes. So going from SATA to NVMe may just be enough to get most of the advantage and anything beyond that mostly wasted.

Of course, 3D XPoint devices have other advantages, but its more subtle, like consistency that doesn't require any maintenance like TRIM, watching for capacity, or worrying about buffers being too small for the workload.

 

[cbn]

The best is eventually if they decide to use Optane dies to replace DRAM dies on future NAND SSDs. While it'll be slower, the capacity can be 5-10x larger. That allows eliminating software overhead and purpose tuning the firmware which you can't do with software schemes. I thought something like a future 770p Intel TLC SSD using Optane would be a great idea.

While the Optane Memory application is more integrated than the SRT version, the limitation is still software running on background, and NVMe protocol. If its on the drive the interface can be optimized for its purpose from the ground up.

CNEX has a controller being developed that can handle a variety of PMEMs:

https://www.tomshardware.com/news/cnex-labs-3d-xpoint-controller,32463.html

However, CNEX shared details of its SSD controllers and the software and hardware architecture that it designed to operate with any persistent memory, such as NAND, 3D XPoint, ReRAM and PCM, et al.

If it can use two different types of media (on the same drive) that would be very interesting and solve problems (I think) as the NAND manufacturers push for capacity. (Then as you noted the firmware could be precisely matched (and tuned) to the exact amounts of media present and their capabilities)

In fact, I was thinking if a future lithography shrunk 3D QLC (or maybe 5 bit per cell NAND) makes the 4K QD1 Read even worse than current 3D QLC then having some 3DXPoint on the SSD would be good idea to regain the lost 4K QD1 Read. Maybe even use MLC Optane*?

But then again, I got to thinking a small amount of 3D SLC could also work for that purpose?

*I am assuming future generations of 3DXPoint will be faster than today's 3DXPoint. (So as IMFT push the 3DXPoint media harder for DDR interface what eventually becomes MLC will be fine for NVMe)

 

[nosirrahx]

The 800P is almost good. You have to run 2 of them in RAID to get reasonable capacity and for the price, its hardly worth it.

I gave 2 of them a go in my NUC8i7HVK and the performance is pretty amazing. RAID 0 does not cripple 4KQ1T1 on Optane like it does on other storage types.

The M.2 drive on the way that don't fit into 2280, probably a mistake. I think they could get away with a 2280 with double the capacity and speed of the 800P, call them 850P or something.

Imagine the above benchmark (speed and capacity) for a single 2280 M.2 Optane SSD, I think that would have been a better move.

 

[nosirrahx]

P.S. Would like to see caching of NVMe NAND RAID arrays as well. (Maybe with Optane DIMMs on the horizon this will come fairly soon.)

Picture that as a RAID card. 4 M.2 NVMe ports and a Optane DIMM port. In a single 16X card you could have some amazing capacity with Optane caching all of the stuff that typically chokes the life out of RAID performance.

You get the added bonus of non-volatile cache. Power on and the stuff you need first is already there. You could also get away with no integrated BBU.

 

[William Gaatjes]

There is something i am wondering about. I hope i do not get burned at the stake by the obvious people for asking this, but 3dxpoint and therefore optane drives shine in a high number of iops yes ?
But are some of the tests not distorted because of using fully patched against meltdown and spectre cpu's from Intel ?
Making the optane drives look worse than they are ?
Because the patched cpus have lower iop counts in general ?
Or are the reviewers taking the lower performance during ideal performance tests in account ?
Have some reviewers done tests with optane drives on unpatched intel cpus and patched intel cpus ?
How about running optane drives on AMD cpu based systems ?
I just want to make sure that when i see a review that the maximum amount of performance is obtained during the performance tests because i am very interested in how 3dxpoint will develop itself. I hope it will take off and that the tech becomes licensable by as many manufacturers as possible. Increasing the development and lowering the costs.

 

[nosirrahx]

There is something i am wondering about. I hope i do not get burned at the stake by the obvious people for asking this, but 3dxpoint and therefore optane drives shine in a high number of iops yes ?
But are some of the tests not distorted because of using fully patched against meltdown and spectre cpu's from Intel ?
Making the optane drives look worse than they are ?
Because the patched cpus have lower iop counts in general ?
Or are the reviewers taking the lower performance during ideal performance tests in account ?
Have some reviewers done tests with optane drives on unpatched intel cpus and patched intel cpus ?
How about running optane drives on AMD cpu based systems ?
I just want to make sure that when i see a review that the maximum amount of performance is obtained during the perofrmance tests because i am very interested in how 3dxpoint will develop itself.

Optane takes a massive hit from these patches (VROC does as well), I did the testing that confirms this.

The left bench is the 905P, the right is 4 900P in VROC 0.

The top pic is old BIOS and 1709 without updates.
The middle pic is new BIOS and 1803 + updates.
The bottom pic is new BIOS + 1803 + updates but with mitigation disabled.

The 4KQ1T1 numbers really take one hell of a hit. In particular comparing the VROC 0 numbers with protection enabled and disabled reveals a 33% drop in performance in terms of 4KQ1T1.

 

[William Gaatjes]

Optane takes a massive hit from these patches (VROC does as well), I did the testing that confirms this.

The left bench is the 905P, the right is 4 900P in VROC 0.

The top pic is old BIOS and 1709 without updates.
The middle pic is new BIOS and 1803 + updates.
The bottom pic is new BIOS + 1803 + updates but with mitigation disabled.

The 4KQ1T1 numbers really take one hell of a hit. In particular comparing the VROC 0 numbers with protection enabled and disabled reveals a 33% drop in performance in terms of 4KQ1T1.

Ah thank you.
But the 4x 900p in VROC is something that a video editor would use, yes ?
That is a bit extreme but those numbers in general are incredibly high.
How would this result in real world numbers for a user having a single optane drive ?
With the all the patches enabled does this result in noticable performance decrease or is it something i would only notice with benchmark tests ?

I had to look it up what VROC means :
https://www.intel.com/content/www/us/en/support/articles/000024550/memory-and-storage.html

 

[William Gaatjes]

To understand the speed difference, i downloaded crystal disk mark 6.0.1. And i did the same tests with the MLC flash drive that i have , the SX8000 512GB.
But i am very happy with the m2 ssd i have.
Although, the que depths are different, did i overlook a setting or is this drive specific or technology specific ?


------------------READ [MB/s]------ WRITE[MB/s]
Seq Q32T1 = 2406.1 MB/s - 936.3 MB/s
4KiB Q8T8 = 659.4MB/s - 744.5 MB/s
4KiB Q32T1 = 383.2MB/s - 185.4 MB/s
4KiB Q1T1 = 36.93MB/s - 124.8 MB/s.

3dxpoint sure is a lot faster than flash based drives.


edit:
off topic for a second...
as a side note, the samsung 960 pro 512GB is about 30% faster than the SX8000 512GB but also costs 30% more.
on topic:
I am going to do a lot more reading on 3dxpoint memory developments...

 

[Cerb]

The best is eventually if they decide to use Optane dies to replace DRAM dies on future NAND SSDs. While it'll be slower, the capacity can be 5-10x larger. That allows eliminating software overhead and purpose tuning the firmware which you can't do with software schemes. I thought something like a future 770p Intel TLC SSD using Optane would be a great idea.

It would make it easier to implement power failure recovery, at least on consumer drives and lower-end server arrays, too (maybe not good enough for a big remote SAN, but plenty good enough for a local array, or NAS).

If they can get the power usage and latency down (from what little I've read of those problems, it looks like that's going to happen in newer generations of it), and come up with better controllers (or they could even integrate common interface controllers in their larger dies, or packages, and make that easier), I could see that becoming fairly popular. While slower than DRAM for reading, the ability to use the Xpoint as a write cache, in addition to holding mapping tables, could make up for it in realistic workloads, plus it could be cheaper than DRAM for large SSDs.

 

[Cerb]

I just want to make sure that when i see a review that the maximum amount of performance is obtained during the performance tests because i am very interested in how 3dxpoint will develop itself. I hope it will take off and that the tech becomes licensable by as many manufacturers as possible. Increasing the development and lowering the costs.

If Sandisk is successful at making a competitive product out of RRAM, that may not be necessary. We might see very different technologies competing, instead.

 

[nosirrahx]

But the 4x 900p in VROC is something that a video editor would use, yes ?

4X VROC Optane would be for use cases where you want much better sequential speed for massive files and far more space than a single drive would allow for.

Video editing would be a use case for this.

How would this result in real world numbers for a user having a single optane drive ?

Aside from the massive sequential boost, not much. There is actually a FAR better way to achieve huge capacity with Optane speed:

That is a 860 EVO using a 900P Optane drive as cache. This is expensive but the capacity and performance is unmatched for any single drive solution.

With the all the patches enabled does this result in noticable performance decrease or is it something i would only notice with benchmark tests ?

This totally depends on what you are doing. The higher queue depth and threads climb and/or the more sequential the data is the less you will feel it. If you were to copy a massive number of small files the performance delta would be very noticeable.

 

[William Gaatjes]

4X VROC Optane would be for use cases where you want much better sequential speed for massive files and far more space than a single drive would allow for.

Video editing would be a use case for this.



Aside from the massive sequential boost, not much. There is actually a FAR better way to achieve huge capacity with Optane speed:

That is a 860 EVO using a 900P Optane drive as cache. This is expensive but the capacity and performance is unmatched for any single drive solution.



This totally depends on what you are doing. The higher queue depth and threads climb and/or the more sequential the data is the less you will feel it. If you were to copy a massive number of small files the performance delta would be very noticeable.

I see. I read about that but wondered if it is true.
In the future it would be a nice upgrade if i would buy an m2 3dxpoint drive or pcie drive.
I do need another motherboard for that.
2019 is perhaps going to be very exiting.

 

[nosirrahx]

What I would like to see is a double sided M.2 drive with a small Optane cache + Optane cache controller on one side and 2TB+ of NAND on the other. It would be nice to have the impressive sequential speed of NAND and impressive everything else of Optane on a single package where you would not need to configure anything in the OS, its all handled by the drive controller itself. Properly implemented the OS and motherboard would not be aware of anything unusual, it would just look like a typical NVMe SSD.

 

[William Gaatjes]

What I would like to see is a double sided M.2 drive with a small Optane cache + Optane cache controller on one side and 2TB+ of NAND on the other. It would be nice to have the impressive sequential speed of NAND and impressive everything else of Optane on a single package where you would not need to configure anything in the OS, its all handled by the drive controller itself. Properly implemented the OS and motherboard would not be aware of anything unusual, it would just look like a typical NVMe SSD.

Yes, an flash based ssd with optane cache would indeed be nice. When there is a power loss, there is no worry as with dram cache ssd drives to write the dram contents to flash before the supply voltage drop below the minimum level. The big advantage of 3dxpoint vs dram. I agree. A controller that does it all is much better than os intervention. A transparent ssd device.