Small Optane drive as swap

[richaron]

Sorry if this is obvious and/or has been answered before, I'm just having trouble finding a concrete answer...

Do those small (32/64GB) Optane drives function as a regular NVMe M.2 SSD when outside their special cache Intel/Windows ecosystem? And from what I've gathered they do perform very well with regard to response times and random small accesses, so I'm guessing they would be significantly better than a regular NVMEe SSD for swap purposes. Would this be correct?

I'm fantasizing about a low power ARM Linux system like one of these, and I thought an Optane drive could give me a bit of flexibility to install a more full featured (read: bloated) distro and run a few more services than the 1/2GB onboard RAM could handle.

Note: Please address whether this setup is possible before telling me it's silly. I know it's silly. For the record I'm fully aware of the low performance nature of a dual core A53, the poor "value" of these small Optane drives, and I know the difference between a mPCIe and mPCIe v2. I'm not overly worried about being limited to ~500MB/s (minus overhead) because:
1) A PCIe SSD will free up the single SATA port for a spinner.
2) Limiting speed will be about the same as a SATA SSD.
3) ~500MB/s is actually huge for random small read/writes, and I can see Optane's big brother does relatively well in this area.

 

[TheELF]

For it to function as optane you need a kabylake cpu and a mobo/bios that supports it,it's not just the drive,it will still work as a normal NVMe M.2 far as I know but not with the extra speed.

 

[ArtForz]

Yes, they're standard PCIe3 x2 m.2 NVMe SSDs.
If you plan on using something that fast as a swap device on linux, a recent kernel (4.11+) could improve performance quite a bit (https://lwn.net/Articles/704478/ goes into the technical details).

 

[richaron]

Sweet, thanks guys.

Anyone have thoughts on the response time and random read/writes advantages? At least the response time thing seems huge... How would something like this run when the system was thrashing swap constantly? And am I right thinking 3d Xpoint would be way better then a standard NVMe SSD?

 

[Space Tyrant]

[Edit: the original test described below had a significant bug. Comment #11 describes the bug, the bug fix and other improvements made to the test, as well as the very different results.]



I saw the 16GB optane nvme drives at newegg for $25 and immediately ordered one to pop into my unused NVMe slot and test it as a swap drive. Evidently, $25 is my "impulse buy" threshold.

I set up a test program that allocates a 3GB array and randomly accesses pieces of it 75,000,000 times. While using a partition of my 3.5 year old Kingston 240 GB SATA SSD for swap space and running 8 concurrent copies of this test it completes in ~165 seconds.

When I installed the optane drive as a replacement for my SSD swap partition and ran the test, It completed in a disappointing ~235 to ~250 seconds -- more than 40% slower.

___Details___
Note that I tested with /sys/block/nvme0n1/queue/io_poll set to "1" as recommended for Optane drives as well as "0" and only saw a small difference between the two settings.

Also note that I am running a 4.10 Linux kernel which is somewhat slower in swap than newer kernels, but that effect isn't very important for this test -- it simply adds a constant latency penalty to all swap accesses.

My machine has 16 GB of memory and the test requires a total of 24 GB, therefore forcing approximately 33%, or ~200,000,000 of the memory accesses to use the swap partition.

___tl;dr___
The optane 16GB drive was substantially slower than my SATA SSD when used for a swap drive. It would be a (huge) improvement only if you're using an actual hard disk drive for swap. [Edit: see comment #11 for updated results]

 

[cbn]

Also note that I am running a 4.10 Linux kernel which is somewhat slower in swap than newer kernels, but that effect isn't very important for this test -- it simply adds a constant latency penalty to all swap accesses.

A quote from the link in post #3:

According to Chen, current kernels add about 15µs of overhead to every page fault that is satisfied by a read from a solid-state swap device. That, he says, is comparable to the amount of time it takes to actually read the data from the device. With the patches applied, that overhead drops to 4µs, a significant improvement. There have been no definitive comments on the patch set as of this writing, but it seems like the sort of improvement that the swap subsystem needs to work well with contemporary storage devices.

I would think 9us difference would have a far greater effect (proportionally) on Optane than it would on (MLC?) SATA? So it will be interesting to see what your results will be on Linux Mint 19 (which will get the same 4.15 kernel as Ubuntu 18.04).

Also remember Optane's strength is QD1 4K read. At high QD the advantage over NAND decreases quite a bit.

P.S. Would be interesting to also see your results with swappiness of 100 vs. the default of 60.

 

[IntelUser2000]

I would think 9us difference would have a far greater effect (proportionally) on Optane than it would on (MLC?) SATA? So it will be interesting to see what your results will be on Linux Mint 19 (which will get the same 4.15 kernel as Ubuntu 18.04).

Also remember Optane's strength is QD1 4K read. At high QD the advantage over NAND decreases quite a bit.

I was thinking another factor is involved.

Space Tyrant has said he was running 8 concurrent copies. Even if its QD1, it might be like having QD8.

Plus, the write performance of the 16GB is anemic at only 145MB/s. So the closer it gets to sequential or multiple queue depth, the advantage goes increasingly in favor of the SATA SSD.

 

[IntelUser2000]

Note: Please address whether this setup is possible before telling me it's silly. I know it's silly. For the record I'm fully aware of the low performance nature of a dual core A53, the poor "value" of these small Optane drives, and I know the difference between a mPCIe and mPCIe v2. I'm not overly worried about being limited to ~500MB/s (minus overhead) because:

The Optane Memory drives and the 800P uses the B+M key, which makes it physically compatible with both x4 PCIe and M.2 SATA devices. It seems to be the only x2 drives with that key, so you want to make sure the slot supports M.2 NVMe, not just SATA. Some slots only support M.2 SATA.

I would think 9us difference would have a far greater effect (proportionally) on Optane than it would on (MLC?) SATA?

11us actually.

11us will more than double the latency of the Optane Memory module. Still, it seems on his system everything that can go wrong went wrong. If its just accessing files, than it should depend on read performance, which even the 16GB has more than enough to beat the SATA.

Am I wrong and maybe its mostly writes? The low cap write of the Optane Memory then is disadvantaged against the DRAM-buffered writes of the SATA SSD.

 

[Space Tyrant]

Am I wrong and maybe its mostly writes? The low cap write of the Optane Memory then is disadvantaged against the DRAM-buffered writes of the SATA SSD.

Good guess; most accesses in the test included a write. So, most (swap) accesses were in fact two accesses; a read followed by a write back to the optane drive. Therefore, for general purpose swap access patterns, it was heavily biased to the write side -- and misrepresents how well the small optane drive would perform in 'general purpose' swap usage.

At the end of the random read/write pattern, it sequentially reads the entire array.

The test was intended to test random accesses but turned out to be much more of a test of random *write* accesses.

If I get time today, I'll try to retest with a more appropriate mix of reads and writes.

 

[Space Tyrant]

A quote from the link in post #3:



I would think 9us difference would have a far greater effect (proportionally) on Optane than it would on (MLC?) SATA? So it will be interesting to see what your results will be on Linux Mint 19 (which will get the same 4.15 kernel as Ubuntu 18.04).

Also remember Optane's strength is QD1 4K read. At high QD the advantage over NAND decreases quite a bit.

P.S. Would be interesting to also see your results with swappiness of 100 vs. the default of 60.

I always run swappiness set to 1 -- and didn't change it for the test. In this case, swapping was forced anyway by the size of the 8 arrays exceeding physical memory by 50%.

I'm running kernel 4.10 because I experienced performance regressions with later kernels. I think my most recent experiment was with 4.14 where my worst case scenario was 16% in one particular benchmark. Geekbench was more representative of the degradation which I vaguely remember to be in the 2 or 3% range.

 

[Space Tyrant]

In updating the swapspace test code to add reads into the previously all write random operations, I found a significant bug. My PRNG was truncating 6 bits and causing the writes to not access the entire size of the arrays. This resulted in only the initial access of a block forcing a swap-in operation. Subsequent operations within that I/O block would probably never have to do another swap-in.

The obvious effect is that the test would run much faster than expected and with far fewer swap operations, for both SSD and Optane.

Fixes made:
1) fixed truncated bits in PRNG
2) added a 60 second sleep after allocating the array so that the *last* program to allocate its array would still be in memory when the *first* starts its random access pattern. Now, the copies with a quicker start should still be forced to use swapspace instead of quickly running in main memory before later programs fully allocate their arrays.
3) added reads at 80% of random accesses, random writes now at 20%.
4) increased the proportion of random accesses to average down the effects of the two full sequential passes described below.

At the start of the program there is a single sequential pass through the array which writes a zero to every cell and forces the array to actually be allocated. There is also a final sequential pass through the array reading every cell. After the increase in random reads and writes, the random writes are the equivalent of a single full pass and the random reads are equal to four passes.

In this test the Optane drive beat the SSD by a bit more than double. The averages were ~6 minutes for the Optane and ~13 minutes for the SSD.

Ideally the test should be a single program with a single large array to eliminate race conditions between the instances -- which add considerable variance to the results.

___tl;dr___
Debugged the test, increased proportion of random accesses, and changed the mix to 80% reads / 20% writes. The optane drive scores over twice as fast as the SSD in the debugged and extended test.

___Edit___
Note that this is the smallest Optane "memory" NVMe stick at 16 GB.

 

[cbn]

Anyone tried using the 16GB Optane (14.4GB available in Linux) with a portion of the capacity used for swap partition and rest of the capacity used for Bcache?

https://bcache.evilpiepirate.org/#index2h1

*For some reason the 16GB Optane only has 13.4GB capacity in Windows. (Where did the 1GB go?)