Samsung 960 PRO m.2 NVMe and CPU spikes

[Korporativ]

I'm not sure if this is really the right forum or not for this, honestly. Apologies if so.

Thus far I've root caused sporadic CPU spikes down to the m.2 drive itself.

I recently built a new system using a Ryzen 1800X, and MSI X370 Titanium motherboard. My issue is that when loading an application for the first time, or something caching (like streaming video) to the drive, occasionally all 8 cores and 16 threads shoot up to 100% utilization. Shortly thereafter, they drop back down to 5-10%.

I noticed this initially while watching a video in Firefox. My system fans would all ramp up considerably. After debugging for a while, and trying multiple options, I decided to disable disk caching. Lo and behold, there are no longer 100% CPU spikes when streaming videos. I still get multiple 100% spikes when the harddrive gets an initial hit for data.

Generally not an issue except when in a game. The occasional texture loading hiccups occur when I first load, but once everything is up... it settles down.

I've installed the latest Samsung NVMe drivers. I've also gone through and updated the firmware on the m.2 drive. I've played around with bios settings, ensured that all the standard "SSD Settings" are optimized in Windows 10. Still I get the spikes.

Using Acronis, I've cloned the contents of the NVMe drive to a Samsung 850 Pro drive, and the CPU utilization hit doesn't happen when running with that as my primary OS drive.

Is this to be expected with NVMe drives? My understanding is that processor overhead should be significantly less.

I've gone through all of my installed drivers to make sure they are the latest, and upgraded firmware on attached components to the latest. Am I missing something fundamental like a Samsung AHCI controller or something? I'm running out of ideas to troubleshoot.

 

[deustroop]

Have you run any tests on the device that would suggest whether it is performing the expected reads and writes ? That result--Samsung Magician has such a test-might indicate that the device is defective.

 

[BonzaiDuck]

How are you "caching" to the 960 Pro, and what are you caching?

Because I've never experimented to see what happens attempting to Ram-Cache movies and streamed content, I would not know what to expect in terms of the CPU usage. I never experimented, because my own inferences and "intel" I received elsewhere explains why you would least like to cache HD video content.

You shouldn't be caching video content, since you don't benefit from it. You should be able to play it directly from any SATA-II HDD without any adverse affect on video presentation. That's my own experience, and it's partially substantiated by others over in HTPC Forum.

You should only need the Samsung 960 (Pro, if model-specific) drivers.

What caching program are you using? Is it possible you have mis-configured it in some way? I'd eliminate everything that's unnecessary on a temporary basis until you determine what causes your CPU usage to spike.

I can't compare to your Ryzen because I'm only running a Skylake 6700K @ 4.5Ghz (my clock setting at the moment). I don't even keep track of how much Ryzen surpasses Skylake. But I do both Ram-caching and SSD caching with PRimo-Cache, and I don't have any spikes. Right now, I'm doing a "selective-full" scan of all storage on my Skylake, and the highest CPU usage may climb to 45%, but no higher.

Since the symptom shows itself in CPU usage, I'd look at as many causes that remain after you disable any complications like caching.

 

[BonzaiDuck]

And off the top of my head, thinking of jforce's remarks, I'd look at DPCs through some utility. I think there's one named "Latency-Checker." It can help you diagnose conflicts between drivers and so forth.

Sammy Magician has indicated to me about twice in six months that I needed to update my Intel controller driver. Your mobo and processor could be new enough that you might also examine that angle. Look to see that you've installed the latest drivers. And there must be more advice forthcoming from our colleagues here.

 

[Korporativ]

Have you run any tests on the device that would suggest whether it is performing the expected reads and writes ? That result--Samsung Magician has such a test-might indicate that the device is defective.

Solid suggestion, Jforce. I ran through the Performance Benchmark in Samsung Magician initially, and while the results were significantly lower than should be expected, they weren't abysmal. 100% CPU usage across all cores for the duration of the test as I've come to expect.

Sequential Read: 1,833 MB/s
Sequential Write: 1,571
Random Read: 172,632 IOPS
Random Write: 133,847 IOPS

As well, System Compatibility shows a VID 1022 caution stating that some functions of Magician cannot be run. "If multiple iterations of Read and Write are performed, RAPID mode may become inactive due to system internal errors on some of the AMD / AsMedia Controller or Driver."

I've disabled all of the AsMedia drivers, and uninstalled the AsMedia driver suite itself.

 

[Korporativ]

How are you "caching" to the 960 Pro, and what are you caching?

Because I've never experimented to see what happens attempting to Ram-Cache movies and streamed content, I would not know what to expect in terms of the CPU usage. I never experimented, because my own inferences and "intel" I received elsewhere explains why you would least like to cache HD video content.

You shouldn't be caching video content, since you don't benefit from it. You should be able to play it directly from any SATA-II HDD without any adverse affect on video presentation. That's my own experience, and it's partially substantiated by others over in HTPC Forum.

You should only need the Samsung 960 (Pro, if model-specific) drivers.

What caching program are you using? Is it possible you have mis-configured it in some way? I'd eliminate everything that's unnecessary on a temporary basis until you determine what causes your CPU usage to spike.

I can't compare to your Ryzen because I'm only running a Skylake 6700K @ 4.5Ghz (my clock setting at the moment). I don't even keep track of how much Ryzen surpasses Skylake. But I do both Ram-caching and SSD caching with PRimo-Cache, and I don't have any spikes. Right now, I'm doing a "selective-full" scan of all storage on my Skylake, and the highest CPU usage may climb to 45%, but no higher.

Since the symptom shows itself in CPU usage, I'd look at as many causes that remain after you disable any complications like caching.

Bonzai,

Sorry for the confusion. I'm not using anything specific to enable caching... simply run of the mill Firefox caching. I set browser.cache.disk.enable to false in about:config, and the CPU spikes from streaming video vanish. I tried disabling the page file in the hopes of minimizing issues. No discernible difference there.

I can predictably reproduce the issue on the drive by opening pretty much any program from C: (which is my m.2 drive)

 

[deustroop]

Solid suggestion, Jforce. I ran through the Performance Benchmark in Samsung Magician initially, and while the results were significantly lower than should be expected, they weren't abysmal. 100% CPU usage across all cores for the duration of the test as I've come to expect.

Sequential Read: 1,833 MB/s
Sequential Write: 1,571
Random Read: 172,632 IOPS
Random Write: 133,847 IOPS

As well, System Compatibility shows a VID 1022 caution stating that some functions of Magician cannot be run. "If multiple iterations of Read and Write are performed, RAPID mode may become inactive due to system internal errors on some of the AMD / AsMedia Controller or Driver."

I've disabled all of the AsMedia drivers, and uninstalled the AsMedia driver suite itself.

I think the read and write ops are way off.
Check out the VID 1022 thing.
Is rapid mode engaged ? This is what it does:

This optional, software-based solution commandeers a portion of system memory for use as a separate drive cache.

What is the system memory-use as the problem arises ?

 

[BonzaiDuck]

I'm more confused about the OP's problem at this point than any of the respondents. But just to clarify about something I do know for sure: the 960 Pro does not work with RAPID. There's no . . freakin' . . . RAPID mode in Magician for the 960 Pro, no for the EVO I would surely bet.

 

[Korporativ]

I'm more confused about the OP's problem at this point than any of the respondents. But just to clarify about something I do know for sure: the 960 Pro does not work with RAPID. There's no . . freakin' . . . RAPID mode in Magician for the 960 Pro, no for the EVO I would surely bet.

Bonzai,

I'm not trying to get RAPID working on the 960 PRO, that is just the Samsung Magician error that pops up to let me know that RAPID will potentially conflict with some of the existing drivers on the system for my 850 PRO in the system.

My issue is a CPU utilization spike whenever the drive is initially accessed by a program. I open Firefox, I hit a 100% utilization spike across all 8 cores and 16 threads. I open IE/Chrome, and get the same response. I open something like a blizzard or origin launcher, and I get multiple 100% CPU utilization spikes, again, across all 8 cores.

Once programs are up and in a steady state, I generally have less than 15% CPU utilization. The first exception being streaming video if the browser is caching to disk. If I disable disk cache in the browser... everything levels out, and I see at most 20% CPU utilization.

The other strange point here is that when seeing high CPU utilization, I maintain less than 25% memory commit, and below 50% disk utilization.

Not sure what it would add to the conversation or troubleshooting, but loading a virtual machine sees the initial 100% CPU utilization as I'd expect, but subsequent loads within the guest OS do not translate to CPU spike behaviors in the host OS. This leads me to believe it is a driver / chipset issue more than storage itself.

Per an earlier suggestion, I loaded LatencyMon, and didn't see anything that looked overly terrible... any tips on root causing DPC latency issues with that tool?

 

[TemjinGold]

OP: Not sure if this helps but I just tried launching stuff with Task Manager open and CPU went from 3% to at most 10% (usually 4-5%). Tried with a bunch of different programs I haven't used in a while. Not seeing what you're seeing...

 

[deustroop]

The board has something fancy called Turbo M.2 . Is Turbo working properly ?
The board also boasts 2x M2 slots. Try the other one. M2_2 slot supports PCIe 2.0 x4 (approx 2GB/s). If this is in use, PCIe 2.0 may account for the lower than expected read/writes but not the spikeage. Note the caution in the manual about using the shielded slot by first removing the rubber protector.

 

[Korporativ]

Deus,

Any idea how I'd know if the Turbo was working? I didn't notice anything related in the bios, or the MSI apps installed.

I've tried both slots on the board both with and without the armor. I removed the rubber stopper from the shield as well, but at most I'd think that just prevents proper contact with the thermal material, and perhaps a bit of unwanted mechanical stress.

Pretty positive it is a driver issue after having resorted to a windows boot disk. No issues running the drive from an external windows boot disk or installation on the drive itself. I haven't checked speeds under this configuration yet.

Anyone have tips on tracking down bad AHCI/Storage drivers to resolve? I'll do another clean install, but nothing to keep me from reinstalling the bad driver and shooting myself in the foot again.

 

[deustroop]

Ok I don't know about the workings of turbo but if there is nothing in the bios, it's just a gimmick.
If the samsung nvme driver is version 2.2 and it logs into device manager under storage controllers, then the driver is the proper one so long as UEFI also engages AHCI mode, which Magician can confirm. There was some debate here whether the Intel or MS version of AHCI was better but I would try both .
Finally as a matter of interest, I cannot figure how you are running the device when you say you are " running the drive from an external windows boot disk or installation on the drive itself."

 

[Korporativ]

So... "problem" ultimately solved. A combination of Task Manager reporting incorrectly, broken MSI gimmicks, and likely a healthy dose of PEBCAK sprinkled in.

Out of pure chance, I had HWMonitor open while debugging, and noticed that Maximum reported CPU usage through a smattering of tests was 34% despite multiple 100% usage spikes being shown in task manager. Essentially, the reported CPU usage was a false alarm.

Deus's response regarding MSI gimmicks got me thinking. There really was quite a bit of gimmick bloat from MSI that have caused strange system behaviors in the past. While systematically uninstalling MSI related items, I checked inside MSI "x-boost". Under "Advance", there was an option for "Storage Boost". Possibly the afore mentioned m.2. Turbo? I disabled it. Lo and behold, Task Manager now more or less agrees with HWMonitor. No more fictitious spikes reported in Task Manager. Without a reboot, even.

Finally, with MSI "Storage Boost" disabled, my 960 PRO is getting results more in line with reported specs:
3,450 MB/s Sequential Read
2,015 MB/s Sequential Write

Hopefully chasing things down odd rabbit holes serves to help save others some time.

A big thank you to everyone who jumped in to assist! As always, Anandtech forums have the absolute best community.

 

[deustroop]

Ok,
The drive specs are excellent and better than mine. What app did you use? Is it Magician ? I understand queue depth plays a role. Do you know the figure for the app ?

 

[Korporativ]

Deus, I used Samsung Magician to pull the performance test result data. What do you mean by the figure for the app?

 

[deustroop]

Deus, I used Samsung Magician to pull the performance test result data. What do you mean by the figure for the app?

Ok, The spec sheet for the 512 GB drive uses CrystalDiskMark for the seq data. I get within that range using that app, the same range as reflected in your Magician score . But the Magician score I get for for seq transfer here is lower, ~ 2.2 GB/s. So I wonder why my Magician score for seq transfer is low compared to yours. Perhaps you could try the CDM app and post your read/write scores / MB/s using the Seq Q32T1 and the Seq, i.e., default settings.
Much obliged.

 

[Korporativ]

Deus, getting slightly lower scores on the 960 in CDM as compared to magician. Random IOPS are significantly lower in CDM as compared with magician. I assume that means magician is a bit bias / gimmicky?

-----------------------------------------------------------------------
CrystalDiskMark 5.2.1 x64 (C) 2007-2017 hiyohiyo
Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

Sequential Read (Q= 32,T= 1) : 3405.168 MB/s
Sequential Write (Q= 32,T= 1) : 1990.177 MB/s
Random Read 4KiB (Q= 32,T= 1) : 229.362 MB/s [ 55996.6 IOPS]
Random Write 4KiB (Q= 32,T= 1) : 203.072 MB/s [ 49578.1 IOPS]
Sequential Read (T= 1) : 2675.899 MB/s
Sequential Write (T= 1) : 1969.466 MB/s
Random Read 4KiB (Q= 1,T= 1) : 51.603 MB/s [ 12598.4 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 131.587 MB/s [ 32125.7 IOPS]

Test : 1024 MiB [C: 10.8% (51.4/476.4 GiB)] (x5) [Interval=5 sec]
Date : 2017/04/28 9:36:27
OS : Windows 10 Professional [10.0 Build 15063] (x64)

 

[deustroop]

NVM

 

[deustroop]

Deus, getting slightly lower scores on the 960 in CDM as compared to magician. Random IOPS are significantly lower in CDM as compared with magician. I assume that means magician is a bit bias / gimmicky?

Ok, I cannot explain the different scores between us. The test here shows

CrystalDiskMark 5.2.1 x64 (C) 2007-2017 hiyohiyo
Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

Sequential Read (Q= 32,T= 1) : 3130.050 MB/s
Sequential Write (Q= 32,T= 1) : 1936.161 MB/s
Random Read 4KiB (Q= 32,T= 1) : 642.295 MB/s [156810.3 IOPS]
Random Write 4KiB (Q= 32,T= 1) : 568.394 MB/s [138768.1 IOPS]
Sequential Read (T= 1) : 1996.255 MB/s
Sequential Write (T= 1) : 1835.989 MB/s
Random Read 4KiB (Q= 1,T= 1) : 53.672 MB/s [ 13103.5 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 232.118 MB/s [ 56669.4 IOPS]

Test : 1024 MiB [C: 10.0% (47.7/476.4 GiB)] (x5) [Interval=5 sec]
Date : 2017/04/28 15:46:12
OS : Windows 10 [10.0 Build 15063] (x64)

 

[Hail The Brain Slug]

-----------------------------------------------------------------------
CrystalDiskMark 5.2.1 x64 (C) 2007-2017 hiyohiyo
Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

Sequential Read (Q= 32,T= 1) : 3300.926 MB/s
Sequential Write (Q= 32,T= 1) : 2102.025 MB/s
Random Read 4KiB (Q= 32,T= 4) : 1436.650 MB/s [350744.6 IOPS]
Random Write 4KiB (Q= 32,T= 4) : 1204.464 MB/s [294058.6 IOPS]
Sequential Read (T= 1) : 2445.071 MB/s
Sequential Write (T= 1) : 2068.533 MB/s
Random Read 4KiB (Q= 1,T= 1) : 61.407 MB/s [ 14991.9 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 269.912 MB/s [ 65896.5 IOPS]

Test : 1024 MiB [C: 7.1% (57.1/800.0 GiB)] (x5) [Interval=5 sec]
Date : 2017/04/29 15:10:55
OS : Windows 10 Professional [10.0 Build 15063] (x64)


One of the changes from Magician 4 to Magician 5 is that they changed the random test to run at Q32T4 instead of Q32T1. They have stated that in order to get a correct read in CrystalDiskMark that you have to change to T4 for randoms.

Obviously the crystaldiskmark still isn't quite to rated specs, but it's much higher than T1.

Try re-running it and see if your Q32 randoms are still super low.

 

[deustroop]

--

One of the changes from Magician 4 to Magician 5 is that they changed the random test to run at Q32T4 instead of Q32T1. They have stated that in order to get a correct read in CrystalDiskMark that you have to change to T4 for randoms.

Obviously the crystaldiskmark still isn't quite to rated specs, but it's much higher than T1.

Try re-running it and see if your Q32 randoms are still super low.

Ok, those numbers do show better here too.

Sequential Read (Q= 32,T= 1) : 3207.548 MB/s
Sequential Write (Q= 32,T= 1) : 1956.200 MB/s
Random Read 4KiB (Q= 32,T= 4) : 1346.617 MB/s [328763.9 IOPS]
Random Write 4KiB (Q= 32,T= 4) : 1202.458 MB/s [293568.8 IOPS]
Sequential Read (T= 1) : 2082.453 MB/s
Sequential Write (T= 1) : 1841.279 MB/s
Random Read 4KiB (Q= 1,T= 1) : 53.228 MB/s [ 12995.1 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 232.002 MB/s [ 56641.1 IOPS]

Test : 1024 MiB [C: 10.0% (47.8/476.4 GiB)] (x3) [Interval=5 sec]
Date : 2017/04/29 20:12:09
OS : Windows 10 [10.0 Build 15063] (x64)

I wonder whether the extra data on the drive here results in slower performance numbers, (tho not likely one noticeable in real world output).?

 

[Korporativ]

Interesting... I see an improvement moving to 4 threads on the random as well, but still looks like about half of what most of you are seeing.

-----------------------------------------------------------------------
CrystalDiskMark 5.2.1 x64 (C) 2007-2017 hiyohiyo
Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

Sequential Read (Q= 32,T= 1) : 3443.791 MB/s
Sequential Write (Q= 32,T= 1) : 2006.215 MB/s
Random Read 4KiB (Q= 32,T= 4) : 583.834 MB/s [142537.6 IOPS]
Random Write 4KiB (Q= 32,T= 4) : 519.103 MB/s [126734.1 IOPS]
Sequential Read (T= 1) : 2616.376 MB/s
Sequential Write (T= 1) : 1966.212 MB/s
Random Read 4KiB (Q= 1,T= 1) : 52.487 MB/s [ 12814.2 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 134.635 MB/s [ 32869.9 IOPS]

Test : 1024 MiB [C: 10.9% (52.1/476.4 GiB)] (x5) [Interval=5 sec]
Date : 2017/04/30 6:28:09
OS : Windows 10 Professional [10.0 Build 15063] (x64)


What do you all have enabled / disabled in your windows 10 configuration for performance?
I've installed the latest samsung firmware, and the Samsung NVMe Controller for windows 10.
Write caching is enabled on the device, but write-cache buffer flushing is disabled.

I disabled indexing on the drive, and disabled superfetch within windows.

I believe that is the extent of tweaking I've gotten to on this new install.

Also - do you have page file enabled? And is the m.2 drive you have installed your primary disk?

 

[BonzaiDuck]

Interesting... I see an improvement moving to 4 threads on the random as well, but still looks like about half of what most of you are seeing.

-----------------------------------------------------------------------
CrystalDiskMark 5.2.1 x64 (C) 2007-2017 hiyohiyo
Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

Sequential Read (Q= 32,T= 1) : 3443.791 MB/s
Sequential Write (Q= 32,T= 1) : 2006.215 MB/s
Random Read 4KiB (Q= 32,T= 4) : 583.834 MB/s [142537.6 IOPS]
Random Write 4KiB (Q= 32,T= 4) : 519.103 MB/s [126734.1 IOPS]
Sequential Read (T= 1) : 2616.376 MB/s
Sequential Write (T= 1) : 1966.212 MB/s
Random Read 4KiB (Q= 1,T= 1) : 52.487 MB/s [ 12814.2 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 134.635 MB/s [ 32869.9 IOPS]

Test : 1024 MiB [C: 10.9% (52.1/476.4 GiB)] (x5) [Interval=5 sec]
Date : 2017/04/30 6:28:09
OS : Windows 10 Professional [10.0 Build 15063] (x64)


What do you all have enabled / disabled in your windows 10 configuration for performance?
I've installed the latest samsung firmware, and the Samsung NVMe Controller for windows 10.
Write caching is enabled on the device, but write-cache buffer flushing is disabled.

I disabled indexing on the drive, and disabled superfetch within windows.

I believe that is the extent of tweaking I've gotten to on this new install.

Also - do you have page file enabled? And is the m.2 drive you have installed your primary disk?

I forgot what I posted here earlier if at all. After my Build 1703 installation-crisis ended in (my) triumph, the Sammy 960 Pro driver had merely been repaired, and Magician uninstalled and then reinstalled to eliminate my problem without confirming the exact cause in a nexus of possible causes. Suddenly, I was getting about 3,400 seq-read instead of closer to 3,100 -- in benchmarks. The new benchmark followed before I thought about the manual tweaks I'd given to all past SSDs (SATA). I stuck my nose out on the web briefly.

Somewhere, I saw someone's recommendation that Magician's "optimization" feature was more "optimal" than manual tweaking. Of course, the usual thing we'd come to do routinely included disabling SuperFetch and Prefetch. My little web-foray was spurred by my initial intention to do just that. So I chose not to touch anything.

The Z170 motherboard (mine -- only perhaps the chipset of others here) gives 16 PCIe lanes from the CPU for slots "x16" #1 and #2. The x1 slots and the "x16" x4 slot have lanes provided by the chipset, communicating with the processor in an equivalent of x4. I'm wondering if the benchies would improve in any observable degree for moving my 960 Pro from the x4 slot to the #2 x16/x8 slot. There are obvious reasons for thinking so, especially if devices using part of 20 PCIe lanes compete in any way for the DMI link to the processor.

That would also eliminate for me the future option of adding another GTX 1070 of same make and model, but supposedly a single card is fully 4K-capable @ 60Hz, and the performance hit of running a single card at x8 instead of x16 is a mere 1% -- give or take.

 

[deustroop]

I do not have any tweaks applied. While my seq are low the randoms are not bad. Perhaps Xabanakfanatik has secrets ?