Question M.2 storage access lag on Z590 Asus RoG Strix E-Gaming mobo

[yacoub]

System:
Asus ROG STRIX Z590 E-Gaming WiFi
Core i7 11700
32GB Corsair (2x16) compatible DDR4-2132 RAM in the recommended slots 2 & 4 per the mobo manual
GeForce RTX 2060 Super in the top PCI-E slot
Samsung 980 Pro 2TB M.2 in the PCIe4.0 slot, uses default Microsoft driver (any day now, Samsung), over 1TB of free space remaining
Samsung 970 Evo Plus 2TB M.2 in the next M.2 slot which is PCIe 4.0 capable but this drive is PCIe3.0, uses Samsung's latest driver, over 1TB of free space remaining

Note: If you populate both M.2 PCIe 4.0 slots on this board, manual says they will both run at 4x and the GPU is cut to 8x from 16x.

Mobo is running latest firmware as of last month (April, 0707 I believe) and I believe I installed all the drivers for the various hardware and whatnot from Asus' website, including the Intel Management Engine, but not the Wifi drivers since I'm running on a wired LAN connection, nor the Bluetooth driver since I'm not using any BT devices, and I installed the Intel RST driver but not the Optane app (even though I'm not using RAID or SATA drives, I installed the RST driver to see if it would help somehow. It did not change the symptoms.

Issue:

When I pop open Windows Explorer and click into a folder with media files, be it videos or mp3s, sometimes, randomly, not to any pattern I can detect, instead of the contents displaying right away, nothing will appear while the system "thinks" and the green bar scrolls across the directory location bar as if the drive has to remember what-all is in the folder. It can take up to 20-30 seconds to display the contents. This happens even if I have it set to List view and normal alpha a-z sort order so it shouldn't have to read meta data on the files or do anything fancy with the sorting before displaying the contents to me.

I am coming from another Asus rig I build a couple years ago with Samsung 970 Evo and 970 Evo Plus drives that worked flawlessly. They never choked trying to open a folder in Windows Explorer and even the meta-data came right up when required.

I don't think it's a bad drive, as it happens on occasion with folders located on either drive. And I let Samsung Magician scan both drives for any bad cells and they came up 100% fine.

I am guessing this is either a driver issue or some sort of weird bandwidth issue (maybe even a bug?) with the BIOS or chipset driver, where running this particular configuration of a PCIe4.0 M.2 drive in the first 4.0 capable slot, a PCIe3.0 M.2 drive in the second 4.0 capable slot, and a GPU in the first PCIe GPU slot.

I haven't been able to find any other reports of similar issues out there, but maybe I'm not looking in the right places. Any help is appreciated.

The only other thing I've noticed is this system, particularly the CPU, runs quite a bit hotter than the previous one, but the M.2 SSDs are around the same temps as my old rig, hitting 50C-52C without any issues.

 

[Justinus]

This happens on my X99/5960x system with a 970 Pro as well as my X570/5950x system on both an SN850 and Rocket 4 Plus.

It happened for the last couple years on my 5960x box. I assumed it was something to do with spectre/meltdown mitigations hitting the I/O, so I experimented with reverting the microcode and fully disabling the software mitigations and it had no effect.

When I built my 5950x box in November, it did it right out the gate on both boot drives.

After speaking with someone who has 8700k and 9900k boxes that also do it, I think it's something with Windows. That 9900k even has an Optane 905P as the boot drive.

It happens on far too much completely different hardware to not be Windows.

 

[yacoub]

Oh wow, that is so strange! So I was just lucky my old system never did it? Thanks for the additional info that even AMD boxes do it, so it must be a Windows "thing". It was kind of scary the first time it happened, I thought the drive might be bad and I'd lost the contents of the folder.

 

[legolas_tk]

According Asus's ROG STRIX Z590 E-Gaming WiFi Storage specifications, M.2_1 slots is PCIE 4.0 NVME and recommend for M.2 that is solely PCIE 4.0 for optimum performance. I recommend to use M.2_2 or M.2_3 or M.2_4 slot for PCIE 3.0 drives for best performance. Please check that your bios is setup properly for M.2_3/4, go to BIOS - > Advanced -> Onboard Devices Configuration -> M.2_3/M.3_4 -> M.2_3 / M.2_4 + PCIEX16_1 and PCIEX16_2 (if exist)

Also, see if you can set the M.2 link to PCIE 3.0 to see if performance is better. Please install latest chipset drivers from Asus's download website and also the ME chipset. Please set High performance mode for power options.
You can also see if you can install latest Samsung drivers (if not RAID), https://s3.ap-northeast-2.amazonaws...281A78151C/Samsung_NVM_Express_Driver_3.3.exe

 

[yacoub]

According Asus's ROG STRIX Z590 E-Gaming WiFi Storage specifications, M.2_1 slots is PCIE 4.0 NVME and recommend for M.2 that is solely PCIE 4.0 for optimum performance. I recommend to use M.2_2 or M.2_3 or M.2_4 slot for PCIE 3.0 drives for best performance. Please check that your bios is setup properly for M.2_3/4, go to BIOS - > Advanced -> Onboard Devices Configuration -> M.2_3/M.3_4 -> M.2_3 / M.2_4 + PCIEX16_1 and PCIEX16_2 (if exist)

Also, see if you can set the M.2 link to PCIE 3.0 to see if performance is better. Please install latest chipset drivers from Asus's download website and also the ME chipset. Please set High performance mode for power options.
You can also see if you can install latest Samsung drivers (if not RAID), https://s3.ap-northeast-2.amazonaws...281A78151C/Samsung_NVM_Express_Driver_3.3.exe

Hey there. As noted in my original post, my M2.1 slot is populated with a PCIe 4.0 drive, so we're good there. M2.2 is populated with a PCEi 3.0 drive, also as designed.
I'll check the BIOS settings you suggested.
I have the latest Samsung driver installed and the 970 Evo Plus uses it, however the 980 Pro does not use it, it has to use the default Windows driver (this is a known thing from Samsung and we're all waiting for them to release a driver that includes the 980 Pro as it should improve performance). The issue I'm describing affects both drives though, so it's not likely related to one driver or the other.
I have the ME chipset installed.
I have Power mode in Windows set to High Performance.
Cheers.

 

[legolas_tk]

Hey, what version of ME are you using? Are you using the ME driver from Asus or Intels? Thanks

 

[yacoub]

Hey, what version of ME are you using? Are you using the ME driver from Asus or Intels? Thanks

I installed it via "Intel MEUpdateTool_15.0.21.1549_T" and "DRV_MEI_Intel_Cons_RKL_TP_W10_64_V204710001039_20210127R"

The version it tells me in Device Manager is "2047.100.0.1039" provider is Intel and date is 11/18/2020

If there's a newer version, it's not clear to me where that is or how to update it further. Through Device Mgr, if I choose Update Driver and point it to the folder where I extracted the MEUpdate Tool, it says I have the latest version installed.

 

[legolas_tk]

Hello Yacoub,

Did you open all the folders for a long time? Or does the problem occur only when the folder under the system disk is opened?
for example, first connect only one M.2 SSD and graphics card, so that the graphics card will run at X16 speed, then reinstall a clean OS and the latest driver, and try again after the installation is complete
In addition, from the your description, it looks like it is caused by the files in the folder. You can try to replace the files in the folder and try again

 

[yacoub]

Hello Yacoub,

Did you open all the folders for a long time? Or does the problem occur only when the folder under the system disk is opened?

Again, it happens to folders housed on either SSD. Not just the drive with the OS installed.

for example, first connect only one M.2 SSD and graphics card, so that the graphics card will run at X16 speed, then reinstall a clean OS and the latest driver, and try again after the installation is complete
In addition, from the your description, it looks like it is caused by the files in the folder. You can try to replace the files in the folder and try again

Okay now you're starting to sound like tech support "reboot the cable modem", lol. I am certainly not going to reinstall the OS and start over. Everything is working fine aside from something with the motherboard/chipset/drivers for the SSDs that is causing it to randomly choke when trying to read the contents of folders on either SSD. I'm sure a BIOS setting or a driver or a chipset update will fix it, unfortunately it sounds like others who experience this issue have no solution either.
I wonder if it's related to the 980 Pro using Windows drivers because Samsung has failed to deliver a driver that works for them, and that's causing the entire system to suffer when the buffer is full on the 980.

 

[legolas_tk]

Sorry, I didn't want to sound like tech support.

It is possible that it could be Samsung drive. Did you talk to Samsung to see if they have solution for that?

In technical terms,
The file resides in the file system in OS[fat32/NTFS] which stores in a physical disk i.e 980 Pro.
So, it has to be something with drive talking to CPU, because the M.2 has direct bus to the CPU now.

Can you provide task manager -> Performance -> Disks 0 (drive in question)? CPU? when you access folder please. Very curious.

 

[blckgrffn]

I have issues like this when I have an external 14tb hard drive connected to my system. Every once in a while, a big pause. Finally just disconnected except for when in use. Also, browsing the directory on the drive with about 100 large files in it, it would lag hard. Sometimes it would never really be usable. This is just with the “detailed” view and the number you of files should be low enough the drive layout should essentially be cached to the drive buffer.

But it was *also* a shared directory and accessing it remotely always worked quickly and without pause.

That there might some glitch occurring with the native file explorer seems reasonable thinking to me, but resolving it?!? Who knows. Sorry whatever the bug is but you.

And not to sound like tech support, if you copy the files off and nuke it with a fresh install of windows and put the files back, does it make a difference? PITA I know, but for science!

 

[kschendel]

How many files in the directory, roughly? hundreds? hundreds of thousands?

I very very much doubt that this is a hardware thing, unless the drive isn't responding at all until multiple retries are done. That sort of thing ought to show up somewhere in the system logs. My guess is some sort of windows strangeness, maybe some sort of anti-virus kicking in. A delay of 20-30 seconds simply can't be hardware unless it's not responding at all for a while - or unless the CPU is totally maxed out doing system tasks.

 

[Justinus]

How many files in the directory, roughly? hundreds? hundreds of thousands?

I very very much doubt that this is a hardware thing, unless the drive isn't responding at all until multiple retries are done. That sort of thing ought to show up somewhere in the system logs. My guess is some sort of windows strangeness, maybe some sort of anti-virus kicking in. A delay of 20-30 seconds simply can't be hardware unless it's not responding at all for a while - or unless the CPU is totally maxed out doing system tasks.

My guess is it's windows defender related.

 

[yacoub]

The folders can have as little as a couple dozen files in them, typically media files.
Here's what it looks like while it's waiting to populate Explorer with the contents.


I think I MAY have resolved it after weeding through the BIOS settings and tweaking a bunch of them.

Settings I changed included:
M2_1 Link Speed to Gen4 (from Auto)
PCIe Speed to Gen3 (from Auto)
PCI Express Clock Gating to Disabled (from Enabled)
Re-Size BAR Support to Enabled (from Disabled)
Native ASPM to Enabled (from Disabled)
PCI Express Native Power Management to Enabled (from Disabled)
SVID Behavior to Typical Scenario (from Intel's Fail Safe)
AI Overclock Tuner to Manual (from XMP II) which adjusted a bunch of DRAM settings
BLCK Frequency to Auto (from 100)
And I disabled the entire SATA Controller set since I'm not using any of them

Regarding the changes to SVID and AI Overclock Tuner: I am not overclocking nor trying to overclock anything. The BIOS is a bit confusing with tons of OC related features and I tried to disable them all when I first built the rig, but those two I changed now based on reading what those settings and the options for them mean.

Anyway, I'll let you know if the slow folder loads reappears but at the moment it seems to be loading them quickly... though that could also be because I just rebooted and everything's fresh.

 

[Justinus]

If it persists and you can easily reproduce it, try temporarily disabling every windows defender protection and seeing if it stops.

My memory is hazy but I think I may have experimented with that and found success. Unfortunately I rely on defender/windows security so I leave it enabled.

 

[yacoub]

I think it's gone! Hasn't happened again, so far.

 

[legolas_tk]

That is good news

 

[Scarpozzi]

I think it's gone! Hasn't happened again, so far.

I was thinking indexing service....but not sure. Windows does a lot of stuff out of the box these days. Hopefully it'll stay fixed.

 

[yacoub]

So it's starting to do it again today. No identifiable reason. It's not quite as slow to do it as it was previously but it's definitely taking time to figure out the folder contents and read their attributes. Maybe there are some Win10 indexing features I should be disabling for SSDs? It's honestly been so long since I did a fresh Win10 install, I had forgotten about that sort of stuff but now I vaguely remember there were some indexing things that are supposed to be shut off to preserve SSD lifespan and performance, right?

edit: The drives were set to index file contents in addition to properties. I am changing that right now. It's under General tab when doing Properties on the drive. Unchecking “Allow files on this drive to have contents indexed in addition to file properties to disable disk indexing".

Still looking into what I might do to turn off super fetch or indexing functions to see if that helps.

edit2: Apparently Superfetch is now called SysMain.

 

[Scarpozzi]

So it's starting to do it again today. No identifiable reason. It's not quite as slow to do it as it was previously but it's definitely taking time to figure out the folder contents and read their attributes. Maybe there are some Win10 indexing features I should be disabling for SSDs? It's honestly been so long since I did a fresh Win10 install, I had forgotten about that sort of stuff but now I vaguely remember there were some indexing things that are supposed to be shut off to preserve SSD lifespan and performance, right?

edit: The drives were set to index file contents in addition to properties. I am changing that right now. It's under General tab when doing Properties on the drive. Unchecking “Allow files on this drive to have contents indexed in addition to file properties to disable disk indexing".

Still looking into what I might do to turn off super fetch or indexing functions to see if that helps.

edit2: Apparently Superfetch is now called SysMain.

Was windows installed with those drives initially or did you clone/move your data?

SSD optimization in Windows 10

Let's get the most out of that speedy new SSD in your Puget Systems computer. Optimizing settings and configuration of the SSD can provide improved performance and reliability.

www.pugetsystems.com

I built my system for MSFS2020. I ended up giving it a pretty big page file and it has a ton of RAM. I believe though, the recommendation to have more of a page file will optimize windows, but not the SSD drive itself. Page files are like swap space in Linux....saving things to the hard drive in a dedicated partition (non-journaling partition). I really don't know how windows actually uses this space in the real world. The Red Hat Oracle servers I built back in the day would have swap match the available RAM, but that's likely excessive. Just lock it to the recommendation.

 

[yacoub]

Was windows installed with those drives initially or did you clone/move your data?

The drives were installed during the build before the OS was installed.

I built my system for MSFS2020. I ended up giving it a pretty big page file and it has a ton of RAM. I believe though, the recommendation to have more of a page file will optimize windows, but not the SSD drive itself. Page files are like swap space in Linux....saving things to the hard drive in a dedicated partition (non-journaling partition). I really don't know how windows actually uses this space in the real world. The Red Hat Oracle servers I built back in the day would have swap match the available RAM, but that's likely excessive. Just lock it to the recommendation.

I haven't messed with the page file yet as I wasn't sure what Min/Max sizes would really be appropriate for a system with 32GB of RAM.

 

[VirtualLarry]

I haven't messed with the page file yet as I wasn't sure what Min/Max sizes would really be appropriate for a system with 32GB of RAM.

Pagefile should be 2X-3X maximum physical RAM size, for best overall performance.

 

[acale75]

Maybe it's a driver problem? Ask your question here. Fernando is very knowledgeable when it comes to ssd and nvme drivers.

Forum - Win-Raid Forum (Drivers - Firmware - Modding)

Drivers, Firmware, Modding

www.win-raid.com

 

[Jimminy]

So it's starting to do it again today. No identifiable reason. It's not quite as slow to do it as it was previously but it's definitely taking time to figure out the folder contents and read their attributes. Maybe there are some Win10 indexing features I should be disabling for SSDs? It's honestly been so long since I did a fresh Win10 install, I had forgotten about that sort of stuff but now I vaguely remember there were some indexing things that are supposed to be shut off to preserve SSD lifespan and performance, right?

edit: The drives were set to index file contents in addition to properties. I am changing that right now. It's under General tab when doing Properties on the drive. Unchecking “Allow files on this drive to have contents indexed in addition to file properties to disable disk indexing".

Still looking into what I might do to turn off super fetch or indexing functions to see if that helps.

edit2: Apparently Superfetch is now called SysMain.

Thanks for mentioning this. I put windows on an SSD several years ago, and at the time I disabled indexing. But recently, I added a couple of nvme ssds (storage only), and I didn't even think about indexing. I saw your post and now have turned it off. I guess windows doesn't default to OFF even if an SSD is involved.

 

[yacoub]

So I still get the symptom and it does seem to get worse over time the longer the system has been up. Oddly, rebooting seems to start it over with no lag and it builds up eventually like buffers are getting full and not emptying properly in the RAM.
I'm noticing opening files in apps still sometimes has a similar pause like Windows Explorer does listing folder contents (eventually).

For example, image editing software, the software loads quickly but when trying to do something like open a file in it, it has a long pause doing that. Then it will run fine with no issues during the editing. Going to save the file may or may not have a pause. Winamp right now has a long pause when I click on an mp3 I want it to play.

Everything else on the system is responsive like normal, it's sort of like only certain calls to memory suffer this intermittent "thinking" lag and it's something that gets worse the longer the system has been on (days).

I wonder if it's some other BIOS setting I don't have right. The BIOS is genuinely confusing when it comes to overclocking settings. I just want everything at their defaults, no overclocking, no nonsense, and Asus makes that so difficult to do because there's no clear master OFF setting for all the stupid AI stuff, the way things are laid out or labeled. I probably have some more settings to improve in the BIOS but not sure which.

It sure would be nice to solve this. I will eventually get tired enough of dealing with this to just build another computer with different motherboard or even go back to my old one given current prices on new stuff is out of control. Kind of ridiculous to put up with this much longer though.