Question Do I "Have It Right" About Setting Up An NVME Drive On My Z170 Motherboard?

[BonzaiDuck]

Below, I've added the image of the relevant page from my ASUS Z170 motherboard manual.

When I originally built my system in late 2016, I had not yet caught up on NVME M.2 drives like the Samsung 960's, and made extra work for myself by installing the operating system initially on an SATA SSD. Well, I eventually managed to convert all my drives from MBR to GPT, and bought a Sammy 960 1TB and a smaller 256GB.

I had "plans" for the onboard Intel SATA controller and all the conventional SATA ports on the board. So I was initially confused about the "shared bandwidth" of M.2 drives and certain SATA ports. To avoid sorting out the details, I did the simple thing. I bought a $20 Cryorig_M.2 PCIE x4 adapter, configured the BIOS to get the full bandwidth from the "bottom" PCIE x4 slot, and installed the first and largest of the two Samsung M.2's. Given certain things that "I do" with the system (you can check the forums at a product/web-site for "Romex Software"), I added the smaller drive in another PCIE x16 slot. I only have a single graphics card, and reducing the bandwidth of the first PCIE slot to x8 had negligible effect on graphics performance.

Call me "stupid" at this point, but everything has worked perfectly with great NVME benchmarks, despite a less than optimal use of my motherboard resources.

I've come back to this issue because I had to replace my motherboard -- same make and model Z170 -- taking an opportunity to upgrade components as desired.

And so I gave some attention to the two M.2 slots -- never used -- on the board. Did some web-research, looking at forum posts from people with the same questions and confusions that I had. Forum posts were a bit dated, going back to 2017 and 2018. People were just becoming familiar with these NVME M.2 sticks -- it's obvious.

Indications seem to be that the NVME M.2 sticks will work at their full performance level, and likely won't sap bandwidth from any SATA devices. The shared-bandwidth issue becomes relevant when using M.2 SATA sticks instead of NVME's. It apparently only depends on how the M.2 slots are configured in BIOS, and I'm familiar with that as well. I think I can just stick the M.2 NVME's in those slots, since I'd already configured the BIOS for them when I first built the rig!

Can anyone offer a second opinion? Here, as I said at the beginning, is the page describing these slots from the board's manual.

 

[thigobr]

Going by the chipset datasheet it looks like some of the PCIE lanes are multiplexed between SATA ports and PCIE slots (including M2). When using M2 SSDs that are PCIE/NVME it should still take out SATA port resources in those cases.



Another minor issue is that the chipset is connected to CPU using a X4 DMI (essentially a PCIE3.0 X4 with 4GB/s each direction). That's the same bandwidth as a single M2 X4 slot. So using 2x M2 PCIE devices at the same time will result in a potential bottleneck.

 

[BonzaiDuck]

Going by the chipset datasheet it looks like some of the PCIE lanes are multiplexed between SATA ports and PCIE slots (including M2). When using M2 SSDs that are PCIE/NVME it should still take out SATA port resources in those cases.

View attachment 41701

Another minor issue is that the chipset is connected to CPU using a X4 DMI (essentially a PCIE3.0 X4 with 4GB/s each direction). That's the same bandwidth as a single M2 X4 slot. So using 2x M2 PCIE devices at the same time will result in a potential bottleneck.

Actually, I should beg to apologize for some inaccuracies about this matter. I oversimplified the situation thinking it would make posing the question easier.

The system for which I'd avoided using the M.2 "PCIE" motherboard ports used an ASUS Sabertooth Z170 S. In the process (and panic) of getting it up and running again, I didn't wait for ASUS to come through with its RMA process. They were absolutely stellar, but it took about two weeks, which would be expected. When this sort of thing happens, I'm inclined to spend extra money so I have a fallback. On EBay, I picked up an ASUS Z170-WS motherboard. So I decided to build another system with it.

The workstation motherboard has a much bigger compliment of PCIE lanes, working through a bridge chip. I had come across this fact before I picked the Sabertooth, and I remember looking at the same diagram you posted. The WS board allows three (possibly four?) full-size PCIE slots to run at x8, even if they're all filled.

Now looking at it from the other side, given the limitation of the four-lane DMI connection to the processor, I can say that my "big plans" for the SATA ports are much more modest. Of the six ports that can be connected, two of them will be allocated to a pair of eSATA plugs I'll custom-install on the case front-panel, so they're not likely to use any resources at all except when an external drive is connected to them. In fact, that may be the only use I have for the Intel ports, or -- I might connect two SATA devices -- some 2.5" spinners. One will be a media drive; the other one devoted to daily Macrium backups. Only the Monday differential backup takes much time -- maybe a couple hours. The incremental dailies usually complete in less than 20 minutes. At most, I may only have two SATA devices active on the INtel controller. I also have a PCIE x2 Marvell 4-port SATA/AHCI card -- SATA-III and PCIE 3.0-capable -- which I can add into a minor PCIE slot as an option. But I expect this to be a "one or the other but not both" decision.

On the Sabertooth board, I don't see any degradation in performance of the Sammy 960's deployed in the PCIE slots. On the other hand, putting two M.2 NVME's into the M.2 motherboard connections may, as you say, compete for resources. I might just connect one M.2 to the motherboard M.2 port, and add the other to one of the PCIE slots with an adapter card.

You can comment on this, given the extra PCIE lanes on the WS board. I've got months of time to build this system. It's almost just "something to do" with some spare-parts accumulation. I'm a hoarder of PC parts! I'll have to retire another household PC as an attempt at discipline when I complete this project!

 

[thigobr]

This Z170-WS has a PCIE switch PEX8747 upstream connected to CPU PCIE3.0 16X and downstream it offers up to 32 lanes.

In this case yes, it's better to use the upper slot for the GPU, a lower slot for the M2 adapter and another M2 SSD can go into the motherboard M2 slot. This way all devices can get full bandwidth.

 

[BonzaiDuck]

OK, then. 1TB NVME M.2 drives now seem cheap, compared to what you paid four years ago. I'll pose a question about what I chose this time on "Storage".