Question Newer CPU PCIe and DMI confusion

[luckywales]

Hi everyone - let me begin by saying I've seen posts that are similar to this but don't appear to address my direct concern.

I currently have an i7-6850k on an X99 board, and am about to upgrade. I purchased this to take advantage of (almost all) the available PCIe lanes from the CPU...not CPU + chipset...just the CPU, which had 40 lanes. Included in my setup are two M2 NVME PCIe drives that I think are taking full advantage of this setup on my motherboard (960 pros).

Has something fundamental changed over the last 3 years? I'm asking because when I look at the latest i7-9700k, it has less lanes. Whilst I understand that the CPU, plus the chipset lanes (on say a Z390) sum to 40 or more, aren't the lanes which are additional to the CPU just sharing the same Direct Media Interface (DMI) bandwidth? Or is something else happening in the background? Does this cause a bottleneck to the 970 pro speeds that I think I'll purchase for the M2 slots?

I'm not sure I'm asking the right question here, but with multiple GPUs, LAN and wifi cards will I be able to dedicate the entire bandwidth to my two M2 drives (not worried about the board shutting off SATA ports)?

I've looked at 4 YouTube videos on the basics of PCIe lanes vis-a-vis CPU and chipset, but I can't find an answer to the fundamental issue that the chipset might ultimately be dividing up the available bandwidth made available by the CPU. So did we go backwards in reducing the number of CPU lanes?? That can't be right? Right?

 

[VirtualLarry]

You're on a an HEDT system, which has more PCI-E lanes, than the mainstream consumer socket, of which the 9700K is. The consumer socket only has PCI-E x16 3.0 from the CPU for the primary GPU slot (or divided into x8/x8 for two GPU slots), and then a PCI-E x4 3.0 for the DMI (essentially), which handles the M.2 sockets. So, yes, on a 9700K rig, there IS a bottle-neck.

Stick with HEDT if you're concerned or need the PCI-E lanes.

 

[Markfw]

What Larry said. But what you really need is a new HEDT system with lots of PCIE lanes, like the threadripper. 64 lanes, and most motherboards have 2-3 nvme right on-board. Make sure to get heat sinks for them, or they will slow way down.

 

[VirtualLarry]

Yeah, ThreadRipper has NVMe RAID support, without having to buy a "Key" like Intel's HEDT platform. It's also (generally) cheaper.

 

[luckywales]

Ahhhhh - that makes more sense. So weird about Intel pricing now between the consumer and this HEDT line. The difference between the 6700k and the 6850k was not that bad, which is why I went that direction.

OK, so if 16 CPU lanes are dedicated to GPU(s), and then 4 for...everything else? Are those somehow "shared" over the additional lanes offered on the motherboard? (completely aware those might be the wrong words). Honestly half of this battle of understanding is making sure I ask the right questions!

 

[Markfw]

Ahhhhh - that makes more sense. So weird about Intel pricing now between the consumer and this HEDT line. The difference between the 6700k and the 6850k was not that bad, which is why I went that direction.

OK, so if 16 CPU lanes are dedicated to GPU(s), and then 4 for...everything else? Are those somehow "shared" over the additional lanes offered on the motherboard? (completely aware those might be the wrong words). Honestly half of this battle of understanding is making sure I ask the right questions!

If you are worried about PCIE lanes, you need to be thinking AMD HEDT, the threadripper. 16 cores and 32 threads for $570 or $869 deprnding on gen 1 or gen 2. And you can update to gen 3 when they come out later this year, Intel has nothing for you at this time with your requirements.

Edit: And to try and answer your last question, yes, the PCI card gets 16x and the rest are all that is left. The other PCIE lanes are on the chipset I think.

 

[LTC8K6]

Ahhhhh - that makes more sense. So weird about Intel pricing now between the consumer and this HEDT line. The difference between the 6700k and the 6850k was not that bad, which is why I went that direction.

OK, so if 16 CPU lanes are dedicated to GPU(s), and then 4 for...everything else? Are those somehow "shared" over the additional lanes offered on the motherboard? (completely aware those might be the wrong words). Honestly half of this battle of understanding is making sure I ask the right questions!

What motherboard do you currently have?

 

[Topweasel]

So quick run down of 4 platforms. Ignore CPU prices they are irrelevant (Platform is all that mattered including when you got your CPU).

Z370/Z390 CPU has 20 PCIe 3.0 Lanes. 16 to PCIe 16x #1 or Split to PCIe 16x #1 and #2. 4 PCIe lanes to the chipset. It then uses a PLX mulitplier like function to provide more lanes. All devices are limited back to the CPU at that 4x speed and are shared (so NVME and a video card in a 3rd PCIe 16x slot would be sharing 4x PCIe 3.0 lanes).

AM4. CPU has 24 PCIe 3.0 Lanes. Same 16x PCIe 3.0 to 16x #1 or split to PCIe 16x #1 and #2. 4x PCIe 3.0 dedicated to NVME port #1. 4x PCIe lane to Chipset. From Chipset AMD splits and sometimes down converts those last lanes for extra slots or NVME ports. Example my Taichi X370 board Uses 2x PCIe 3.0 slots from the CPU to feed either 1 4x PCIe 2.0 slot or a NVME 4x PCIe 2.0 M.2 slot. No matter how it's accomplished anything using lanes from chipset are bottlenecked at the same 4x PCIe 3.0 lanes that the consumer intel solution is limited to.

TR4. Threadripper is the easist to describe. All CPU's have1 64 PCIe lanes. 4X PCIe 3.0 to the chipset. The rest of the 60 lanes are handed out from the CPU. This means most PCIe ports and M.2 ports are connected to the CPU. Read the manuals though as some of the PCIe ports and M.2 ports use shared lanes (one or the other) or certain PCIe 16x slots split to 8x when partner slots are used (so PCIe 16x #3 and PCIe 16x #4 are split to 8x a piece when both are used).

X299. This one is mess. Because this Platform supported 3 different configurations of PCIe ports, the motherboard configuration can be confusing. One configuration mostly ignored is the Kabylake-X CPU's these were normal Kabylake 4c CPU's jerry rigged to fit on X299 and had the same limitations as the Z370 platform. Small Skylake-X CPU's 6/8c chips were limited to 28 PCIe slots. Higher end 10c and up chips support 44 lanes. To handle this a lot of boards went on the safe side and designed the boards to mostly use 28 PCIe lane CPU's and not the 44 Lane models, due to likely higher volume in sales and not wanting to have to many features disabled due to CPU choice (with all but the "economy" X299 boards ignoring the existence of KBL-X). This means that if you get a 44 Lane CPU you need to make sure the board you are looking to get has the features you want (more PCIe 16x slots or more M.2 slots) are being fed from the CPU. The chipset supports upwards of 28 PCIe lanes but those are bottlenecked through the same 4x PCIe lanes fed from the CPU as the X370 platform is. Most motherboards only have the first M.2 and most PCIe slots outside the first two PCIe 16x slots fed from the chipset because of the earlier mentioned designing them around the 28 lane model CPU's.