Skylake platform and the horrible status of the multitude of I/O options topology

zir_blazer

Golden Member
Jun 6, 2013
1,206
502
136
Back when Haswell platform was released, the way that it homogenized Chipset I/O made the Motherboards topology mature and stable in a curious way. Intel provided Flex I/O that allowed in Z87 to have either a max of 18 I/O Ports made with a mix of 4-6 USB 3.0, 4-6 SATA-III and 6-8 PCIe 2.0 Lanes (With 18 maximum along them).
Since normally you would have 3 PCIe Slots attached to the Processor PCIe Controller, at most the Chipset would need to handle 4 PCIe 2.0 1x/4x Slots, which you had to feed with up to 7 Chipset PCIe 2.0 Lanes (Since from the 8, one was ate mandatorily by the integrated NIC). This allowed to do something like 4x/1x/1x/1x, but I think I never actually saw this arrangement since the Chipset lanes were mostly used to feed third party controllers for extra SATA or USB Ports, making it impossible.
However, at the end of the day, the topology of the Haswell generation Motherboards was usually rather consistent and predictable. The only exception was if Thunderbolt was added to the mix, since the Thunderbolt Controller ate 4 Chipset PCIe Lanes all by itself. But normally, PCIe Lanes, while limited, were often enough.


This changed with the introduction of the new hybrid connectors to allow for faster SSDs, which are actually making the situation that on Haswell generation was rather standarized, a total mess in H97/Z97 and Skylake.
The hybrid connectors usually uses SATA and PCIe Lanes, however, since the amount of lanes varies by connector and they can be 2.0 or 3.0, the possible theorical maximum bandwidth varies depending on the specific Motherboard implementation. This is worse since it also depends if the lanes comes from the Chipset or the Processor itself. I would say that lanes coming from the Processor should perform better than Chipset ones even if both are 3.0, simply because data doesn't have to travel an extra hop to get to the Processor itself (Nor risk getting bottlenecked by the Chipset DMI).

First, we have SATA Express, which is a ugly, PATA-like connector that fuses two standard SATA Ports and 2 PCIe Lanes. This was supposed to be the future Desktop high end SSD connector. However, one year after its introduction, I still have to see a SATA Express device. At this moment, I would say that the industry actually wants to prematurely abort it and forget that it happened.

Second, we have M.2. This was sort of a SATA Express version for Notebooks. However, the connector is intended not only for storage, it actually provides a whole bunch of additional stuff. Suddently it became widely popular in Desktops Motherboards, and now you usually see one on most of them.
Problem is that M.2 has several versions since the slot itself is keyed and supports a whole bunch of stuff. For SSDs, it can provide either 2 or 4 PCIe Lanes which can also be 2.0 or 3.0, and come from either Chipset or Processor. But there are simply too many variations, so its rather annoying.


When Broadwell Chipsets were introduced, they provided for either one SATA Express or a 2 PCIe Lanes M.2. Regardless, these 2 PCIe Lanes came from the maximum 8. Thankfully some good through implementations preferred to take it from the Processor to provide 4 3.0 PCIe Lanes, like this Motherboard.
Good thing is that neither SATA Express nor 2x 2.0 M.2 could saturate Haswell/Broadwell platform DMI 2.0 (Effectively 4 2.0 PCIe Lanes communicating Processor and Chipset), so at least until then, it could be possible to see a Motherboard with a near identical arrangement to a Z87 Motherboard with Thunderbolt BUT with a 4x 3.0 M.2 coming from the Processor working in 8x/4x/4x mode, which wasn't a bad idea for as long as you were running with a single Video Card (nVidia restricts SLI to at least 8x/8x).


Finally, Intel also released their SSD 750 line. You have the famous PCIe AIC version, and also a 2.5'' SSD type. This last one has a UGLY way to plug to the rest of the computer, since it requires a special SAS-based connector. The SSD connector was a SFF-8639, which includes data and power (Whose power component could be connected straight from a SATA Power of the Power Supply to a sort of Y cable), which was recently renamed to U.2. It provides 4 PCIe Lanes instead of SATA Express 2 lanes. However, the connector that goes to the Motherboard is a SFF-8643 miniSAS. This can't be connected directly, so you need an adapter, which would need a 4x 3.0 M.2 to work at its fullest. M.2-to-miniSAS adapters are like this. And if you want to troll, I THINK you can get a PCI Express 4x-to-M.2 card, plug in that M.2-to-miniSAS adapter, and finally plug the U.2 SSD. Now you get why I love the simplicity of the Intel 750 PCIe version, don't you?

There is also USB 3.1, which is usually done using two Chipset lanes and a third party controller.


Then we enter Skylake. The Skylake platform includes a new Flex IO system, providing 6-10 USB 3.0 (Still maximum 14 USB Ports including the USB 2.0 Ports, as with Z87/Z97), 0-6 SATA-III, a 1 GBps NIC, and 7-20 PCIe 3.0 Lanes (Assuming you sacrifice all but 6 USB 3.0 Ports).
While Skylake platform upgraded the Processor to Chipset connection to DMI 3.0 (4 3.0 PCIe Lanes), and all the Chipset PCIe Lanes are now 3.0 too, you suddently face a lot more possible bottlenecks since all the new SSD biased connectors heavily uses PCIe Lanes. For example, now you can put a 4x 3.0 M.2 to the Chipset lanes, but then you would saturate the DMI 3.0, so on Skylake, it would still make sense to use the Processor 8x/4x/4x arrangement in the same way that on Haswell platform.
Then you also have a possible third party controller for USB 3.1, and maybe the premium Thunderbolt controller eating 4 lanes. Assuming you are using 10 USB 3.0 and the 6 SATA-III, just by having Thunderbolt you're left with just 3 PCIe Lanes for maybe 3 PCIe 1x slots. Suddently, Thunderbolt and a 4x M.2 don't seem to fit together unless you sacrifice something else.


If you read all this, I think you get what I mean. Today, you got a lot of Motherboards with a plethora of I/O that can't really work together, since if you use the M.2 slot, maybe you lose some of the SATAs or a PCI Express slot gets disconnected since its lanes are being redirected to the M.2.

The topologies should be ridiculous complex at the moment, there are just TOO MANY different ways to do things and variety compared to Haswell platforms. This is not exactly good, I see it more like walking on a labyrinth.
Wouldn't it have been more simple if in Desktop Motherboards, instead of a dedicated M.2 slot (Usually at the cost of a PCIe Slot), you get a 4x PCIe Slot THEN use a PCIe-to-M.2 adapter like this? Except by the fact that you usually lose PCIe Slots thanks to Dual Slot Video Cards, I think that SATA Express and M.2 are a bad idea, and would be more flexible with such adapters than by making Motherboards so overly complex.
 

Deders

Platinum Member
Oct 14, 2012
2,401
1
91
The M.2 slot on z170 is interchangeable with the first 2 Sata ports which I think also double as Sata Express.

So basically you can have 1 ultra fast SSD (without taking up any PCIe slots) and 4 sata3 drives, plus 2 more on the Asmedia controller, or you can have a total of 8 Sata3 drives.
 

zir_blazer

Golden Member
Jun 6, 2013
1,206
502
136
The M.2 slot on z170 is interchangeable with the first 2 Sata ports which I think also double as Sata Express.

So basically you can have 1 ultra fast SSD (without taking up any PCIe slots) and 4 sata3 drives, plus 2 more on the Asmedia controller, or you can have a total of 8 Sata3 drives.
This depends on how the Motherboard manufacturer decides to implement the M.2 slot. For example, if you check the Supermicro C7Z170-SQ manual, you will see that on the Block Diagram (Page 1-9) they are using all 6 SATA-III from the Z170 Chipset, the M.2 is M-Keyed and uses 4 PCIe 3.0 Lanes with no SATA at all (I also noticed that it uses a total of 24 Flex IO Ports, is missing 2 for the full 26). A M.2 slot that is not connected to SATA can't use AHCI SSDs at all if I recall correctly.

Also, it is a lie that you don't lose PCIe Slots. The M.2 slot is usually put either at where the first PCIe Slot used to be, or between two of them, so you are losing one anyways.
Since the PCIe Lanes are spend regardless, I would take a PCIe 4x Slot any day of the week, then use a M.2 adapter. This option is plainly more flexible. I think you only really need M.2 on the Motherboard itself if you need to attach SATA to it.


Sata Express was DOA.

U.2 and M.2 are where its at
U.2 is still EXTREMELY hard to work with on consumer platforms, since you have very limited PCIe Lanes and need a lot of adapters. Assuming you have a 4x 3.0 M.2 slot, you could use the M.2-to-miniSAS adapter. Otherwise you need a PCIe 4x-to-M.2, then again, the M.2 adapter.
There is only one Motherboard which I'm aware of that provides onboard miniSAS, the Gigabyte MX11-PC0. You could connect the 2.5'' Intel 750 SSD straight to it. Since there is no public Manual for that Motherboard nor a Block Diagram, I don't know how Gigabyte is actually using that connector.
However, getting more than two U.2 drives on LGA 1150/1151 is pushing it too far. Remember that they can't go through the Chipset since a single one is already bottlenecking it (PCIe 4x 3.0 = DMI 3.0, but add in all the other IO that is also attached to the Chipset and you have an important bottleneck). I suppose than the second best option would be a single of these cards coming from the Processor using 8x/8x mode, or two, if you want to sacrifice the discrete Video Card (But then, it would be a Server machine).



My best idea for a straightforward topology, coming from Intel Flex IO chart, would be...

PCH%20Allocation.png


Ports #1, 4, 5, 6 = USB 3.0 (Not that there is another option in those 4!)
Ports #7, 8, 9, 10 = USB 3.0
Ports #2, 3 = USB 3.0 or SSIC, assuming that there is a reason to use the second
That gives us either 8 or 10 USB 3.0, depending on what SSIC can be justified to be used for (I googled around and can't find what it could be useful for, originally I thought than that was routed to provide for Thunderbolt USB support).

Ports #11, 12, 13, 14 = Thunderbolt 3
You require 4 PCIe 3.0 Lanes for it, that should do it. Alpine Ridge's Thunderbolt 3 provides for 2 premium USB 3.1 Type-C ports, since you can use them as standard USB 3.1 or the extra Thunderbolt features, so that's get you nearly fully covered (You're just missing USB 3.1 Type-A, but I think you can use a Type-C to Type-A adapter).


The other 12 Ports can be though as 3 PCIe RST devices based on Intel grouping. However, you can't really go for all 3 of them since you're left with no standard SATA nor NIC, not at least without sacrificing one USB 3.0 Port (#10) or the Thunderbolt group (#11). So, this is where things gets hairy...

Ports #15, 16 = SATA-3
Ports #17 = PCIe 1x slot (Or an secondary physical NIC for Dual LAN)
Ports #18 = Intel Chipset NIC

Ports #19, 20, 21, 22 = Intel RST compatible PCIe SSD (In PCIe 4x slot?)
Ports #23, 24, 25, 26 = Intel RST compatible PCIe SSD (In PCIe 4x slot?)

Alternatively, I suppose these could be arranged in two groups where each 4-port group has two SATA ports (#21, 22, and 23, 24) plus a PCIe 4x slot, which gets full lane routing if left alone but gets downgraded to 1x if the SATAs are used.


PCIe Slot arrangement for full ATX of this topology would be (From the top, which is closest to Motherboard IO panel, to bottom):

Slot #1 = PCIe 1x (Port #17)
Slot #2 = PCIe 16x (Processor, for main Dual Slot Video Card. Could be hardwired to 8x so you avoid using the PCIe switch chips)
Slot #3 = -Nothing-
Slot #4 = PCIe 16x (Processor, for either second SLI/Crossfire Dual Slot Video Card in 8x mode, RAID Controller, or 4x PCIe SSD)
Slot #5 = -Nothing-
Slot #6 = PCIe 4x (Ports #19, 20, 21, 22)
Slot #7 = PCIe 4x (Ports #23, 24, 25, 26)

Alternatively, if you want to do 8x/4x/4x with the Processor, you could get an additional PCIe 16x physical/4x data slot. And personally, since you lose pretty much nothing if going from 16x to 8x, I think that most 2-16x PCIe 3.0 Motherboards could have it hardwire to 8x/8x simply to avoid using the 4 PCIe switches ICs for 8x/8x, and 6 for 8x/4x/4x.

A second good idea with Dual NIC, no PCIe 1x slot and Processor in 8x/4x/4x mode, would be...

Slot #1 = PCIe 16x (Processor, for main Dual Slot Video Card. Could be hardwired to 8x so you avoid using the PCIe switch chips)
Slot #2 = -Nothing-
Slot #3 = PCIe 16x (Processor, for either second SLI/Crossfire Dual Slot Video Card in 8x mode, RAID Controller, or 4x PCIe SSD)
Slot #4 = -Nothing-
Slot #5 = PCIe 16x/4x (Processor)
Slot #6 = PCIe 4x (Ports #19, 20, 21, 22)
Slot #7 = PCIe 4x (Ports #23, 24, 25, 26)

That would be the topology of my dream Skylake Motherboard. Such a Motherboard would be even cheap to boot since its a topology intended to get rid of extra, redundant and usually not needed third party controllers.
 
Last edited:

HexiumVII

Senior member
Dec 11, 2005
661
7
81
Makes me want to wait for Skylake-E, but I already have a 6700K on its way. Still picking out a motherboard though. I remember I went through topographys back in the old days to make sure I didn't have the soundcard mixed in with the USB. Looks like we have to go through all that again. I usually have my computers filled with 2-3 SSD and 4-5 HDDs, so it can really matter. I'm also moving to mostly SSDs now as they are so cheap, they can actually saturate the DMI.
 

VirtualLarry

No Lifer
Aug 25, 2001
56,570
10,200
126
First, it was manually configuring IRQ jumpers on ISA cards. Then PCI came along, and you had to plug your PCI cards in according to PIRQ routing tables, in which certain slots shared certain hardware IRQs (but then had seperate software IRQ numbers, what nonsense!).

Now we have to configure our motherboard loadouts according to "FlexI/O routing tables".

Back to the bad old days, I guess, because Intel is too cheap to add more I/O pins on their PCH chips.
 

StrangerGuy

Diamond Member
May 9, 2004
8,443
124
106
If Intel can't be bothered to de-bottleneck by putting more lanes, nobody else can. It's classical Intel behaviour really: cost cutting for their own benefit by providing only the bare minimum functionality possible while relying on the OEMs and mobo makers to work around the limitations, almost always haphazardly and detrimental to the end consumer.
 

zir_blazer

Golden Member
Jun 6, 2013
1,206
502
136
I remember I went through topographys back in the old days to make sure I didn't have the soundcard mixed in with the USB. Looks like we have to go through all that again.
I recall that ASUS had a rather popular budget Motherboard in AMD Socket AM2 generation (Can't remember the model) that had an integrated Sound Card that used USB. As you added USB devices, the sound started to get glitchy or choppy, or something like that.
Some guys workarounded that by using a PCI Sound Card. However, we can agree than that was an awful design choice to begin with.


If Intel can't be bothered to de-bottleneck by putting more lanes, nobody else can. It's classical Intel behaviour really: cost cutting for their own benefit by providing only the bare minimum functionality possible while relying on the OEMs and mobo makers to work around the limitations, almost always haphazardly and detrimental to the end consumer.
Technically Intel DID improved the situation with Skylake since now you have DMI 3.0 (4 PCIe 3.0 Lanes) for Processor-Chipset communications. Haswell generation had DMI 2.0 (4 PCIe 2.0 Lanes). And since all Chipset PCIe Lanes are also 3.0, you could use a single lane to feed stuff that previously needed two to get full performance (Assuming the devices also can use PCIe 3.0), like the USB 3.1 Controllers.

However, the point is, during Haswell generation, unless you wanted Thunderbolt 1/2, the Chipset max 8 PCIe 2.0 Lanes were adequate enough to get a fair share of dedicated PCIe 1x Slots with some extra onboard third party controllers (Which I don't like anyways), and it was hard to come to a DMI bottleneck situation since it required for you to use a lot of things simultaneously for that to happen.
You could also get very clean Motherboard topologies since there wasn't a lot of need for variety, as Haswell Chipsets consolidated most of the SATA-III and USB 3.0 needs of that era (Sandy Bridge platform was a disaster in that regard, remember it? And SB-E X79 Motherboards were even worse), so you could have solid Motherboards with just the Chipset for I/O and barely anything else, and it did most or all of what you could want to do.

Even through Skylake platform is way better than Haswell, the new PCIe SSDs did a complete reversal of the situation, the platform simply isn't adequate enough for the new high end I/O demands. A SINGLE 4x SSD can saturate DMI 3.0 (And remember that everything else but the Video Card have to use DMI to talk to the Processor).
Good thing is that the bottleneck is still not in full effect since PCIe SSDs are just around over half of the 4x 3.0 bandwidth, and you even do get performance increases by using the new Intel RST to make a RAID 0 of PCIe SSDs. But in a generation or two of SSDs performance increases, it will not suffice, they will catch PCIe 3.0 4x, and the whole Intel RST RAID 0 thing would also be pointless. Not to talk about that two RST 4x ports means sacrifices like only having two SATAs.

The last thing could be fixed by adding third party SATA Controllers. However, I like to avoid them when possible, and for me premium Motherboards are awful since they're full of them.
The reasons why I dislike third party controllers is that besides that usually they perform worse than the Intel/AMD Chipsets integrated ones, they don't provide neither the same feature levels, have poor Driver support, quircks or bugs. The Windows folks may not notice it, but for IOMMU Passthrough, every now and then you see Linux horror histories like this one with a Marvell SATA Controller. Intel and AMD have teams that develop Drivers for Linux, but I don't know about the others. So the closer that I stay with Intel and AMD, the less potential issues I will face.

Finally, Skylake Motherboard topologies are screwed up since there are now too many variety of connectors that take real estate (Mainly M.2), and that since they require a lot of PCIe Lanes, you have to decide what to pick and what to sacrifice. A premium Motherboard with tons of connectors, slots and third party controllers doesn't make a lot of sense when you're aware that you can't use everything simultaneously due lane rerouting, or that DMI will bottleneck you anyways. Reason why I came out with that wet-dream topology, relying only on Skylake Chipset and Alpine Ridge for basic I/O, and PCIe Slots with adapters for universal I/O connectivity. I believe it should be better overall.
 
Last edited: