Total bandwidth of PCIe?

[uOpt]

So, PCIe is point-to-point, with a central switch.

PCIe x1 is 2.5 Gbit/sec, and higher PCI ratings accordingly.

But that leaves some questions:

1) what is the total bandwidth that the switch can handle? Is that specified in the standard is is that a matter of each implementation being free to offer as much as they can?

2) what happens if several PCIe devices all want to force data on the switch? How does the switch prioritise this? Is the switch getting advance notice of incoming packets so that it can make a decision? Is that under OS control?

Unreleated to this, but fits in: if you have PCI-X of some rating, is that always a totally seperate bus from PCI? If you have 32bit/33MHz PCI and PCI-X in one machine, does it always have at least two PCI busses?

Thanks

 

[Matthias99]

Originally posted by: MartinCracauer
So, PCIe is point-to-point, with a central switch.

PCIe x1 is 2.5 Gbit/sec, and higher PCI ratings accordingly.

But that leaves some questions:

1) what is the total bandwidth that the switch can handle? Is that specified in the standard is is that a matter of each implementation being free to offer as much as they can?

Up to the switch, although obviously it should offer at least enough throughput for an x16 connection if it intends to provide an x16 port, and more than that if it wants to support multiple ports at full speed simultaneously.

2) what happens if several PCIe devices all want to force data on the switch? How does the switch prioritise this? Is the switch getting advance notice of incoming packets so that it can make a decision? Is that under OS control?

I believe there's some sort of QOS built into the PCIe spec, but I'd have to go digging through their technical documents to figure out exactly what they do. The exact implementation may be left up to the switch designer; there are a variety of techniques that could be used to ensure that no devices are starved for access to the CPU.

Switches can't get 'advance notice' of packet-based communications; how exactly would you send the notification except via a data packet?

Unreleated to this, but fits in: if you have PCI-X of some rating, is that always a totally seperate bus from PCI? If you have 32bit/33MHz PCI and PCI-X in one machine, does it always have at least two PCI busses?

I believe so, although I'm no expert on PCI-X.

 

[uOpt]

Originally posted by: Matthias99
Switches can't get 'advance notice' of packet-based communications; how exactly would you send the notification except via a data packet?

There might be an extra physical line in the bus where the device tells the switch to get the data, analogous to an interrupt.

Or there might be a small "header" going over the wire as opposed to forcing the whole paket down the wire without asking first.

I would actually guess that there is something like this in place, I would be surprised the the device would just dump their data on the bus without asking first.

 

[Matthias99]

Originally posted by: MartinCracauer

Originally posted by: Matthias99
Switches can't get 'advance notice' of packet-based communications; how exactly would you send the notification except via a data packet?

There might be an extra physical line in the bus where the device tells the switch to get the data, analogous to an interrupt.

Or there might be a small "header" going over the wire as opposed to forcing the whole paket down the wire without asking first.

I would actually guess that there is something like this in place, I would be surprised the the device would just dump their data on the bus without asking first.

It's not a bus. There are no "lines" other than the serial data line(s) and the clock. Every device has a private full-duplex channel to talk to the switch with; they don't have to care what anyone else is doing. However, since only one device can be sending data to the CPU at a given moment, it is possible there's some sort of flow control built in.

 

[Peter]

The simple solution is that the fanout switches have broader uplinks toward the system than the individual links where the devices connect. They can handle and merge/dispatch traffic to and fro the individual links without starving any of them.

E.g. an 8-port switch very possibly has at least an 8x uplink. At least in system designs that make sense

 

[Peter]

Oh yes, the PCI-X vs. PCI question. Yes, usually these are separate busses. Often, there even are multiple PCI-X busses. The goal here is to avoid slow cards (or a high card count) slowing down the entire bus for everyone.

E.g. PCI-X starts out at 133 MHz if there's just one device on the bus, 100 for two, 66 for more. You'd be back to 33 MHz operation if you'd plug a standard PCI card in there.

 

[PumpActionWalrus]

From what I have read, all of the devices on a pcie bus have the same priority, but it is the way the switch works, supposedly the switch works about like a server does when multiple people are downloading from it.

 

[Peter]

As I said ... the fanout switches have enough uplink bandwidth to avoid contention, even if all the downstream clients go at full blast at the same time. (At least, that's how it should be in a well designed system.)

 

[Sahakiel]

Originally posted by: Matthias99
It's not a bus. There are no "lines" other than the serial data line(s) and the clock. Every device has a private full-duplex channel to talk to the switch with; they don't have to care what anyone else is doing. However, since only one device can be sending data to the CPU at a given moment, it is possible there's some sort of flow control built in.

There is more than one device listening. It is a bus.

 

[Peter]

No it's not. It's individual point-to-point connections.

 

[Sahakiel]

Originally posted by: Peter
No it's not. It's individual point-to-point connections.

1 upstream port + 1 downstream port = bus.