motherboard with most 4x PCI-E slots?

[azeem]

Hi,
I am a student and I am interested in doing some experiments with some data aquisition cards in our college lab. I need names/models of motherboards that contain maximum possible number of PCI Express slots of 4X speed or less (since 4X translates to 10Gbps througput in PCI-E which is what the fastest of our cards can handle.)

Thanks.
Azeem.

 

[peternelson]

Originally posted by: azeem
Hi,
I am a student and I am interested in doing some experiments with some data aquisition cards in our college lab. I need names/models of motherboards that contain maximum possible number of PCI Express slots of 4X speed or less (since 4X translates to 10Gbps througput in PCI-E which is what the fastest of our cards can handle.)

Thanks.
Azeem.

Hi Azeem,

Well, first I will say that 4x does NOT translate to 10Gbps.

1 lane of PCI express operates at 2.5 Gigabaud raw modulation rate. It uses at 8/10 encoding scheme which means the MOST possible data you get is less ie a real 250 MByte/sec (or238 Mibibyte/s).

Further, you would only get that in situation of burst transfers with little interrupts, etc. (ie depends on pci protocol overhead)

Ok, so 4x gives you 1000 MByte/sec = or 8000 MegaBITS/s [NOT 10 as you said]
8x gives you 2000 MByte/s = 16000 MegaBIT/s
16x gives you 4000 MByte/s. = 32000 MegaBIT/s

And of course those are one-directional rates, but since PCIE is full duplex you also get that in reverse.

Now consider You want to have 10Gigabit/s of data acquisition.

x4 is NOT enough for that. That is why say MYRINET who make a 10 Gigabit ethernet card (tested to 9.6 Gigabit/s ie WIRE RATE) make it with a X8 not a X4.

x4 could not handle that much traffic, even with ZERO overhead. So if you actually really do want to acquire at 10 Gig you need x8 board(s), and even then, Myrinet are doing clever things like protocol offload and interrupt aggregation on the board in order to not be limited by the cpu overloading.

Also it is important to distinguish between the PHYSICAL connectors on the board and the number of ACTIVE ELECTRICAL LANES.

OK.

Boards to look at are:

a) Gigabyte QUAD ROYAL (has FOUR x16 slots capable to work at x8 EACH) has been reviewed and is on their website www.gigabyte.com.tw but I have not seen any for sale anywhere so you may find it hard to get hold of!

b) Various boards based on the 975X chipset which have two physical x16 and can use them as x8 each, PLUS additional x4 or x1 slots

eg ASUS 975X board or Gigabyte 975X board or Intel 975X "BADAXE" board [all three have been reviewed by Anandtech] or maybe others from Foxconn etc

Note that Gigabyte's (turbofan) implementation of 975X has OPEN ENDED slots. eg put in a 16x or 8x shaped card into your "open4x" slot.
Intel have gone with (fewer) 16x physical slots (but you still get less than that in electrical lanes).

c) Various SLI x16 boards which use NVIDIA CHIPSET to give 16 true lanes to each 16x pci-e slot, and maybe spare lanes too.

OK, so for example if you go with Gigabyte's 975X board (noisy fans aside) you should get

one 16x graphics card operating at 8x
one DATA ACQUISITION CARD operating at 4X or 8X
one DATA ACQUISITION CARD operating at 4X in a 4X physical slot
one physical x4 slot operating at only 1X (because lanes are limited availability).

The manual is not entirely clear about this but that is how it should work.

If you put a faster card in a slot with less electrical support (or available lanes) it SHOULD auto-negotiate how many lanes to use.

Asus 975X "-E (enthusiast)" board should offer very similar.

Now going further if you could for example use legacy pci for your graphics or an X1 slot or a X4 slot operating as X1 for your graphics, then you may free up extra lanes. ie you could then be also using one of the primary x16 slots (operating at x8) for an additional high speed data acquisition. I cannot guarantee this will work depending on bios implementation but it probably CAN work.

Some vendors chose to have more legacy pci slots than pcie ones.

c) Various SLI x16 boards which use NVIDIA CHIPSET to give 16 true lanes to each 16x pci-e slot, and maybe spare lanes too.

I would be personally inclined to steer clear of nvidia chipset because of poor network implementation (activearmour) and some reported disk corruption problems over sata.

d) There are some "server" type boards supporting pciexpress but they generally don't have many slots so would not really be suitable for your needs.

-----

Now you also have to consider where that data goes. Is it buffered somehow on the data acquisition board into onboard RAM?

Or are you trying to process it in the main cpu or send it over a net or log it to disk? If any of these you may have trouble keeping up with it!

Hope the above helps. Can you post details of the data board you intend to use? eg Manufacturer, part, or is it a custom one?

What are you acquiring? Digital or analogue data?

Peter

P.S. I think my choice would be the Gigabyte but I would try to fit some fan speed limiters to reduce the noise level a little. Indeed I think I am about to purchase it (but Gigabyte cannot say yet if it is "Conroe ready")

 

[peternelson]

Just as an aside, something that may be of use to you in this application is a clever little PCIe adapter from Twin Industries.

This will hold your X4,X8 or X16 card and allow it to plug into a X1 physical slot.

http://www.twinhunter.com/catalog_detail.php?id=303

Of course you only get X1 speed, but it lets you use those slots.

By raising your board from the motherboard (to fit the adapter), you may find the rear slots don't fit in your chassis so time to get your hacksaw and dremel out to make it "customised". Or you could run it without a case - good for cooling ;-)

Check out their other pcie products too (like riser cards).

 

[azeem]

Hello Peter,
Thanks a lot for the detailed answer. Actually, my professor is not quite clear on how he wishes to proceed with 2-3 ideas. One idea is to have multiple 10Gbe cards to build a fast router for testing some routing protocol code. Another is for doing signal processing on raw video. Anyway, I guess, I have something more concrete to work with now. Thanks again.

-Azeem.

 

[peternelson]

Originally posted by: azeem
Hello Peter,
Thanks a lot for the detailed answer. Actually, my professor is not quite clear on how he wishes to proceed with 2-3 ideas. One idea is to have multiple 10Gbe cards to build a fast router for testing some routing protocol code. Another is for doing signal processing on raw video.-Azeem.

If you want to work with 10 Gigabit ethernet, you will be talking x8 cards.

Go buy a couple of these

http://www.myrinet.com/Myri-10G/NIC/10G-PCIE-8A-C.html for $795 each.

I doubt you will find any much cheaper or as FAST! With additional software $$$ and a myrinet switch they will also do the proprietary low latency myrinet networking.

Basic ethernet drivers are available for plenty of platforms including LINUX which would probably be most suitable for experiments building a DIY router. Just emulate a Cisco and make sure you support all RFCs like BGP4 LOL. But with only two interfaces you have limited reason to route? Surely a switch would do that based on mac tables? You could filter ie block certain traffic or ports, or do L4-L7 switching, but for routing to be useful you will probably need to direct traffic at the mac address or the next hop router even if it is on the same segment as another router. Or else you can succeed in routing nowhere ie traffic coming in just goes to some NULL "black hole".

You could try building that in a 975X based board with a cheapo graphics card which should give you a couple of x8 slots to play networking with (and maybe a x4 if you want to put (say) an Intel DUAL gigabit port NIC in another slot for routing some slower traffic flows). Then again there is some gigabit onboard (maybe that is your "console or management port").

A dualcore processor eg 920,930,940,950 (cooler than 8xx processors) may be required to keep up (or maybe even approach) wire rates on two cards and some routing decisions too. Fortunately the ASICs on the NICS are already doing a lot of hard work. 9xx should (one day) eg Redhat end of 2006, support VIRTUALISATION which may be a nice feature to run the NICs and routing process virtualised away from other tasks which may crash it.

In a commercial high-end routing switch such operations would probably be done by a custom route-switching asic connected to the PHYs.

It is probably possible using Xilinx Virtex 4FX fpga using the onboard MGT high speed serial links and the embedded powerpc could just be used for stats and control purposes not actually read and write all the traffic. Still quite ambitious to develop a routing switch in hardware in FPGA, as well as writing 10Gbps MAC as the two provided on chip are only 1 Gbit/s ether MACs. However, for prototyping some kind of router, a couple of those Myrinet cards would be sweet. The CX4 versions are cheaper and can go 15m. Fiber versions are available but more $$$.

Being a router guy, now you are giving me ideas.... I had planned just one card per machine for clustering interprocess communications, but can I resist a DIY router? LOL

In 10G ethernet mode the nics can be connected between machines back to back (but that's not what you're doing).

A (relatively) cheap 10G capable switch (without myrinet protocol) is from Hewlett Packard the 3400cl (24 port version) at gigabit speeds plus put in a module that supports two CX4 connections at 10G.

If you want more than two 10G switch ports you're looking more money, something like the 6400cl series with eight 10G ports.

As for "signal processing on raw video" this is something I am familiar with LOL.

To do it in hardware, maybe take SDI @270Mbps uncompressed or HDSDI @1.5Gbps, deserialise using gennum chipset into Xilinx FPGA. Do your processing in the Xilinx (in parallel!) but then you need moderately expensive hardware and maybe ISE software development license depending on device. Look at Altera and Xilinx for boards set up for digital video as both were at the IBC broadcast show, because they want to get into that market more. Saves making up your own boards ;-) Also Xilinx have some XAPP application notes on processing video in hardware like that. Quantel (the high end tv effects hardware company) make their gear with Altera fpgas (I found that out when I took the lid off my Quantel Printbox).

Of course using simple analogue video like CCTV is much easier at consumer quality just use some "prosumer" grabber board (some even do SDI/HDSDI to Quicktime video). However if using compressed formats, consider artifacts, frame accuracy when seeking etc. In software a Pentium 4 can actually do quite a bit of image processing - I know we are doing object detection and motion analysis from a camera on a microscope, and some maths too ;-)

Best wishes with your project(s).

If you come across any 10 Gigabit ethernet boards with x8 pcie at any reasonable price (compared to Myrinet) please let me know. Most vendors are late to the party. Peter.