Gigabit Hardware Reviews/Reccomendations

[Armitage]

I have the opportunity to assist in building a small (between 16 and 32 cpu) new Beowulf cluster. We've used Myrinet in the past, but its damn expensive, and we've had stability/reliability issues with it, so I'd like to consider gigabit ethernet.

I saw a review some months back ... maybe more like a year, that showed some large performance differences between various brands/models of gigabit hardware. Does anybody have anything more recent?

My primary performance emphasis is latency rather then throughput.

 

[cmetz]

ergeorge, I would suggest you get a couple of pairs of a few vendors' gigabit NICs and test them before buying a bunch. In particular, look at the Intel Pro/1000MT (there's two and four port gigabit NICs from Intel out also, btw!), the 3Com 3c996B-T, and the SysKonnekt SK-9821. For all of those, you really should use jumbo frames (8192 - 9180 bytes, see what's fastest) and you should disable all interrupt coalescing behaviors (which will decrease latency a lot).

If you're really latency sensitive though, Myrinet is going to be a lot better for you. If not that, also consider using 100BaseTX instead - I could be wrong, but I believe that the PHY DSP requirements of 1000BaseT mean increased latency is simply required for 1000BaseT vs. 100BaseTX.

If you're really concerned about latency, a good gigabit switch is very important, as is making sure that the backplane is never oversubscribed and that the switch properly supports jumbo frames (the cheap ones don't). Extreme, Foundry, Enterasys, and Cisco make credible switches in this space, though I think only Extreme has a non-oversubscribed backplane.

 

[Armitage]

Originally posted by: cmetz
ergeorge, I would suggest you get a couple of pairs of a few vendors' gigabit NICs and test them before buying a bunch. In particular, look at the Intel Pro/1000MT (there's two and four port gigabit NICs from Intel out also, btw!), the 3Com 3c996B-T, and the SysKonnekt SK-9821. For all of those, you really should use jumbo frames (8192 - 9180 bytes, see what's fastest) and you should disable all interrupt coalescing behaviors (which will decrease latency a lot).

If you're really latency sensitive though, Myrinet is going to be a lot better for you. If not that, also consider using 100BaseTX instead - I could be wrong, but I believe that the PHY DSP requirements of 1000BaseT mean increased latency is simply required for 1000BaseT vs. 100BaseTX.

If you're really concerned about latency, a good gigabit switch is very important, as is making sure that the backplane is never oversubscribed and that the switch properly supports jumbo frames (the cheap ones don't). Extreme, Foundry, Enterasys, and Cisco make credible switches in this space, though I think only Extreme has a non-oversubscribed backplane.

Thanks! Looks interesting. I honestly don't understand about half of what you mentioned (interrupt coalescing, oversubscribed backplane, etc.). But it gives me a starting point & things to look for!

Any resources you could reccomend for learning about stuff at that level?