Anyone with Infiniband Experience

[Ipno]

I'm getting about 150 new linux servers that are going to be using infiniband as a backend. Was wondering if anyone with infiniband experience could share any experiences they've had, any troubles or quirks I might find.

 

[nightowl]

The biggest issue that I have seen with IB is the distance limitation of the cables. However that has been with really large clusters. For about 150 nodes you should not have any trouble with the cable distances. I will say for best performance you should use PCI-express HCA's with RDMA. What kind of appications were you planning on running on the cluster?

 

[Ipno]

Mostly fluid dynamics applications. The systems are coming pre-racked and pre-installed from Dell, so I really don't have any say in the adapters. I just have to support them.