Processors for CFD

[Phew]

At my work we do a lot of computational fluid dynamics (CFD) and we are currently expanding our processing cluster from 4 dual-xeon nodes to include 4 or more Athlon64 single-core nodes. When cluster-building, the key is bang for the buck; for instance, the dual-Xeon nodes only iterate 10-20% faster when utilizing both processors than if you only use one of the processors, so all of the associated costs (dual-proc mobo, psu, etc) ended up being wasted money.

So my question is: Is there some online resource where CFD users share their knowledge on parallel processing hardware? For instance, I'd like to know if AMD's X2 processors are worth the cost premium over single core nodes when using the GASP CFD solver. We have been told by the code developer that memory bandwidth is a bigger bottleneck than clock cycles, so I imagine the X2 processors wouldn't perform well. I'd like to see some hard numbers though.

 

[Vegitto]

For what you're looking for, I'd get Opterons. Safer, and more memory bandwidth. Ideally, you'd be looking at either 2x 150 or 2x 270 and 2~4 GBs of RAM per machine. Or you could wait a month or two and get the new Xeon's, I heard they rock.

 

[thecoolnessrune]

Vegitto, the Opterons first number stands for how many processors you can have:

1xx- 1 processor
2xx- 2 processors
8xx- 8 processors

So 2 Opteron 150s aren't possible. It would have to be more of 2 Opteron 242s.

 

[aka1nas]

Originally posted by: Phew
At my work we do a lot of computational fluid dynamics (CFD) and we are currently expanding our processing cluster from 4 dual-xeon nodes to include 4 or more Athlon64 single-core nodes. When cluster-building, the key is bang for the buck; for instance, the dual-Xeon nodes only iterate 10-20% faster when utilizing both processors than if you only use one of the processors, so all of the associated costs (dual-proc mobo, psu, etc) ended up being wasted money.

So my question is: Is there some online resource where CFD users share their knowledge on parallel processing hardware? For instance, I'd like to know if AMD's X2 processors are worth the cost premium over single core nodes when using the GASP CFD solver. We have been told by the code developer that memory bandwidth is a bigger bottleneck than clock cycles, so I imagine the X2 processors wouldn't perform well. I'd like to see some hard numbers though.

The problem isn't that your CFD simulations don't scale well with more processors, it's that Xeons are pretty bandwith limited as they share a FSB. X2s also share the same dual channel memory controller at the moment, but they are quite a bit less dependent on memory bandwith to maintain good performance than netburst derived chips. the 2xx dual core Opterons might be prefferable as you should probably have ECC support as you are doing simulations and probably don't want to risk having any uncorrected memory errors.

 

[lifeguard1999]

We have run benchmarks on all the latest HPC machines, but I do not think they would be of any use to you. After all, you are building a cluster, not buying a Cray XT3. In general, I will say that CPU speed is the first thing to look for. Memory latency and bandwidth, while good, is something that you can do little about. AMD is stuck with DDR; Intel with DDR2. More memory per node is good if it enables you to run on fewer CPUs; this gains you more parallel efficiency. However, 2GB & 4GB modules are expensive. Since you are running in parallel, I would advise for as fast a network as your money can buy. Gig-E is nice and cheap; Infiniband is expensive, has large bandwidth, and low latency.

 

[Phew]

Our cluster runs on gigabit ethernet, and right now the network doesn't seem to be a bottleneck (probably because our cluster is so small). Veg, remember cost efficiency is the number one concern for us; a quad Opteron box with 4GB of ram may be fast, but it will cost like 4 grand (guessing here). Four cheap A64 machines are like $1600 total and are probably about the same speed as a quad Opteron system.

As far as ECC ram, in 3 years of running CFD, we have only had one job die from a memory error, so the memory bandwidth penalty associated with ECC probably wouldn't be worth it.

I'd love to have the luxury of buying several different systems and benchmarking them, but that just isn't cost-effective for our small company. I just wish I could find a site on the web that benchmarks parallel CFD runs on similar P4, Xeon, A64, and Opteron systems to see where the best bang for the buck is. Since we can build sub $500 nodes with socket 939 A64s that have onboard video and drives for debugging, we will just stick with these until I see hard numbers showing our money is better spent elsewhere.