Need suggestions on a monster workstation build for FEA work

hoorah

Senior member
Dec 8, 2005
755
18
81
We run some pretty beefy computers at work, and we've been allowed to start looking at a round of upgraded machines.

The person in charge of this task went to a vendor and they put together this system and gave us a price of about $8500. This was presented at a meeting and I thought we could do better in at least a few areas, if not the whole thing. I'd like to hit around $5k and bank the rest for a mid-cycle upgrade.

Here's what I'm looking at:

2 x XEON e5-2637V2 @ 3.5ghz
Supermicro Dual Socket Motherboard (doesn't specify which one)
8 x 16GB DDR3-1600 ECC RAM
Nvidia Quadro K5000 video card
Hard Drive Controller: LSI 2208 SAS2 Controller
HDDs: 1x256GB SSD, 2x600GB Hitachi 15k SAS2 drives in RAID0
900W Platinum Power Supply

So I realize that most of those parts are not brand specific. For now, I'm just trying to iron out which platforms are the best to go with. I'm mostly looking for suggestions on the CPU and HDDs.

To start out, I suppose I should say what we do. These systems will be mostly for people performing finite element analysis, which is very CPU, RAM, and HDD intensive. There is no set requirement for performance - we use all of what we have. If we have more CPU than we need, we build bigger models until we're using 100% of the capability. I typically will use 32-64GB of ram, but some people use more. If we don't have enough RAM to run in memory, our solutions will page to the hard drive, which is incredibly slow but sometimes necessary. So, drive speed is huge as well.

The software we typically use is late to the party when it comes to GPU calculations. They're releasing some versions that have support for Nvidia Quadro K5000 and Tesla cards, but anything other than that is unsupported. I may have the opportunity to see if I can get it to work with some gaming level cards, but that will come later. The software license to use GPU calculations is separate and may be prohibitively expensive. I still need to do some research on that. So for now lets not worry about the GPU.

On the CPU side, I can't even find the 2637V2 on Newegg. Searching around the web it seems to be a $1k part, 4 cores each, at 3.5Ghz for a total of $2k.

I'm thinking about suggesting a single E5-2630 (6 cores @ 2.2 ghz) for $650 instead, and allowing room in the future for a second CPU upgrade with the savings. I am not familiar with how fast the prices on these intel xeons drop over time to know if this is feasible. Is there a 'sweet spot' for xeon performance vs. cost?

On the hard drive side, I'm not sure I see the point in the extra $400 controller and the 2 15K rpm drives. Those 600GB drives are $300/each on newegg, which is pretty close to SSD prices.

I would think you would get far better performance for the money with 2 512GB SSDs in RAID0 right off of the motherboard, no? For the $1200 their setup costs ($400 controller card, 2x$300 600GB drives), I would much rather run 2 256GB SSD or 3x128 in RAID0 as my work space with a 2TB drive for storage (total ~$500) and put the rest into CPU and RAM instead. Admittedly I don't have experience to know if the controller card is necessary or not. I highly doubt we'd be doing anything more than 2 drive RAID0 as scratch space.

Thanks for the help guys.
 

LurchFrinky

Senior member
Nov 12, 2003
303
57
91
What do you have now?
How well does the software scale, and how far?
When you say "some people use more" [memory], how much more?
How many computers are we talking about?
Can I assume these are Windows machines located right next to the users?

I ask because FEA should scale very well to many many processors (like clusters), but this is when Massively Parallel Processing (MPP) is used. Single workstations typically use Symmetric Multi-Processing (SMP), which doesn't scale as high. I would normally suggest moar coars, even if they are a little slower, but be sure to check case studies with your actual software.

As you seem to be aware, if you run low on memory, performance sucks. Setting up scratch space is a band-aid. If you can afford it - get enough memory! If not, I would suggest a single SSD to get you through the current job and make a mental note not to make your jobs quite so big.

If everyone is running big jobs all the time, then I would suggest looking into a dedicated server/cluster so you only have to max out the memory on one machine.

That graphics card is a bit pricey and there are a lot of professional cards that are cheaper, but I won't second guess something that has such a direct impact on productivity. Are you planning on using this for GPU compute?

I would also suggest against planning on upgrading later. By the time you would think about upgrading these machines, you will want new ones instead.

If this is for work, don't worry too much about the price. You are going to want (need) many high performance cores, a lot of registered memory, and a nice professional video card, and these things are not cheap.
 

hoorah

Senior member
Dec 8, 2005
755
18
81
What do you have now?
How well does the software scale, and how far?
When you say "some people use more" [memory], how much more?
How many computers are we talking about?
Can I assume these are Windows machines located right next to the users?
Currently, most users are working with intel 6 core xeons with 12GB of ram and standard spindle disks.

The software will scale with as many cores as you can throw at it with the caveat that each additional core (over 2) requires an additional license from our pool. If I work at night, I will hog all the licenses and run 8-12 CPUs. During the day, when I have to share, the most I will usually take is 4, but others are less considerate or have close deadlines and may take up to 8. There is talk of expanding our number of licenses, but nothing yet.

I'd say some people will use up to 128GB of ram, but usage is always increasing.

We're talking about probably 20-40 computers.

These are windows machines located at the users desks.

While running in memory is always the best method, sometimes it just doesn't happen. Sometimes we will run a model that refuses to run unless it pages. Sometimes the time it would take to rebuild the model (to get it to run in RAM) would be more than just letting it chug over the weekend while paging to the disk. As much as it makes sense to say "only run within ram" it just doesn't always happen.

If everyone is running big jobs all the time, then I would suggest looking into a dedicated server/cluster so you only have to max out the memory on one machine.

That graphics card is a bit pricey and there are a lot of professional cards that are cheaper, but I won't second guess something that has such a direct impact on productivity. Are you planning on using this for GPU compute?

We do have a cluster, but it is owned and maintained by a different department. We use it on occasion, but past history says its more efficient to have each user having a powerful workstation.

We are going to be testing a single system with the GPU computational capabilities of the K5000 and see if its worth it for all the machines. Even though most of the information on GPU computing suggests huge gains in number crunching ability, the software we use is quoting improvements of 15-25%.

The biggest thing I want to change on this project is the $1200 spindle disk RAID0 setup for a few SSDs in RAID0, but not knowing much about those higher end controllers and drives I wnat to be sure I know what the benefit is.
 

LurchFrinky

Senior member
Nov 12, 2003
303
57
91
Trust me, you are not the first to say it is easier to use your own machine over sending jobs to a cluster.

I did have to ask about scaling, though, because some users are not always familiar with the differences between SMP and MPP and the ramifications of Amdahl's law. I would expect the cpu usage to scale linearly with more cores, but what I would like to know is whether the time it takes to complete is also linear. Usually, using up to 4 cores will yield linear results (2 cores = 1/2 time, 4 cores = 1/4 time). But at some point there will be a flattening of the curve and you may find that 8 cores is just a little faster than 4, and 12 cores is actually slower! I don't know if this is the case with your current setup, but if it is, you might want the cpu decision to be weighted more towards higher clock. If you do have nearly linear scaling, you will want more cores (and licenses).

As for the drives, there is a difference between random and sequential IO. Platter drives - especially high-speed SAS - are good at sequential IO, but not so much random. I suspect that the scratch space usage may be sequential in nature and the SAS drives in RAID may outperform SSDs in this scenario. I will let you know that I am not an expert on this, so I may be wrong.
But, if these are going to be at your desk, I would totally suggest a single SSD. The noise of the SAS drives may be as loud as your cpu fans at 100%. Plus, I am not so convinced that the speed increase of the SAS drives is worth the $. Either the SAS or SSD will be a huge improvement over your current situation, but, again, if you are frequently using your scratch space, you are not using your resources efficiently.

Have you considered building a server (not cluster), for bigger jobs? If you have some way to predict which jobs will be pushing the limits of your workstations, then transfer them to the monster server. If you can shave $1000 off of each workstation, you can build a machine with 512 GB of memory (or more), I don't know how many cores (16?), and a few SAS drives - just in case. Stick this in a closet or corner cubicle and you should be more productive all around.
 
Last edited:

LurchFrinky

Senior member
Nov 12, 2003
303
57
91
Looking at your original post again, if someone specified 8 high-speed cores over 12 or 16, they may know how well your app scales. Adding in that the SAS drives might give better performance than SSDs, I think the vendor might know what he/she is talking about.

Additionally, the server idea I suggested may not be so great if you can't take advantage of more cores. It's still not bad if you want to have one machine maxed on memory, but you would have to look at how often people would need it versus the trouble of managing users and sending jobs to another machine.
 

hoorah

Senior member
Dec 8, 2005
755
18
81
Trust me, you are not the first to say it is easier to use your own machine over sending jobs to a cluster.

I did have to ask about scaling, though, because some users are not always familiar with the differences between SMP and MPP and the ramifications of Amdahl's law. I would expect the cpu usage to scale linearly with more cores, but what I would like to know is whether the time it takes to complete is also linear. Usually, using up to 4 cores will yield linear results (2 cores = 1/2 time, 4 cores = 1/4 time). But at some point there will be a flattening of the curve and you may find that 8 cores is just a little faster than 4, and 12 cores is actually slower! I don't know if this is the case with your current setup, but if it is, you might want the cpu decision to be weighted more towards higher clock. If you do have nearly linear scaling, you will want more cores (and licenses).

As for the drives, there is a difference between random and sequential IO. Platter drives - especially high-speed SAS - are good at sequential IO, but not so much random. I suspect that the scratch space usage may be sequential in nature and the SAS drives in RAID may outperform SSDs in this scenario. I will let you know that I am not an expert on this, so I may be wrong.
But, if these are going to be at your desk, I would totally suggest a single SSD. The noise of the SAS drives may be as loud as your cpu fans at 100%. Plus, I am not so convinced that the speed increase of the SAS drives is worth the $. Either the SAS or SSD will be a huge improvement over your current situation, but, again, if you are frequently using your scratch space, you are not using your resources efficiently.

Have you considered building a server (not cluster), for bigger jobs? If you have some way to predict which jobs will be pushing the limits of your workstations, then transfer them to the monster server. If you can shave $1000 off of each workstation, you can build a machine with 512 GB of memory (or more), I don't know how many cores (16?), and a few SAS drives - just in case. Stick this in a closet or corner cubicle and you should be more productive all around.

I would favor a higher clock over more cores simply for the times where I can't get enough licenses to run with more cores.

As for how it scales (overall time), it can be linear or it can be all over the place depending on the analysis. Some take lots of CPU time, but you only need one result from them, and all of the time is CPU time. Others have to write the results to the disk (20-60GB) before moving on to the next data point, which can take awhile.

The quote comes from a vendor that does work in our industry, so I suspect they do know what they are doing to a degree, but the SAS drive setup just didn't seem right.

The reason I mentioned the upgrades as is because while deciding on new work systems, we're also trying to decide on a refresh cycle as well. I'd love a monster system with 12 cores and tons of ram, but not if it means I have to keep it for 8 years.

Thanks for the help. I think I am going to recommend staying with the higher clocked, lower core CPU provided something else doesn't jump out at me and look at some benchmarks on those SAS drives. If the SAS drives are slower than SSDs, I will recommend an SSD RAID0 over the SAS setup.
 

mfenn

Elite Member
Jan 17, 2010
22,400
5
71
www.mfenn.com
One thing to keep in mind about going to 1 CPU is that that halves the number of available RAM slots. Most workstation boards only have 4 slots per CPU, so that would take you down to 64GB from 128GB. You should definitely only be looking at the Ivy Bridge-EP parts (E5-xxxx v2), not Sandy Bridge.

Speaking of RAM, 128GB per workstation is sucking up a pretty big part of the budget. Does your usage go over 64GB often enough to justify it?

As for storage, I agree that the dedicated RAID card is a bit of a waste. Since you're running Windows, you can leverage the built-in RAID with SSDs. Grab a couple of highly-consistent SSDs like the Intel S3700 and you should get much higher performance even when your models swap to disk.

Finally, 20-40 high-end workstations is a significant purchase. Definitely negotiate as you can.