Question Choice of computer hardware for running simulation batches?

[drchr]

Hi, I'd like some inputs on a computer build I'm going to do.

1. Typical tasks:
Running batches with hundreds of simulations that use the CPU, i.e. not the graphics card. The software is MATLAB/Simulink if anyone wonders. A batch run could take 8-12 hours, so temporary CPU boost doesn't really help. So it's not for gaming.
The computer will be near my desk, but might not fit, so I expect to connect via SSH/remote desktop to it.

CPU: Due to the workload, the more cores the better. I expect a Ryzen 3900X to be suitable. If the 3950X had been released, I would consider it (and bump the budget accordingly)
Cooling: Likely important as a task will run for many hours.
SSD: Fast SSD is needed. Though not expecting to need Optane or PCIe 4.0.
Nice-to-haves: Less loud. Smallish chassis.
Graphics: Not important at all. Cheapest alternative should be fine.
Fanless is preferred. The task does not include gaming.

See towards the end for questions.

1. Budget: $1200-$1400
2. Country: Sweden
3. Vendors: Several possibilities, some examples. I'm more looking for thoughts on component choices though. URL: https://www.dustin.se/ or https://www.inet.se/ or https://www.komplett.se/ or https://www.webhallen.com/ etc.
4. Brand preference: No preference.
5. Reuse parts: No
6. Overclocking: No manual OC.

Would likely let the system boost CPU frequency though. Maybe I'll have to do something about RAM speed though, depending on the motherboard.

1. Graphics resolution: Not relevant. Expecting to primarily connect to machine via SSH or remote desktop.
2. When: As soon as possible, baring availability of CPUs
3. Software for computer: Plan is e.g Ubuntu, but I might end up using Windows Pro.

Example build, mini-ITX:

• CPU: Ryzen 3900X
• CPU cooler: Wraith Stealth cooler
• Motherboard: Something using the B450 chipset [or is it too cheap?] Don't know what to pick here.
• Case: Fractal Design Define Nano S
• RAM: At least 32 GB, perhaps something with CL16?
• SSD: Samsung Evo 970, at least 500 GB
• PSU: Don't know, guessing I heed >= 400 Watts, but it's a guess.

Questions:

1. Any thoughts/objections on CPU choice?
2. Is it a bad idea to use the B450 chipset, could I e.g. expect stability issues when running for a long time? Would it be worth the extra money to go for X570?
3. I suspect I should get a better cooling system. But what? Air cooled or water cooled?
4. RAM speed. 3600 MHz? Should I expect CL16 vs e.g. CL18 to make an actual difference?
5. Should I got with mATX or ATX instead of mini-ITX for some reason?

E.g. better cooling performance or something? Or perhaps it results in a lower total cost?

cheers,

 

[Markfw]

You want a fast SSD ? then you want a gen 4 NVME like the Corsair, $230 last I looked(one tb) , but I don't know about your suppliers. That means an x570 motherboard though. But you can get them in the US for $200, a decent one. I can link later if needed. Ram, 3600 cl16 or better preferred. ATX pretty much required. PSU ? The biggest thing is you need 2 CPU 8 pin connectors. You CPU choice sounds good.

This is the least expensive PSU I could find that is platinum or titanium and has 2 CPU https://www.newegg.com/evga-supernova-220-p2-0650-x1-650w/p/N82E16817438055

 

[drchr]

Thanks for the tip about 2 CPU connectors, I hadn't thought about that.

Regarding SSD: One simulation is perhaps 5-10 minutes, producing perhaps 1 GB of data.
So the SSD ought to be able to save that in less than a second. Therefore I expect a PCI 4.0 on the x570 to not really do much for that part.
A fast SSD might help with loading programs (and compiling, as the Simulink mode is running in accelerated mode and there's a significant "build" process before the simulation really starts that actually takes 10-30 seconds). Hopefully the regular disk caching will be able handle this just fine, although I've been playing with the idea of using 10-15 GB for a RAM for the programs and compiled results.

 

[drchr]

Regarding PSU, why "platinum or titanium"?
Environmentally friendly to waste less, or is there some added benefit when running the CPU for long periods?

 

[Markfw]

Thanks for the tip about 2 CPU connectors, I hadn't thought about that.

Regarding SSD: One simulation is perhaps 5-10 minutes, producing perhaps 1 GB of data.
So the SSD ought to be able to save that in less than a second. Therefore I expect a PCI 4.0 on the x570 to not really do much for that part.
A fast SSD might help with loading programs (and compiling, as the Simulink mode is running in accelerated mode and there's a significant "build" process before the simulation really starts that actually takes 10-30 seconds). Hopefully the regular disk caching will be able handle this just fine, although I've been playing with the idea of using 10-15 GB for a RAM for the programs and compiled results.

If you don't want to spend the money on the fast SSD and the x570 you also loose future upgrade-ability. 32 gig of ram should be enough, but in your situation, make sure the motherboard has 4 dimm slots and get 2 16 gig dimms,so you can go to 64 gig later if needed.

As for the platinum or titanium, its not just efficiency, they are also better quality.

One other thing I had not mentioned. Threadripper may be a good fit for you. I know the current generation is soon going to be releasted, but get the cheapest 16 core/43 thread cpu now, and upgrade when they come out. The motherboards are a little more. The 2950x is $665 or less (not much more than the 3900x) and has more cores. The motherboards are not too bad either. Same PSU would work, and same ATX case.

 

[Flayed]

The Corsair Gen 4 1TB SSD is $325 in Sweden :O

 

[Flayed]

Gone over budget but would be a very nice system ($1709.57)
PCPartPicker Part List

CPU: AMD Ryzen 9 3900X 3.8 GHz 12-Core Processor (kr6091.00 @ Dustin Home)
CPU Cooler: be quiet! Pure Rock 51.7 CFM Sleeve Bearing CPU Cooler (kr453.00 @ Dustin Home)
Motherboard: MSI MPG X570 GAMING PLUS ATX AM4 Motherboard (kr2449.00 @ Dustin Home)
Memory: Corsair Vengeance LPX 32 GB (2 x 16 GB) DDR4-3200 Memory (kr1949.00 @ CDON)
Storage: Corsair MP600 Force Series Gen4 1 TB M.2-2280 NVME Solid State Drive (kr3130.00 @ Dustin Home)
Video Card: Gigabyte GeForce GT 1030 2 GB Silent Low Profile Video Card (kr1022.00 @ Dustin Home)
Case: Fractal Design Focus G ATX Mid Tower Case (kr449.00 @ CDON)
Power Supply: SeaSonic FOCUS Plus Gold 550 W 80+ Gold Certified Fully Modular ATX Power Supply (kr949.00 @ CDON)
Total: kr16492.00
Prices include shipping, taxes, and discounts when available
Generated by PCPartPicker 2019-09-08 02:39 CEST+0200

 

[Flayed]

A more wallet-friendly option. Only 8 cores / 16 threads with roughly 20% lower CPU performance compared to the 3900x

PCPartPicker Part List

CPU: AMD Ryzen 7 2700X 3.7 GHz 8-Core Processor (kr3099.00 @ CDON)
CPU Cooler: be quiet! Pure Rock 51.7 CFM Sleeve Bearing CPU Cooler (kr453.00 @ Dustin Home)
Motherboard: ASRock B450 Pro4 ATX AM4 Motherboard (kr1099.00 @ CDON)
Memory: Corsair Vengeance LPX 32 GB (2 x 16 GB) DDR4-3200 Memory (kr1949.00 @ CDON)
Storage: Crucial P1 500 GB M.2-2280 NVME Solid State Drive (kr869.00 @ CDON)
Video Card: Gigabyte GeForce GT 1030 2 GB Silent Low Profile Video Card (kr1022.00 @ Dustin Home)
Case: Fractal Design Focus G ATX Mid Tower Case (kr449.00 @ CDON)
Power Supply: SeaSonic FOCUS Plus Gold 550 W 80+ Gold Certified Fully Modular ATX Power Supply (kr949.00 @ CDON)
Total: kr9889.00
Prices include shipping, taxes, and discounts when available
Generated by PCPartPicker 2019-09-08 02:47 CEST+0200

 

[drchr]

If you don't want to spend the money on the fast SSD and the x570 you also loose future upgrade-ability. 32 gig of ram should be enough, but in your situation, make sure the motherboard has 4 dimm slots and get 2 16 gig dimms,so you can go to 64 gig later if needed.

I appreciate the tip regarding four DIMM slots btw. I am considering the importance of future upgrade-ability, although that's just making it more difficult for me to decide, but see my remark at the end. :-/

Regarding RAM use: I budgeted 2 GB for the OS etc, leaving (32-2)/12 = 2.5 GB/core.
That should be enough if I don't use SMT, in which case I might actually need more.
I should run a few simulations and measure actual RAM usage now, although it might of course increase in the future.

As for the platinum or titanium, its not just efficiency, they are also better quality.

Ok, makes sense.

One other thing I had not mentioned. Threadripper may be a good fit for you. I know the current generation is soon going to be released, but get the cheapest 16 core/43 thread cpu now, and upgrade when they come out. The motherboards are a little more. The 2950x is $665 or less (not much more than the 3900x) and has more cores. The motherboards are not too bad either. Same PSU would work, and same ATX case.

This sounds like an interesting idea...

Initially I actually looked at building a Threadripper based machine with 32 cores, but to be honest I got a little scared of screwing up regarding choosing motherboard, chassis, PSU, cooling etc.
This kind of high-performing HW is outside my experience :-(
Perhaps such a machine could be built for $5000, although perhaps what I really should do is start a separate thread asking about that?

Anyway, then I thought of the idea building say four "cheap" 3900X-machines instead of one expensive.
If one can be built for $1250 I'd get four of them for the same cost as one 32-core Threadripper machine.
Note: This doesn't account for the effort of assembling and maintaining multiple machines of course. Nor the extra work when having to split up say a set of 600 simulations into four sets of 150 simulations to be run separately on the different machines, but I think that part is easily manageable.

Performance wise I'm thinking 4 x 12-core 3900X would be better than a single 32-core 2990WX.
Also e.g. from a thermal point of view, spreading out the heat generation into four chassis.
These are just guesses from my side though.

Redundancy wise it might also be good with multiple machines. This connects back to the quality aspect. With 3-4 four machines, if say one PSU breaks it's no big deal to keep going with one less machine for a short while before a replacement PSU can be bought.

Yet another aspect is the software. I'll be using R2016b of MATLAB, so the code isn't going to be optimised for modern CPUs :-(. Today we're using a 6-core Xeon for simulations, and I can't be sure that Intel CPUs aren't for some reason significantly better when it comes to running binaries compiled in 2016.

It'd have been really nice if e.g. AMD had a service where I could pay them to temporarily get an account on machines, install the SW and run some actual tests of my particular use case. It'd be really interesting to compare performance between e.g. a 3900X and a 32-core threadripper. Oh well.

I have to say I really appreciate being able to discuss things like this with people who know what they're talking about!

 

[drchr]

A more wallet-friendly option. Only 8 cores / 16 threads with roughly 20% lower CPU performance compared to the 3900x

Given what I just explained in my other reply, this is actually quite interesting. Especially given that the 3900X is currently out of stock everywhere.

But.. is the performance really only 20% lower?
Looking at power/cores/base clock, and as tasks will run for several hours, I thought I could as a very rough performance indicator use the product of the number of cores and the base frequency.
This indicates that the 2700X performs at about 65% of the 2900X:

• Ryzen 2700X: 105 Watts for 8 cores at 3.7 GHz (29.7 core-GHz = 65%)
• Ryzen 3800X: 105 Watts for 8 cores at 3.9 GHz (31.2 core-GHz = 69%)
• Ryzen 3900X: 105 Watts for 12 cores at 3.8 GHz (45.5 core-GHz, = 100% = one "performance unit")
• Ryzen 2950X: 180 Watts for 16 cores at 3.5 GHz (56.0 core-GHz, = 123%)
• Ryzen 2990WX: 250 Watts for 32 cores at 3.0 GHz (96.0 core-GHz, = 211%)

I'm aware my performance estimation is very rough, but I haven't really found any useful benchmarks for the kind of task I'm doing.

PCPartPicker Part List

Thanks for the parts list!
I'd already partially started used PcPartPicker --- it's the first time I'm using it --- but now I'm putting in all the alternatives there. Haven't figured out how to copy/paste the list in the nice way you just did though.

I think overall I've got these long term paths forward, assuming a fixed total budget:

• 5 machines at <10 kSEK each that use the Ryzen 2700X. Total is 50 kSEK, for about 3.25 "performance unit".
• 4 machines at abt 15 kSEK each that use the Ryzen 3900X. Total is 60 kSEK, for about 4 "performance" units
• One machine at about abt 65-70 kSEK that use the Ryzen 2990WX Total is abt 65 kSEK for about 2.1 "performance" units.

If the Ryzen 2700X is only 20% less than the 3900X, then five 2700X gives the same performance as four 3900X while being 10 kSEK cheaper.

 

[Flayed]

Each core of the 2700x is roughly 20% less performant than the Ryzen 3 cores plus there are 50% fewer cores compared to the 3900x.

Click the BB button for copy/paste on Pcpartpicker

 

[drchr]

Ok, seems reasonable now.

Thanks for the BB tip.
I put together a wallet friendly option using the 3800X below.

To me it's surprising, and almost ridiculously so, how well my rough performance estimate match the estimated prices.

• 4 x Ryzen 3900X at 15 kSEK each = 60 kSEK, and 4 x 12 x 3.8 = 182 cores-GHz
• 6 x Ryzen 3800X at 10 kSEK each = 60 kSEK, and 6 x 8 x 3.9 = 187 cores-GHz

Given that I expect the 3900X to consume less power I'm surprised the 3900X doesn't cost more.


PCPartPicker Part List

CPU: AMD Ryzen 7 3800X 3.9 GHz 8-Core Processor (kr4661.00 @ Dustin Home)
CPU Cooler: be quiet! Pure Rock 51.7 CFM Sleeve Bearing CPU Cooler (kr453.00 @ Dustin Home)
Motherboard: ASRock B450 Pro4 ATX AM4 Motherboard
Memory: Corsair Vengeance LPX 32 GB (2 x 16 GB) DDR4-3200 Memory (kr2119.00 @ Dustin Home)
Storage: Crucial P1 500 GB M.2-2280 NVME Solid State Drive (kr958.00 @ Dustin Home)
Video Card: Asus Radeon R5 230 1 GB Video Card (kr604.00 @ Dustin Home)
Power Supply: Fractal Design Integra M 450 W 80+ Bronze Certified Semi-modular ATX Power Supply (kr758.00 @ Dustin Home)
Total: kr9553.00
Prices include shipping, taxes, and discounts when available
Generated by PCPartPicker 2019-09-08 13:36 CEST+0200

 

[Flayed]

Ok, seems reasonable now.

Thanks for the BB tip.
I put together a wallet friendly option using the 3800X below.

To me it's surprising, and almost ridiculously so, how well my rough performance estimate match the estimated prices.

• 4 x Ryzen 3900X at 15 kSEK each = 60 kSEK, and 4 x 12 x 3.8 = 182 cores-GHz
• 6 x Ryzen 3800X at 10 kSEK each = 60 kSEK, and 6 x 8 x 3.9 = 187 cores-GHz

Given that I expect the 3900X to consume less power I'm surprised the 3900X doesn't cost more.


PCPartPicker Part List

CPU: AMD Ryzen 7 3800X 3.9 GHz 8-Core Processor (kr4661.00 @ Dustin Home)
CPU Cooler: be quiet! Pure Rock 51.7 CFM Sleeve Bearing CPU Cooler (kr453.00 @ Dustin Home)
Motherboard: ASRock B450 Pro4 ATX AM4 Motherboard
Memory: Corsair Vengeance LPX 32 GB (2 x 16 GB) DDR4-3200 Memory (kr2119.00 @ Dustin Home)
Storage: Crucial P1 500 GB M.2-2280 NVME Solid State Drive (kr958.00 @ Dustin Home)
Video Card: Asus Radeon R5 230 1 GB Video Card (kr604.00 @ Dustin Home)
Power Supply: Fractal Design Integra M 450 W 80+ Bronze Certified Semi-modular ATX Power Supply (kr758.00 @ Dustin Home)
Total: kr9553.00
Prices include shipping, taxes, and discounts when available
Generated by PCPartPicker 2019-09-08 13:36 CEST+0200

Looks good. One thing is that the 3800x is not generally recommended because its performance is so close to the cheaper 3700x.

The other thing is that the B450 motherboards need a recent bios to support Ryzen 3. AMD have said they will ship out "loaner" cpu's to customers who are in this position. There are some motherboards that support a feature whereby you can flash the bios without a CPU installed.

 

[drchr]

Hi,

Update: Due to output of stock etc and trying to keep to one supplier, I'm tomorrow planning on ordering the build shown below.

I ended up going with a non-budget SSD, and also an X570 motherboard because it was in stock (and not to pricey actually).
Picking the case was the most difficult part as I have no clue what I'm getting lol.

I expect I'm also setup for switching to a X3950 once they appear.

Thanks for your help. Once I've run some simulations I'll try to update this thread with the outcome.

PCPartPicker Part List

CPU: AMD Ryzen 7 3800X 3.9 GHz 8-Core Processor (kr4661.00 @ Dustin Home)
CPU Cooler: be quiet! Pure Rock 51.7 CFM Sleeve Bearing CPU Cooler (kr453.00 @ Dustin Home)
Motherboard: Asus PRIME X570-P ATX AM4 Motherboard (kr2300.00)
Memory: Corsair Vengeance LPX 32 GB (2 x 16 GB) DDR4-3200 Memory (kr2119.00 @ Dustin Home)
Storage: Samsung 970 Evo Plus 1 TB M.2-2280 NVME Solid State Drive (kr2729.00 @ Dustin Home)
Video Card: Gigabyte GeForce GT 1030 2 GB Silent Low Profile Video Card (kr1022.00 @ Dustin Home)
Case: Fractal Design Define C ATX Mid Tower Case (kr1089.00 @ Dustin Home)
Power Supply: Fractal Design Integra M 450 W 80+ Bronze Certified Semi-modular ATX Power Supply (kr758.00 @ Dustin Home)
Total: kr15131.00
Prices include shipping, taxes, and discounts when available
Generated by PCPartPicker 2019-09-09 17:16 CEST+0200

 

[DrMrLordX]

3900x chips are hard to get right now. Good thinking on the 3950x, though when you get one, you may need a different HSF than that Pure Rock.

 

[drchr]

Hi,

Sorry for being slow to report on the eventual results.

I appreciate all the help and discussion feedback.

We ended up building a few computers with 3900X CPUs, each with 64 GB of RAM. We did this before e.g. the 3950X was released, so we never considered using that CPU.

RAM turned out to be a bottleneck and it ended up being very useful that I first built an 8-core test machine (3800X CPU) with 32 GB of RAM. The 32 GB of RAM only allowed me to fully use seven out of eight cores, and that was just barely.

Note: It turns out Mathwork's parallel computing toolbox (PCT) on Linux does _not_ gracefully handle running out of memory, in my case effectively resulting in the simulation suite crashing. This might be
better in later/future MATLAB releases of course.

Anyway, using up about 32 GB for 7 parallel tasks implies needing about 4.5 GB per task. So with 12 cores (tasks) we need 54 GB, impliying 64 GB must be installed.

With 16 cores (i.e. 3950X), we'd need 72 GB, and would then have to install 128 GB.

With 32 cores (i.e. a Threadripper), we'd need 144 GB, and would then have to install 256 GB.

From the point of view of memory for this application, the 3900X hit a nice spot. A 24-core Threadripper ought to have worked well with 128 GB. It is of course also possible to simply not use all the available cores.

Regarding memory speed, I did try increasing it but the gains were small. So the CPU cache and available memory BW seems to be working fine as there was only a small improvement.

In summary, we're happy with the HW choice and the significantly increased simulation performance.


Below is some hard data in case it helps someone else. However, results are likely very application dependent, so I recommend building a test machine first.

The application used here as a test benchmark is a simulation suite of about 150 simulation runs done using Mathworks' parallel computing toolbox and a Simulink model in accelerated mode. OS is Ubuntu.

The test takes about 1-2 hours to run depending on the HW. Other real simulation suites might have runs of several hours each.

As a side note, an older (Windows) machine that was previously used took seven hours to complete this test run. That's weird, and slower than expected, so I'm not including that data.


The data below is for seven runs of the same simulation suite. It's ran on a simple desktop machine with an i5-8500 CPU for reference, and then on a machine with first an 3800X CPU and then a 3900X CPU.

Runs #2 and #3 on the 3800X could only use 7 out of 8 cores due to lack of memory. Then for run #4 memory was increased and 8 cores were used.

Memory speed was increased in runs #5 and #6, with only a small speed increase.

The table also includes a performance estimate based on the number of cores and the base frequency. It worked surprisingly well in order to predict performance.

The passmark benchmark is also listed for the CPUs. When not all cores could be used, the value was scaled down accordingly.

Suite run       #1      #2      #3      #4      #5      #6      #7
Configuration    cfg-1   cfg-1   cfg-1   cfg-2   cfg-3   cfg-4   cfg-5
Duration [s]    7588    4650    4653    4061    4043    4014    2973
Speedup  [:]    1       1.63    1.63    1.87    1.88    1.89    2.55

CPU      [-]    8500    3800X   3800X   3800X   3800X   3800X   3900X
Cores    [-]    6       7 (8)   7 (8)   8       8       8       12

Base f. [GHz]   3.0     3.9     -       3.9     -       -       3.8
[cores x GHz]   18.0    27.3    -       31.2    -       -       45.6
Ratio:   [-]    1       1.52    -       1.73    -       -       2.53

RAM      [GB]    2x16    2x16    2x16    4x16    4x16    4x16    4x16
.. speed [MHz]  ?       2133    2133    2133    2666    3200    2133
OS       [-]    Ubuntu  Ubuntu  Ubuntu  Ubuntu  Ubuntu  Ubuntu  Ubuntu

Passmark [-]    11824   21500   -       24600   -       -       31859
         [-]    -       (24600) -       -       -       -       -
Ratio:   [-]    1       1.82    -       2.08    -       -       2.69

A final note regarding cooler: Due to lack of time, and to not mess with a working system, we ended up only using the stock cooler.

In theory switching to a better cooler might improve performance.

As temperature measurement wasn't working with use Linux kernel, I can't say much about the temperatures.

The machines seem to work in general though.

 

[DrMrLordX]

As temperature measurement wasn't working with use Linux kernel, I can't say much about the temperatures.

That's interesting. Which version of Ubuntu are you using, and which kernel version is featured in that release?

 

[drchr]

That's interesting. Which version of Ubuntu are you using, and which kernel version is featured in that release?

Ubuntu 19.10, I think e.g. kernel version 5.3.0.23.
However, the just released 5.4 kernel should have fixes to let hwmon get correct temperatures.