Multi-core and compiling software

[think2]

Using GCC or similar command line C++ compiler, does software build faster on a machine with multiple cores or does it require special software to initiate multiple source compilations at a time?

On Windows XP, my single core machine is fairly non-responsive when I'm compiling. Would this be different with a multi-core machine so that I wouldn't lose responsiveness when building software?

 

[Concillian]

Compiling in Linux I'll use 'make -j#'
where # = number of threads for the compiler to use.

on a dual core, using a value of 2 is significantly faster than 1.
Using available cores minus 1 will make it responsive, as it won't be using all cores, the OS will be able to have a core available to attend to your demands (but compilation will obviously take longer than using all available cores).

The other issue is a single core anything is slow compared to "real" CPUs, even if you are running single threaded applications that don't take advantage of multi-core. You can probably get a significant increase just using a decent CPU and not using the command line option that will spawn more compiling threads.

What is your current CPU?

 

[mv2devnull]

Yes and no.

On larger project (multiple compilation units) one naturally does use something equivalent to 'GNU Make', and that make definitely can launch multiple copies of the compiler. The speedup can be substantial.

If one requests so many parallel processes that resources do exhaust, then one does lose responsiveness.

 

[TheRyuu]

If one requests so many parallel processes that resources do exhaust, then one does lose responsiveness.

nice -n 19 make -jx

Where x is your number of logical processors + 2 (so on my i7 with HT it would be -j10).

 

[think2]

My current machine is a single core 3GHz pentium something, about 4 or 5 years old. I'm wondering how a dual core Intel i3 550 with hyperthreading would compare with an AMD phenom II X4 quad core. e.g. with the i3, if one core is used for compiling and the other core for other tasks so the machine is still responsive, it might not be much faster than a single core machine.

 

[Mr. Pedantic]

No, it will be. Clock for clock more than twice as fast I'd estimate, with the IPC improvements and HT added on. Plus, you technically have 4 threads, so spawn 3 threads - having a single thread for OS is plenty. Or, just do what Ryuu suggests and lower the priority to max the i3 out.

 

[TheRyuu]

You should always give make more threads than you do logical cores (thus the cores + 2 works good).

So if you have 4 logical cores as in your above example, you should give make 6 threads (-j6).

 

[Concillian]

My current machine is a single core 3GHz pentium something, about 4 or 5 years old. I'm wondering how a dual core Intel i3 550 with hyperthreading would compare with an AMD phenom II X4 quad core. e.g. with the i3, if one core is used for compiling and the other core for other tasks so the machine is still responsive, it might not be much faster than a single core machine.

Processing only one thread, which gives no advantage to a multi-core processor, an i3 550 is at least twice as fast as a 3 GHz Pentium4. So no matter what, you will have a machine that is going to significantly cut your compile times.

Bench isn't old enough to have many Pentium4 CPUs in it. This is a 3.6 GHz P4 vs. an i3 540:

http://www.anandtech.com/bench/Product/143?vs=92

Check the cinebench single threaded benchmark, the i3 is almost twice as fast when you cripple it down to only using one thread. Take 20% speed from the 3.6 GHz P4 and add 5% to the i3 for a 550 model, and you're well over double the speed. Consider that with Hyperthreading you have 4 threads available, and you're looking at compile times of 1/4th or less of what you have now and still retaining some responsiveness.

Here's a compare on a Phenom x4 955 vs an i3-540.
http://www.anandtech.com/bench/Product/143?vs=88

You can see the i3 usually loses, however if you remove one thread to retain system responsiveness, the i3 gives up less than the Phenom will. I expect they'll be pretty close at 3 threads.

I do more stuff that only uses 1-2 cores, so I chose the i3. In your case the Phenom might be better, but either choice will cut your compile times dramatically.

 

[Concillian]

You should always give make more threads than you do logical cores (thus the cores + 2 works good).

So if you have 4 logical cores as in your above example, you should give make 6 threads (-j6).

If you want to minimize compile time, 'available logical cores +1' is fine, but it seems like his concern is keeping the system somewhat responsive during a long compile, presumably so he can spend some time web browsing or something while it compiles. In this case he might prefer a longer compile time and using 'available logical cores -1'

Edit:
Nevermind, you're right, the correct way to give responsiveness is to use the -nice flag so it will yield resources quickly, but still use all of the processor when it can.

 

[Idontcare]

You should always give make more threads than you do logical cores (thus the cores + 2 works good).

So if you have 4 logical cores as in your above example, you should give make 6 threads (-j6).

This. Even in the days of single-core cpu's my compile times would be better if I used the -j2 flag.

 

[mv2devnull]

I just remembered an incident from the distant past: dual-CPU machine, so two cores. make -j4. OOM. Out Of Memory. Well, there probably was only 128 MB to run out of. Not a problem these days.

 

[Blastman]

Using GCC or similar command line C++ compiler, does software build faster on a machine with multiple cores or does it require special software to initiate multiple source compilations at a time?

If the compiler doesn't support multithreading then it will only use a single CPU no matter how many CPU's you have -- even on a quad-core.

No special software required for multiple CPU support on Windows. Windows XP handles multiple threads automatically if the software (compiler) supports multithreading and the CPU has those threads available.

On Windows XP, my single core machine is fairly non-responsive when I'm compiling. Would this be different with a multi-core machine so that I wouldn't lose responsiveness when building software?

Yes -- a multi-core CPU can make a world of difference.

When I run QB (QuickBasic) simulations (that take 2+ hours to run on my i3-530) I can go off and surf the web or run other applications and my computer is still very responsive. On my old Celeron-T (1.54Ghz), since it only had 1 logical processor the computer was very unresponsive when I tried running my QB simulations. Basically I had to go for coffee or find something else to do.

My i3-530 has 4-logical processors so I can run 3 instances of QB simulations at the same time and my computer still responds well. As long as 1 logical processor is free, the computer still responds well. The i3 is a big upgrade in this respect even from a dual-cores because I would only be able to run 1 QB simulation on a dual core and would have to leave the 2nd core unused if I wanted to still use the computer and have it respond well. I find being able to run 3 QB simulations vs 1 (for a dual core) at the same time is a big upgrade in overall throughput.

 

[cytg111]

http://techreport.com/review/23750/amd-fx-8350-processor-reviewed/10

 

[Phynaz]

Nice necro

 

[cytg111]

damn .. dont ask!

 

[joshhedge]

This. Even in the days of single-core cpu's my compile times would be better if I used the -j2 flag.

May I enquire as to why that is so?

I'm going to assume that the OS prioritises more CPU compute to the task if more threads are allocated?

 

[BrightCandle]

May I enquire as to why that is so?

I'm going to assume that the OS prioritises more CPU compute to the task if more threads are allocated?

Because compilation is also heavily IO driven, it does a lot of small reads and writes. These reads and writes happen before and after compilation work is done and the CPU sits idle while it happens. Thus giving it two things to work on put more parallel IO and work to do when the CPU was busy doing IO. Its this way with a lot of compilers.

 

[Idontcare]

May I enquire as to why that is so?

I'm going to assume that the OS prioritises more CPU compute to the task if more threads are allocated?

I'm not sure what bottleneck it relieved, just something that was empirically observed whilst performing a shedload of compiles over the course of the better part of a year.

edit: what BrightCandle said , it is a way of "hiding" the IO latency.