Best (linux) performance: Source-based or customized system?

[Heisenberg]

I'm trying to figure out which offers the best performance, and have bascially two options. The first is a distro like Gentoo that compiles everything from the source when it installs. The second is basically taking a distro like Debian, installing it, and then compiling a custom kernel, disabling unneeded services, and other general tweaking. I'm leaning toward source-based but am wondering if anybody has any real-world experience about just how much difference there is. This would be for a desktop machine, not a server.

 

[pac1085]

I would go with source based. Binary based distributions are usually compiled for i386, and dont take advantage of many of the new features avaliable on todays processers. You can still do things like compiling custom kernel, disabling services, and other tweaking with a source based distribution, so why not. I dont think that the performance gain will be THAT huge but I enjoy it.

 

[cleverhandle]

I can't see how Gentoo or similar would fail to offer a performance improvement - the programs are simply better optimized for your hardware, and in addition you can do all the same tweaks that you could do anywhere else. The question is whether it's going to be noticeable or not. I've used a variety of RedHats, Slackwares, and LFS systems on a couple of machines, and I haven't found there to a noticeable improvement in LFS compared to the others. Somebody posted some benchmarks some time ago to an LFS mailing list that showed a best-case improvement of about 10-15%, IIRC. But some programs improved much less. In my case, I think my machines are fast enough (a killer fast desktop, and a 1.13 Celeron w/256MB) that those speed boosts aren't significant. If your hardware is older, it might be more worthwhile - of course, it will then take a lot longer to compile, too.

Overall, I would say not to get too worked up over speed boosts from optimizations. They're nice to have, but if you're not going to appreciate the added work and time building everything up, don't bother.

 

[Nothinman]

You'll get very little performance boost and a lot of wasted time if you go with a 'build everything from source' distro. The biggest thing you can probably do is use fast hard disks and have a decent amount of memory.

Some things that are very CPU intensive or do a lot of floating point math can benefit from being rebuilt to use things like mmx or sse, but 9 times out of 10 the difference is negligable.

 

[GL]

This very question led me to recompile several Red Hat 8.0 SRPMS with optimization for my P4. The result? Don't believe the hype. If there is any speed advantaged enjoyed by source-based distros like Gentoo, over binary-based ones, very little of that is due to compiler optimizations. I recompiled glibc, XFree86, QT, kdelibs, kdebase, (of course the kernel) and the only time I felt a speed increase over the standard i386 binaries was with glibc and that was only on loading programs. I'll tell you what does make a noticeable user-land speed increase - altering the kernel system clock frequency from the default 100Hz to a higher value such as Red Hat 8.0s 512Hz or what I've used - 1024Hz. That provides a noticeablely more responsive userland experience.

 

[Nothinman]

...led me to recompile several Red Hat 8.0 SRPMS with optimization for my P4

You probably also ran into the 'Intel jacked up the clock-rate on the P4 but it's not any faster than a 1/2 clocked athlon' feature. For anything to run at full speed on a P4 it has to either be hand-tweaked asm or the compiler has to be very f'ing smart, and gcc just isnt there with P4 optimizations yet. You may get better results using Intel's compiler, but some things are dependent on gcc extensions (most notably the kernel) and won't compile with anything else.

 

[GL]

You may be right, and I did download the Intel compiler thinking I'd try to recompile some SRPMS with that...but I said forget it - it was just too much time. GCC isn't as bad as it used to be but still, the reordering of instructions for i686 should have brought some speed improvements but truthfully I noticed very little if any (the little may have been due to the "well I recompiled it so it must be faster" factor). I guess I'm glad I didn't have to compile all the stuff I have on my computer from scratch because it would largely have been a waste if the goal in mind was speedy performance.

 

[Heisenberg]

Sounds like there's very little to be gained by using a source-based distro. I'll probably end up just using something like Debian and customizing it as much as possible. Thanks for the good info.

 

[n0cmonkey]

If you have the time and need it could be useful. But most of us do not have either. By need I mean you run a webserver with 1000 sites all getting slammed at the same time and you already invested thousands into hardware.

Other tweaks that may show a gain in performance:
custom kernel
very few services
use an appropriate file system
raise the disk caching (I can do this in OpenBSD, but I dont know about linux)
faster processor
more ram
tmp in ram

 

[manly]

Originally posted by: Heisenberg
Sounds like there's very little to be gained by using a source-based distro. I'll probably end up just using something like Debian and customizing it as much as possible. Thanks for the good info.

The funny thing is a lot of people who've tried Gentoo swear by it. I've read at least a few reviews that claim the performance difference is noticeable, whatever that means.

While recompiling the kernel would have some performance benefits, I think some people forget the most performance-critical hot spots are already implemented in inline assembly (correct me if I'm wrong). Most distros these days are targeting i586 for PCs, so the code generation is already getting better. And all the popular distros offer kernels compiled for Athlon, i686, Pentiums, etc.

If performance on my 2-year old box was a problem, I'd probably look into recompiling certain crucial packages. But otherwise, compiling a whole OS from scratch seems like a mighty waste of time if eked out only 10% faster performance. Having said that, I'm not trying to knock what I haven't personally experimented with.

I wonder how GCC 3.2 factors into the picture; it's supposed to be a good improvement over GCC 2.95.x in performance so I'd like to see some objective benchmarks of last generation distros vs. the current crop.

 

[Sunner]

Well, I can give you the point of view of someone who's moved from Debian to Gentoo.

To me the big plus isn't speed, in a few cases you'll notice a speed increase, but most of the times there is none, not in my experience anyway.
I believe when most people talk about speed, it's more of a placebo effect.

The big plus however, is that the portage tree is kept very up to date.
Im running KDE 3.1 RC2, and IIRC it took less than 48 hours for it to be included in the portage tree after the release.

Debian OTOH doesn't even have KDE 3.0x in their unstable tree, sure you can get it from unofficial sources, but as far as Im concerned, that's not the same thing.
As for portage vs APT, both have pros and cons, APT has "reverse dependancy resolving" or whatever you wanna call it, if you remove KDE with apt-get it'll remove all the subpackages as well, portage won't do this.
But OTOH portage is better at handling several versions of packages on the same system, so when you run Gnome2, you can still keep all the old Gnome 1.4 libs around for programs that haven't been ported to GTK2, which is very handy.

All in all, I'd say Debian is more suitable for situations when you need stability above bleeding edge, such as a server, or if you simply feel that you don't need the latest and greatest on your workstation, while Gentoo will be far better if you're like me and like to play with the latest stuff.

Oh and the compile time thing, it isn't really a big issue IMO.
Initial setup will take some time, no doubt, compiling glibc, X, etc will take a load of time, but once you're done, you're set.
And when that new flashy version of KDE comes out, just compile it overnight, no biggie.

 

[N11]

We've found some pretty cool performance boosts with hyperthreading support in our Dual Xeon environments.

Performance wise you can never go wrong with a Red Hat or Suse stamped package. I'd leave compiling to maybe a few situations that require kernel tuning or compilation specifics -- other than that stick with the standards.

 

[Nothinman]

By need I mean you run a webserver with 1000 sites all getting slammed at the same time and you already invested thousands into hardware

If you have that many websites all getting a decent number of hits you're much better off to invest in a few machines and a load balancer. And there's a better chance that tweaking your own web sites code generation would speed things up, than simply recompiling apache.

Debian OTOH doesn't even have KDE 3.0x in their unstable tree, sure you can get it from unofficial sources, but as far as Im concerned, that's not the same thing.

Not to make excuses for Debian, but it's not included because they're transitioning from gcc 2.95.x to gcc-3.x and there's major C++ ABI changes so C++ apps are not binary compatible, this is pretty much KDE and all it's apps. Once gcc-3.x is the default compiler KDE 3 will be put in officially.

We've found some pretty cool performance boosts with hyperthreading support in our Dual Xeon environments.

Considering the price of Xeons, you had better.

 

[Sunner]

Not to make excuses for Debian, but it's not included because they're transitioning from gcc 2.95.x to gcc-3.x and there's major C++ ABI changes so C++ apps are not binary compatible, this is pretty much KDE and all it's apps. Once gcc-3.x is the default compiler KDE 3 will be put in officially.

Well, it's not just KDE, Debian has always been kinda slow to update things.
In the case of KDE, it's pretty extreme, even for Debian, but it seems like there's a reason then, I haven't been keeping track of Debian since I switched my workstation to Gentoo and my router and server to OpenBSD.

 

[Nothinman]

Well, it's not just KDE, Debian has always been kinda slow to update things.

I know, but KDE3 is special because it's been out for a decent amount of time and everyone makes a big deal about it.

Debian being slow to update is what keeps it sane. And things like Gnome2 are in sarge already, so not everything is lagging behind.

 

[Sunner]

Well, Gnome2 isn't exactly that new, and it only got pushed in Sarge a few weeks back, no?

Anyway, I agree, going a bit slower helps keep the quality and stability of Debian on the excellent level it is, and it's the reason I'd definately go with Debian if I were to pick a Linux distro for a server.

It's all just a "can't have it all" type situation

 

[TheOmegaCode]

I love Debian for it's packages. I can have things installed within seconds, rather than waiting 20+ minutes.
If I'm in the mood to compile something, I'll install a *BSD. That's just me though...

 

[Nothinman]

Well, Gnome2 isn't exactly that new, and it only got pushed in Sarge a few weeks back, no?

Gnome2 is only a few weeks into Debian. It took so long because of the migration testing, they wanted to make sure no Gnome1 apps would break because not everything has been ported to Gnome2, I know I'd be pissed if they broke Gnome1 apps because I use a few of the, namely Galeon and Evolution.

It's all just a "can't have it all" type situation

Totally true. But back on topic, neither distro has any real performance differences that stem from complication optimization.

 

[gaidin123]

I've been using Gentoo for most of a year now on a laptop, home router machine, and a desktop at work. I've also recently installed RedHat 8.0 on a couple desktops (and that same laptop again ).

I think that a decent amount of the "performance improvement" people claim from using a source based distro is the lack of extra stuff that distros like RedHat 8.0 install. RedHat 8 boots much slower for me than Gentoo but it's because RH8 loads a ton more services than I need or want. My Gentoo box only loads the services I need to use and so booting is faster, more system resources are available, etc. Gentoo has a base system layout but RH has a ton of utilities/other programs that you may never use regardless of which install type you run.

Both Gentoo Linux and RedHat 8 are very impressive to me in their own ways. I can say for sure that RH is less responsive on my Crusoe underpowered notebook than Gentoo but again that's probably due to the level of services running by default and the (I assume) more intensive GUI.

What's kind of funny is that I partly like RH because of its bloat. It's fun discovering various new utilities that you never knew existed.

Gaidin

 

[Heisenberg]

Originally posted by: TheOmegaCode
I love Debian for it's packages. I can have things installed within seconds, rather than waiting 20+ minutes.
If I'm in the mood to compile something, I'll install a *BSD. That's just me though...

I agree Debian's apt-get is very handy. We use Debian for a webserver at school and apt makes it easy to keep things up-to-date. The fact that it does depedencies automagically makes it superior to rpm, IMHO. The portage system Gentoo offers looks cool though. I use FreeBSD for my fileserver at home, so I'm familiar with its ports system.

I may give Gentoo a try just because. I have a PII with RH 7.3 on it now with a custom kernel, only necessary services running, and other general tweaking so it'll be easy enough to compare speed-wise. If I decide I don't like it, I can always re-image the machine in about 5 minutes and be back to RH.

 

[bromer]

Originally posted by: Nothinman
You'll get very little performance boost and a lot of wasted time if you go with a 'build everything from source' distro. The biggest thing you can probably do is use fast hard disks and have a decent amount of memory.

Some things that are very CPU intensive or do a lot of floating point math can benefit from being rebuilt to use things like mmx or sse, but 9 times out of 10 the difference is negligable.

Do you look at all the output you get when compling, since it wastes your time. I just tab to another window a continue with my other work

 

[Abzstrak]

the gentoo optimized kernels are pretty fast, but are heavily patched (so not considered stable by any means the way it would be in the debian side of things) ... personally I see a difference (about 20% or so) between my gentoo install over running redhat before it, probably just cause its cleaner and running on reiserfs.

The biggest difference i think is knowing exactly what is on your system and not having any uneeded crap running and having a fast filesystem... using xfs or reiser is ALOT faster than ext3 based distros....

compiling doesn't take all that long on a modern system.... its not like its going to be compiling for days or something.....

I personally consider it another tweak... Its on my home system, I overclock it as far as it can go and remain stable, and I can compile everything and get more power out of it... why not? I spend time on other tweaks, in comparison compiling is EASY....

I think you can get about as much of a speed increase in an average system as you can get from overclocking your system.... so do both and double your speed increase... worth it? I think so, but not everyone thinks its worth the trouble..... Its a matter of personal choice, some would rather spend a bit more on faster hardware and for go the tweaking...

 

[Nothinman]

Do you look at all the output you get when compling, since it wastes your time. I just tab to another window a continue with my other work

I have a SMP system with U160 drives so lack of multitasking isn't my problem. When I want to install software I want it installed so I can do whatever and decide whether or not to leave it installed now, not 20 minutes from now after it compiles. And if it's a new installation that's another ~45 minutes for X and a WM even on my box, no thanks. The time is wasted because the benefits from compiling it all just aren't there, all the added time waiting on things to compile and there's no real gains, what's the point?

personally I see a difference (about 20% or so) between my gentoo install over running redhat before it, probably just cause its cleaner and running on reiserfs.

Run it with a 'standard' kernel.org kernel (with whatever reiserfs patches you need to not f' your filesystem) and see if the performance drops. Or run RedHat with the Gentoo patched kernel and see if it helps, either would be a decent test.

and I can compile everything and get more power out of it... why not?

Because the compiling doesn't really get you anything unless you have specific patches that alter the program in a specific way that does get you speed gains, like all the kernel patches Gentoo applies.

I think you can get about as much of a speed increase in an average system as you can get from overclocking your system

Not really, you can atleast measure the gains when you overclock (i.e. I got another 300Mhz out of the chip) but when you compare software you compiled to software someone else compiled, it's hit or miss and all based on 'feel'.

 

[manly]

I sense one major reason for Gentoo's popularity is because Debian's development model is relatively sluggish.

Debian's stability and apt-get are great, but a lot of users prefer bleeding edge. Now I know Debian today with its multiple trees is much different from when I used it a lot, but I dropped Debian when they released 2.1 (Slink) that was dead on arrival. It was instantly obsolete as a desktop OS, and I looked elsewhere.

Even with the testing and unstable branches (which I admittedly do not follow these days), users want Gentoo because the ports are kept at the bleeding edge, for better or worse.