Need to patch a Linux kernel, and I'm scared!

[scootermaster]

Will someone hold me!?

It's my work computer, and there's already five or six kernels on the boot loader, and I even had them install a new hard drive in there so I can do it without messing anything up, but it's still scary.

Tell me this isn't a big deal!

 

[Netopia]

What's the bigger story? What's the mod, why is it needed, etc.

Joe

 

[degibson]

The good news is that, since you have 5-6 kernels already, theres a decent chance you won't screw them all up by patching one of them.

In truth, this isn't a big deal. There's very little that can go wrong that cannot be fixed. Just follow your directions very carefully, and if anything seems fishy, just ask here in the forums and we'll help you out.

Edit: If you do end up asking for help, make sure you specify the distro, the kernel version, HW, etc.

 

[Brazen]

Not that big of a deal, because even if it doesn't work, you just reboot and choose a prior kernel version.

 

[Cogman]

Just be careful because you will be playing with the boot sector (grub) when you get things done, that could be scary. As for patching the kernel, nothings to scary about that. If you do it right then it will compile without a complaint, if you do it wrong then your compiler will use a few derogatory terms and quit out on you. Most likely it wont work if you do it wrong.

 

[scootermaster]

Thanks for the reassurances!

Backstory: I need to install the PAPI performance counter software. It won't work without the kernel patch. There are only specific kernels that this perfctr will work for, but that's not a big issue.

Here's the issue(s):

a). I've never done this before
b). It's my work machine, which is good and bad
c). The good news is, if I eff it up, they'll come fix it for me, and for the most part, all my data is on an NFS server, so all I'd lose is a native compiler installation, which I could re-do if needed.
d). The bad news is, I don't even have the super-secret password for Grub to (presumably?) add the new HD/partition to the boot loader.
e). I don't have the root password, but I do have sudo access.
f). I think one (the 2.6.18) of the kernels that's installed (or at least appears in Grub) is compatible with the patch, but I don't think I have the kernel sources. I'd imagine I'd need those, right? =]

Again, my plan was to try to install it on the new hard drive, completely separate from everything else. I guess I can try running whatever the unix version of fdisk is (something tells me it might be called, uh, fdisk, but I'm not sure) and see if I can't format/partition the drive. But again, getting it to show up in Grub is another matter. I also don't have (yet?) any media (like, Fedora CDs or anything). Ideally, what'd be nice is to install via DVD/CD (to the new hard drive), then somehow get it to appear in grub, then patch the sources, then recompile, and voila!

But the not having total control over my machine makes things difficult. I don't even know what directories are local (the vaaast majority of everything is run off the NFS).

So that's my story.

 

[silverpig]

edit: n/m

 

[degibson]

Originally posted by: scootermaster
a). I've never done this before

First time for everything...

b). It's my work machine, which is good and bad
c). The good news is, if I eff it up, they'll come fix it for me, and for the most part, all my data is on an NFS server, so all I'd lose is a native compiler installation, which I could re-do if needed.

Wow that is good news indeed. You shouldn't be afraid at all.

d). The bad news is, I don't even have the super-secret password for Grub to (presumably?) add the new HD/partition to the boot loader.
e). I don't have the root password, but I do have sudo access.

Why? If you have the powers-at-be's permission, you should be provided whatever passwords you need. Granted, you have physical access to the machine, so passwords can be circumvented.

f). I think one (the 2.6.18) of the kernels that's installed (or at least appears in Grub) is compatible with the patch, but I don't think I have the kernel sources. I'd imagine I'd need those, right? =]

Yes, you'll need 'em. Look in /usr/src/linux, they're often there... not always I suppose. What distribution are you using...?

Again, my plan was to try to install it on the new hard drive, completely separate from everything else. I guess I can try running whatever the unix version of fdisk is (something tells me it might be called, uh, fdisk, but I'm not sure) and see if I can't format/partition the drive.

I'd recommend cfdisk. Its a bit easier to use than fdisk.

But again, getting it to show up in Grub is another matter. I also don't have (yet?) any media (like, Fedora CDs or anything). Ideally, what'd be nice is to install via DVD/CD (to the new hard drive), then somehow get it to appear in grub, then patch the sources, then recompile, and voila!

This is pretty doable, really. To do so, install to your fresh disk, but don't let the installer overwrite the existing MBR when it wants to. Boot one of your regular options (off the 'old' disk), and have grub add the other disk as a boot option to the MBR. You can even use a different distribution, if you like -- i.e. one that already has the patch, if you can find one (no patching required!).

But the not having total control over my machine makes things difficult. I don't even know what directories are local (the vaaast majority of everything is run off the NFS).

cat /etc/fstab

The entries prefixed with a server name (e.g. server: ) are NFS mounts. Also, the type is listed as 'nfs'.

 

[Nothinman]

Again, my plan was to try to install it on the new hard drive, completely separate from everything else. I guess I can try running whatever the unix version of fdisk is (something tells me it might be called, uh, fdisk, but I'm not sure) and see if I can't format/partition the drive.

That'll be a lot more work than you think, you'll essentially need a completely separate installation for that. Just install the kernel like normal, you can have as many installed at once as you want. Another installation in something like VMWare would make testing go quicker but if the second installation is on the bare hardware then you're still looking at the same amount of reboots as installing it along side the current kernels.

 

[scootermaster]

Originally posted by: Nothinman

Again, my plan was to try to install it on the new hard drive, completely separate from everything else. I guess I can try running whatever the unix version of fdisk is (something tells me it might be called, uh, fdisk, but I'm not sure) and see if I can't format/partition the drive.

That'll be a lot more work than you think, you'll essentially need a completely separate installation for that. Just install the kernel like normal, you can have as many installed at once as you want. Another installation in something like VMWare would make testing go quicker but if the second installation is on the bare hardware then you're still looking at the same amount of reboots as installing it along side the current kernels.

What is "like normal"? I've never "installed" a kernel before, so I'm not sure how hard that is, normal or otherwise.

I just figured that a).I could use the hard drive's space and b). having it "separate" might decrease the chance of messing something up.

But I need to just grit my teeth and do it, I guess. My IT people told me I have rpms of the sources for the kernels that are installed (as I said, I have like five or six on Grub) so I guess that's a good place to start.

Thanks for all the great advice!

 

[Nothinman]

What is "like normal"? I've never "installed" a kernel before, so I'm not sure how hard that is, normal or otherwise.

Just like any other package, there's nothing special about the kernel from that perspective. On Debian-based systems you can use make-kpkg to build a .deb from a kernel source tree and install it with dpkg.

I just figured that a).I could use the hard drive's space and b). having it "separate" might decrease the chance of messing something up.

Maybe a bit, but the chances of the kernel messing up bad enough to cause you more work than a reboot into an older kernel are pretty small unless you're doing development in something like the filesystem you're using or the block layer.

 

[degibson]

Again, my plan was to try to install it on the new hard drive, completely separate from everything else. I guess I can try running whatever the unix version of fdisk is (something tells me it might be called, uh, fdisk, but I'm not sure) and see if I can't format/partition the drive.

If you're really that scared and you have another disk, you can just look for an already-patched distro and install it, instead.

 

[Scarpozzi]

Just be aware. If you compiled any custom stuff on your current version, you'll have to recompile for your new kernel.

So if you did any custom compiling before, some stuff may not come back up online (application-wise), but it will probably just take a little work to get the bugs worked out.

 

[DarkThinker]

In your scenario, I think I have the perfect fix for you...
How about you skip patching the kernel in this sensitive situation that you are in and just create an LKM (a Linux Kernel Module)?

Modules have always been the best way to take care of adding code to the kernel without risking / going through modifying the kernel.

Pros:
- No kernel source patching-recompilation required
- Modules save system resources when they are not needed they can just be removed / disabled

Cons:
Not any that I can see pertaining to this scenario.

 

[degibson]

I don't think PAPI can just be loaded as a module -- it needs to instrument various parts of the kernel here and there.

 

[scootermaster]

Originally posted by: degibson
I don't think PAPI can just be loaded as a module -- it needs to instrument various parts of the kernel here and there.

The problem is not PAPI. That's just a software install. The problem is for PAPI to work, I need to install the perfctr kernel module/patch. I [thought?] I installed it as a loadable kernel module, but (and all the compilation steps worked!) but so far it's not working. Which is weird, since dmesg gives me the following:

Please email the following PERFCTR INIT lines to mikpe@csd.uu.se
To remove this message, rebuild the driver with CONFIG_PERFCTR_INIT_TESTS=n
PERFCTR INIT: vendor 0, family 15, model 3, stepping 4, clock 2992910 kHz
PERFCTR INIT: NITER == 64
PERFCTR INIT: loop overhead is 735 cycles
PERFCTR INIT: rdtsc cost is 98.2 cycles (7020 total)
PERFCTR INIT: rdpmc cost is 227.9 cycles (15322 total)
PERFCTR INIT: rdmsr (counter) cost is 355.5 cycles (23490 total)
PERFCTR INIT: rdmsr (escr) cost is 376.6 cycles (24840 total)
PERFCTR INIT: wrmsr (counter) cost is 991.4 cycles (64185 total)
PERFCTR INIT: wrmsr (escr) cost is 981.0 cycles (63525 total)
PERFCTR INIT: read cr4 cost is 24.8 cycles (2325 total)
PERFCTR INIT: write cr4 cost is 394.0 cycles (25957 total)
PERFCTR INIT: rdpmc (fast) cost is 90.7 cycles (6540 total)
PERFCTR INIT: rdmsr (cccr) cost is 386.1 cycles (25447 total)
PERFCTR INIT: wrmsr (cccr) cost is 939.0 cycles (60833 total)
PERFCTR INIT: write LVTPC cost is 27.3 cycles (2483 total)
PERFCTR INIT: sync_core cost is 422.2 cycles (27757 total)
perfctr: driver 2.6.25 DEBUG, cpu type Intel P4 at 2992910 kHz

So it SEEMS like the perfctr stuff is there. But perfex -i (which is the test to make sure APIC/perfctr is working/installed) seems to fail.

And, of course, PAPI doesn't work when I install it. I'm going to try it again, without the whole loadable module thing, I guess.

 

[degibson]

Originally posted by: degibson
I don't think PAPI can just be loaded as a module -- it needs to instrument various parts of the kernel here and there.

Originally posted by: scootermaster
...I installed it as a loadable kernel module...

I stand corrected.

Originally posted by: scootermaster
PERFCTR INIT: rdtsc cost is 98.2 cycles (7020 total)

This seems quite high for rdtsc. Are you running in a virtualized environment?

 

[scootermaster]

Originally posted by: degibson

Originally posted by: degibson
I don't think PAPI can just be loaded as a module -- it needs to instrument various parts of the kernel here and there.

Originally posted by: scootermaster
...I installed it as a loadable kernel module...

I stand corrected.

Originally posted by: scootermaster
PERFCTR INIT: rdtsc cost is 98.2 cycles (7020 total)

This seems quite high for rdtsc. Are you running in a virtualized environment?

Okay, I was being brief -- and stupid -- but I guess it's time for some more detail:

1). First off, let's not get confused between PAPI and perfctr. PAPI is just a software API that accesses hardware performance counters. Installing PAPI is a matter of (theoretically) "configure" and "make install". So we're not talking about PAPI here. I've installed it on an R10k before, and it's pretty straight forward (I hope. If i have trouble, I'll be right back here!)

2). perfctr is a kernel patch/module for Linux. It allows access to the performance counters. Why it's not part of the core kernel, I don't know (I actually read a thread about it; never realized there was so much politics involved in getting something into a kernel. But the whole debate is foolish. The world's biggest linux clusters use this shit. The military does. Good enough for me).

3). I installed it once as a loadable module (the install text says that it can, if you so desire, install "most" of it as a loadable kernel module, for whatever that's worth). Got the results you saw above. Some nice notes from dmesg, but nothing doing when I do "perfex -i" (the test to make sure perfctr and APIC is working correctly).

4). I tried it again (for another three hours. Ugh!) installing it, uh, not as a kernel module. Pretty much the same results.

Other notes

5). First hack: My machine type is defined as i686 (I'm pretty sure it's a single-core, hyperthreaded P4. Either it's a dual core machine, or hyperthreading does some weird and strange things in Linux because my IT people have a). installed some SMP kernels on Grub, and b). dmesg shows a lot of stuff about other processors). Anyway, I had to "setenv ARCH i386" to even get "make mrproper" to work. So that's hack #1

6). In the installation steps, after something like "make dep vmlinux modules" or something, and editing the grub.conf files -- I should/will post the actual install directions when I get a chance; it's notable that the directions are for lilo and I modified them for grub -- it says to make install. So the second hack is that when I do that, it tells me I need to make first. So I obediently follow along. (Mentioned only because it's not in the perfctr instructions).

7). This was all done with a kernel.org 2.6.18 kernel source. I'm in the process of trying it with an RPM of 2.6.18-1.2259 (or something, I don't have it in front of me) source provided by my IT people. I have that exact kernel installed in grub. (I left the kernel compiling and went home at midnight. I'll be back there Thursday)

8). Apparently the vmlinuz (what's with that name, anyway?) link in the /boot directory isn't/doesn't change based on which kernel I boot to in grub. When I first installed it,I had to manually delete it, delete all the patched kernel entries in /boot, and manually create a new link to a different kernel. Is that normal?

9). I just want this to work. =]

That's all I can think of for now.

Anyway, to answer your question, no, it's not a virtualized environment. Fedora Core, 2.6.18 (patched), on a P4 3.0 (I think) with a gig of ram.

I can't help but think there is some sort of problem with either the boot loader, or the fact that 99% of the directories I'm dealing with here are off the NFS. But I just don't see why I'd get those messages in dmesg and then the tests would fail (the system call that the patch installs don't exist/can't be found).

Help?

 

[scootermaster]

Any more thoughts? After 10 hours, this still isn't doing what I want it to.

Boo!

 

[degibson]

My only thought that I can productively yield over a forum is to make sure you're using an SMP-enabled kernel -- Hyperthreading is SMP as far as the software is concerned. Other than that, your make commands seem antiquated to me -- when I build 2.6.*, I only use 'make menuconfig' (once), 'make' (after I've made my hacks), and then 'make modules_install' at the end. Copy the kernel from /usr/src/linux/arch/x86/vmlinuz to /boot/myHappyKernel, run grub, voila! new kernel in place. Now, 5 times out of 10 I've screwed something up with my hacking or with menuconfig to make the whole shooting match unbootable -- thats why I don't usually overwrite the 'known good' kernel in /boot/, but make a new one instead.

 

[Nothinman]

5). First hack: My machine type is defined as i686 (I'm pretty sure it's a single-core, hyperthreaded P4. Either it's a dual core machine, or hyperthreading does some weird and strange things in Linux because my IT people have a). installed some SMP kernels on Grub, and b). dmesg shows a lot of stuff about other processors). Anyway, I had to "setenv ARCH i386" to even get "make mrproper" to work. So that's hack #1

That's strange, the only time you should have to force the arch is when building for one that you're not running. Like AMD64 on i386 or sparc32 on sparc64. Does uname -a show the proper arch?

6). In the installation steps, after something like "make dep vmlinux modules" or something, and editing the grub.conf files -- I should/will post the actual install directions when I get a chance; it's notable that the directions are for lilo and I modified them for grub -- it says to make install. So the second hack is that when I do that, it tells me I need to make first. So I obediently follow along. (Mentioned only because it's not in the perfctr instructions).

At the very least you want 'make bzImage modules' to make the bootable kernel image and modules. Then you'll need to do 'make modules_install' to install them to /lib/modules. I never used 'make install' since after modules_install all that's really left to do is copy the bzImage to /boot and fix the bootloader.

8). Apparently the vmlinuz (what's with that name, anyway?) link in the /boot directory isn't/doesn't change based on which kernel I boot to in grub. When I first installed it,I had to manually delete it, delete all the patched kernel entries in /boot, and manually create a new link to a different kernel. Is that normal?

That link, if used at all, is probably just for the default kernel. I tend to do delete them as one of the first post-installation steps after a new Debian installation.