find out reason for kernel panic

[Brazen]

I had a RedHat 5.0 server freeze up with a kernel panic. I had to reboot it quick and it came back up just fine. Is there any way to find out why it crashed with a kernel panic? Isn't there some logs that would give an indication? I really want to make sure this doesn't happen again.

 

[SleepWalkerX]

Perhaps the dmesg might shed some light? That's the only thing I can really think of. It'd be /var/log/dmesg.

 

[skyking]

What are the applications? check those logs for unusual stuff. Sometimes that is all you'll get. If it is hardware related, there is usually nothing in the logs themselves.

 

[degibson]

Try memtest 86.

 

[cmv]

Originally posted by: degibson
Try memtest 86.

Agreed -- but go for memtest86+ (it is up to date as apparently memtest86 was orphaned). Likely there is NO record of why it crashed. There are some debugging options when configuring a kernel for preserving crash details. RedHat may or may not have enabled these options.

It might sound silly but keeping an inexpensive digital camera in the server room can be beneficial for these kinds of things -- snap a quick photo and then go on with "recover ASAP". Later, you can go back to the photo to research what happened.

Usually, a crash screen has some details on why the crash occurred (which process triggered the crash and so forth).

 

[Nothinman]

Is there any way to find out why it crashed with a kernel panic? Isn't there some logs that would give an indication? I really want to make sure this doesn't happen again.

Not unless you have remote console or remote syslogging setup, one of the first things the kernel does when it panics is stop all disk I/O to avoid corrupting your filesystems. I'm thinking RHEL might also have a crash dump thing in it now too but I'm not sure.

 

[Brazen]

hmm, not too hopeful. This is also running on VMWare ESX Server. I have several RHEL 5.0 servers setup just like this, so I don't think it is any kind of RHEL/ESX combination issue. I would also be surprised if it is hardware related as every other virtual machine on that same system is chugging along fine (although this problem vm does have 2.5GB of ram allocated to it, which is significantly more than any of the others, so it could be RAM related and this server hit it due to probability). Just thinking out loud here.

Anyway, I set it up to reboot on a kernel panic for now, and I'll monitor it for reboots. Maybe it was just bad luck and won't happen again.

 

[degibson]

Originally posted by: Brazen
hmm, not too hopeful. This is also running on VMWare ESX Server. I have several RHEL 5.0 servers setup just like this, so I don't think it is any kind of RHEL/ESX combination issue. I would also be surprised if it is hardware related as every other virtual machine on that same system is chugging along fine (although this problem vm does have 2.5GB of ram allocated to it, which is significantly more than any of the others, so it could be RAM related and this server hit it due to probability). Just thinking out loud here.

Anyway, I set it up to reboot on a kernel panic for now, and I'll monitor it for reboots. Maybe it was just bad luck and won't happen again.

OH well that would have been useful information.
I now blame VMWare. They've got a hard job to do, and its not surprising that they occasionally screw it up.