Linux box reboots on random days but at fixed time

[TheOtherMudit]

I probably also added libCurl at the same time.
But I'll be very worried if a userlevel application/library can crash the kernel !
Then what's the difference between windoze and unix ?

 

[TheOtherMudit]

bersl2,

On sunday, I ran a script which changed the date and time to 7:25AM and then sleep for 10 mins. And then change
the date to next day, and time back to 7:25am.

date 01010725
sleep 600
date 01020725
sleep 600
.......

The script ran fine till Dec 31st, 2005.

I was hoping that some hidden trojan or something will get triggered. But I couldn't change the "uptime". The uptime
still was showing number of mins since boot. If I can somehow change the uptime, may be I can get somewhere.
The uptime was not changing as its incremented by the kernel clock timer. Unless I find a way to directly
change that variable. But it may screw the kernel timers and what not.

I know I can probably upgrade to latest RH9.0 and hopefully the problems will go away. But I'm not more interested
in the cause of the problem.

I think for this week, I'll move the machine to the garage and hookup to a monitor to find out what happens
at 7:30am. I really want to log the console output in a file. Looked around in google, there is some
mention about syslogd, but I dont think it captures all the console output. And my guess it, when the crash
is happening, the filesystem and other drivers are probably hosed by that time.

 

[TheOtherMudit]

Hmmm.. I also installed the Xvnc server around that time.
Lots of new stuff added
a) mysql server
b) libcurl/libssl
c) Xvnc (vncserver) in KDE enviroment

I gotta hookup the monitor to the box and watch for any messages between 7:25 and 7:30 everyday !

 

[n0cmonkey]

syslogd should capture all of the logs. You can have it send them to another machine. Hook up a serial cable, configuring it shouldn't be too tough. You can then connect the serial cable to another machine, use something like tip, and keep an eye on it.

 

[Abzstrak]

just a thought, but why not kill cron.... just stop it and dont let it run anything, then if the system doesn't reboot you know its something to do with cron, otherwise your atleast narrowing it down as NOT being cron related or being cron related.

I agree with Drag, check for other things, like people plugging a vacuum into your UPS... simple things are hard to track down sometimes. Is the UPS good?

 

[TheOtherMudit]

Okay, will kill the cron at 7:25 and restart it at 7:35
But I dont think cron is at issue. I dont see any thing running in the /var/log/cron at 7:25-7:30am time frame.
At the same time, it doesnt list any jobs scheduled by the at command.

At 7:25am, no one is in the laundary room, plugging anything in the UPS. The UPS is working, just this weekend, I unpluggged
the UPS from the wall and let the computer run for sometime on UPS.

The interesting part is that just 1-2 mins before the crash, the entire system is idle. Even though, there is supposed to be
some activity all the time, esp at morning hours as I collect stock data. Some programs are always running.

 

[randalee]

Did you ever get it to reboot again? Any more updates since the 29th when you last posted?

 

[TheOtherMudit]

randalee,

no reboots after 29th. the reboots normally happen anywhere from 3-5 days.

 

[LuckyTaxi]

This once happened to me, but it wasn't at a set time. It would randomly happen throughout the week. It turned out my power supply was set to 230w instead of the normal 115w. Check that. That's all I can think of.

 

[TheOtherMudit]

update: last week, I upgraded to mysql 4.1.10a-standard. So, far the system is stable and no morning crashes at 7:30am.

>uptime
2:44pm up 7 days, 18:24, 4 users, load average: 1.47, 1.77, 1.60

 

[Brazen]

Have you tried a virus scan (didn't read the whole thread if already asked)? I know, I know, it's linux. But linux is getting popular, and it is not immune. You might try checking for a rootkit, if this is still going on. But it looks like it may be fixed?

 

[TheOtherMudit]

i did check for a rootkit etc sometime back. didnt find anything out of the ordinary.
hopefully, mysql upgrade will fix it. if it crashes again at 7:30am, I'll simply reinstall new OS.

 

[screw3d]

I'm scared

This sounds like a script from a horror film

 

[randalee]

You ever do a crontab -l to list the current cron table? I'd love to take a looksie and see what you have in there... Or are you completely fixed now that you've upgraded your MySQL package? I'm doing a Linux course right now (www.gurulabs.com), and discussed your issue in class. Everyone here is agreeing that it's got to be something in there. Whaddya say?

Randy

 

[TheOtherMudit]

i checked crontab. That was the very first thing I did. You can also look at /var/log/cron file. There is a time stamp
and it lists any task which it is about to run. So, if it was cron issue, there would have a log in the file. Here is a sample
of what the /var/log/cron looks like.

Well, power went down twice yesterday, so the system rebooted. Also, I'm running low on CMOS battery, so have to
change it this weekend (got 10 cr2032 batteries for $4 shipped from ebay). But after changing mysql, the system
survived for 8+ days ! That's the longest it has stayed up so far !

But anyway, I'm thinking to upgrade the system by installing ubuntu on it in the weekend. I'm going to install pchdtv HD3000 card in the system and use it as backend mythtv system. Since HD3000 directly dumps the mpeg stream into the file, the cpu usage is almost nothing.
When it runs as a backend, the data is transmitted via TCP/IP (NFS mount). So, hopefully, there is no load on the server. All the work is done in the frontend. Thinking about a P4 3.4ghz front end in near future. But thats a different topic.

--------- sample of my /var/log/cron

As you can see, I'm running "sa" every minute on the minute. I know its inefficient, but thats how the sa is written.

Apr 14 11:57:00 bp6 CROND[8182]: (root) CMD (/usr/lib/sa/sa1 1 1)
Apr 14 11:58:00 bp6 CROND[10527]: (root) CMD (/usr/lib/sa/sa1 1 1)
Apr 14 11:59:01 bp6 CROND[12673]: (root) CMD (/usr/lib/sa/sa1 1 1)
Apr 14 12:00:00 bp6 CROND[15095]: (root) CMD (/usr/lib/sa/sa1 1 1)
Apr 14 12:01:00 bp6 CROND[16995]: (root) CMD (run-parts /etc/cron.hourly)
Apr 14 12:01:00 bp6 CROND[16996]: (root) CMD (/usr/lib/sa/sa1 1 1)
Apr 14 12:02:00 bp6 CROND[19380]: (root) CMD (/usr/lib/sa/sa1 1 1)
Apr 14 12:03:00 bp6 CROND[21048]: (root) CMD (/usr/lib/sa/sa1 1 1)

 

[TheOtherMudit]

Update: 8+ days and still strong.
So I very much suspect something fishy about the mysql 4.1.1-alpha-standard.
Now I'm running mysql 4.1.10a-standard. That is the only component I changed in my configuration.

Current uptime
$ uptime
9:02am up 8 days, 22:51, 3 users, load average: 1.10, 0.71, 0.58

 

[bersl2]

Woo!

 

[TheOtherMudit]

woot !!!!

$ uptime
10:34am up 15 days, 22 min, 3 users, load average: 1.86, 2.06, 2.06

so it was damn mysql 4.1.1a version ! I've send an email to mysql but I doubt very much that they will *believe* it
i hope this version of mysql (4.1.10a) doesnt bomb out after 20-30 days !

keep on trucking !!! errr .. running !!!