Why does the drive seek randomly on my Linux box?

[IaPuP]

I have a Linux Box RedHat 7.2 primarily running Apache w/ PHP & MySQL.

The system has been up for several months, but recently, I've noticed that the harddrive seeks (and the HDD light flickers) ever few seconds.

Is there any way to isolate what program is engaging the harddrive to read/write on a regular basis?? It reminds me an awful lot of the traffic a steady stream of data from a program like TCPDUMP would generate, but I can't find anything too suspicious in the running processes.

Any hints are welcome.

Eric

 

[n0cmonkey]

Apache + PHP + MySQL could be doing this. If you can, kill them and see what happens.

 

[IaPuP]

I'd rather not kill them.

I've watched the box during times that there were no queries going to MySQL and it still does it.

I'll try it out later tonight if there's no traffic and see what it looks like.

I know MySQL uses disk-based tables, but I also know that the queries don't come in anywhere nearly as often as the disk clicks.

Thanks,
Eric

 

[n0cmonkey]

Make sure you have kept up to date on all patches. If there is an errant process out there that is causing this and cannot be seen it may be hidden by a malicious kernel module or something. But this would be very hard to track down without taking the machine down.

 

[IaPuP]

I went ahead and killed them now.

Left them down for 1 minute, but the damn thing kept going.

I'm more concerned about what it's doing (since this is a new thing) than stopping it.

I'd just like to know what my system is doing. Usually, I can tell from the sound how busy the server is... and when cron jobs are running. But since this noise started it's gotten me thrown all off. lol

Judging server load by the noise it makes... *chuckles*
*shakes head*

I'm a nerd.

Eric

 

[n0cmonkey]

<< I went ahead and killed them now.

Left them down for 1 minute, but the damn thing kept going. >>



Ok, you have eliminated 2 potential problems (apache and mySQL). This is what you do in troubleshooting, eliminate potential problems until you are left with only one possibility



<< I'm more concerned about what it's doing (since this is a new thing) than stopping it. >>



I understand.



<< I'd just like to know what my system is doing. Usually, I can tell from the sound how busy the server is... and when cron jobs are running. But since this noise started it's gotten me thrown all off. lol >>



I do the same thing with my home systems. Atleast I did until I added some fans

Check /etc/modules.conf (may not be valid on your distro) for any additions. Also, check the last modified date of your init scripts (and modules.conf for that matter) to see when they were last modified. This can be faked, but may not have been.

 

[IaPuP]

Patches are recent as of about a month ago. I don't want to take the system down to rebuild the kernel or something.

I guess I could... but I'd rather not.

Any other suggestions?

The system is behind a firewall, with no external access aside from ports 80 and 443 (HTTP and HTTPS).

Although..... I do have a wireless segment inside the firewall... that could be a weak point- so I guess it's possible someone was able to break in... but I have every thing but TTY access disabled on the system (no telnet or SSH).

I don't know... it's pretty secure but not impregnable. No handy utilities to see what processes have open files or file handles??

thx,
Eric

 

[Sunner]

Does the used space on any partitions change?

Good way to find out if it's reads or writes.

 

[Nothinman]

run 'vmstat 1' for a bit, the bi column is blocks in and bo is blocks out, that will tell you whether it's reads or writes.

lsof will show you what files are open.

 

[IaPuP]

It's 244 Blocks In. I'm assuming that means it is reading the disk.

The memory cache is also growing by 1 each time it reads.

What would it be caching??

The lsof doesn't exist. No executable. I did a find / -name lsof and didn't turn up anything, so I imagine its safe to assume its not installed.

I'm searching out a download right now- I"ll let you know if It tells me anything.

Thanks,
Eric

 

[IaPuP]

I chcked out the init config and didn't see anything unusual so I went ahaed and rebooted the box.

Thanks to some fast disks, it was back up in less than 30 seconds and now has stopped making the noise (and filling up the disk 10KB per minute).

Not sure if it was a rogue process or a runaway script or something I ran and forgot to turn off.

I'm going to lock it down a bit more since i did notice that I had sendmail running (oops) even tho my external firewall had it shut down, the internal side was still vulnerable (I guess).

Thx for the tips.

Eric

 

[IaPuP]

By the way... LSOF was't very useful. It spit out about 4000 file handles at me to every process and every pipe and every FIFO and every device that was open.

samba and httpd and Mozilla each had several hundred entries.

I'll admit I haven't gotten to dig through its features much yet. I'll take a look more in depth later.

Eric

 

[Nothinman]

By the way... LSOF was't very useful. It spit out about 4000 file handles at me to every process and every pipe and every FIFO and every device that was open.

You wanted a list of all the open files =)

Anyway if you know it's not big things like mozilla just run 'lsof | grep -v mozilla' to script out it's entries.