What would have caused apache, postfix and ssh to restart?

[Red Squirrel]

I got a fail2ban alert that yesterday these 3 services restarted. I did not restart those myself. First time I see this and this server has been up for over 500 days. I'm just paranoid maybe I got hacked or something. I don't see anything unusual in logs other than successful authentications at times that I would have been in bed but it could be backup jobs that were logging in and it does not coincide with the time the restarts happened. (they happened before)

Is there anything I can check to ensure I did not get compromised? I looked for any new users, or other stuff like that, and don't see any unusual processes, but then again that's not saying much as they could be hiding themselves well. Root is not allowed to login to SSH so for someone to break in they'd have to know both the password of a user account AND root, so to me it seems unlikely.

Does logrotate have a tendancy of restarting services? First time I would see that happen though. But I do have a cron email for logrotate that happens around the same time the services restarted.

Also, I see lot of entries like this in dmesg, they are kinda alarming, is this someone trying to hack in somehow?

TCP: Treason uncloaked! Peer 216.185.83.238:50839/80 shrinks window 2978575884:2978575904. Repaired.
TCP: Treason uncloaked! Peer 216.185.83.238:1883/80 shrinks window 4225344903:4225344951. Repaired.
TCP: Treason uncloaked! Peer 216.185.83.238:1883/80 shrinks window 4225566823:4225566871. Repaired.
TCP: Treason uncloaked! Peer 216.185.83.238:1883/80 shrinks window 4226526043:4226526091. Repaired.
TCP: Treason uncloaked! Peer 216.185.83.238:46660/80 shrinks window 1712997817:1712997865. Repaired.
TCP: Treason uncloaked! Peer 216.185.83.238:46660/80 shrinks window 1713194633:1713194681. Repaired.
TCP: Treason uncloaked! Peer 216.185.83.238:50839/80 shrinks window 2982776304:2982776352. Repaired.

I've gotten those since day one of this server being deployed though so I'm guessing it's nothing major.

 

[KillerBee]

I got a fail2ban alert that yesterday these 3 services restarted. I did not restart those myself. First time I see this and this server has been up for over 500 days. I'm just paranoid maybe I got hacked or something. I don't see anything unusual in logs other than successful authentications at times that I would have been in bed but it could be backup jobs that were logging in and it does not coincide with the time the restarts happened. (they happened before)

Is there anything I can check to ensure I did not get compromised? I looked for any new users, or other stuff like that, and don't see any unusual processes, but then again that's not saying much as they could be hiding themselves well. Root is not allowed to login to SSH so for someone to break in they'd have to know both the password of a user account AND root, so to me it seems unlikely.

Does logrotate have a tendancy of restarting services? First time I would see that happen though. But I do have a cron email for logrotate that happens around the same time the services restarted.

Also, I see lot of entries like this in dmesg, they are kinda alarming, is this someone trying to hack in somehow?

TCP: Treason uncloaked! Peer 216.185.83.238:50839/80 shrinks window 2978575884:2978575904. Repaired.
TCP: Treason uncloaked! Peer 216.185.83.238:1883/80 shrinks window 4225344903:4225344951. Repaired.
TCP: Treason uncloaked! Peer 216.185.83.238:1883/80 shrinks window 4225566823:4225566871. Repaired.
TCP: Treason uncloaked! Peer 216.185.83.238:1883/80 shrinks window 4226526043:4226526091. Repaired.
TCP: Treason uncloaked! Peer 216.185.83.238:46660/80 shrinks window 1712997817:1712997865. Repaired.
TCP: Treason uncloaked! Peer 216.185.83.238:46660/80 shrinks window 1713194633:1713194681. Repaired.
TCP: Treason uncloaked! Peer 216.185.83.238:50839/80 shrinks window 2982776304:2982776352. Repaired.

I've gotten those since day one of this server being deployed though so I'm guessing it's nothing major.

500 days! what kernel are you running?
ie: all security patches applied?

 

[Red Squirrel]

Os is CentOS 5 and kernel string is 2.6.18-194.32.1.el5PAE.

Good point about security patches... TBH I have not done a yum update in a while. I should probably do that. Hopefully it wont break anything.

Kernel updates is not something I feel comfortable playing with on a production server though. Most kernel related exploits that I've seen require someone to be logged in so I'm not too worried about those. But updating software like ssh is probably a good idea...

 

[KillerBee]

To investigate I would recommend capturing an image with a boot CD first before doing anything

 

[Red Squirrel]

That's not an option as I don't have physical access. It's in a data center 1000km from me. (closest data center)

 

[KillerBee]

well that sucks ... especially trying to upgrade SSH while being logged in remotely
I guess asking if you had good ole' Tripwire installed is pointless?

 

[Red Squirrel]

Nah that went fine, I'm just updating the critical stuff now. SSH seems to be coded fairly smart where it lets you restart it while still logged in and keeps the session.

Kernel on the other hand would be impossible. Pretty sure you have to have physical access for that. Never actually done it though. But really since I'm the only person that logs into this server what's critical is the apps that have open ports or that could potentially be interfaced through such apps. So all the web stuff basically.

 

[KillerBee]

Congrats on the SSH upgrade...that can get hairy upgraiding remotely.
Best thing to do is limit SSH access with TCP Wrappers to as few IPs possible

If you can contact someone in the Data Center to be around for Kernel upgrade
it would be cool ...usually have no problems - as long as the old one still remains as a last resort.

Also contact them to see if they had any power interruptions around the same time

 

[mv2devnull]

Kernel update on CentOS. Yum installs the new kernel instead of replacing the old. Boot loader (grub) config will get a stanza for the new kernel added. Reboot.

Reboot of a server should not be a big deal, just short service interrupt. After 500 days of uptime the fsck will most likely go through the filesystems. That may not be so short.

In principle updates on enterprise distro should not break anything. In practice there have been non-trivial feature changes.

With reboot there is a chance that the new kernel does not boot correctly. At that point one would need to reboot with the old kernel. There are hardware solutions (KVM, ILO, etc) that allow remote connection and hook up to the console of a server. Another thing is, does the datacenter in question offer/allow such access. If not, someone has to "stand by".

 

[Red Squirrel]

They do offer that access but it's like 50 bucks extra per month. The problem is, I do not know how to do a kernel update, and it's not something I want to screw around with on a production machine. Guess I could try it in a VM a few times to make sure I have the process down path once I read up on it more.

But back to the original question, is there any kind of process that could have caused those services to be restarted? Logrotate maybe? (it ran around the same time). Maybe the logs got too big and it never had access because the services have been running so long so it decided to restart them?

Is there anything I can look for to confirm the server was not hacked? I really don't think it was but some kind of confirmation would be nice.

 

[mv2devnull]

Logrotate should not do that. "yum update openssh\*" does restart sshd, as far as I know.

If your system has been broken into, you must assume that the intruder has replaced relevant binaries with hacked versions that hide the intruder and that all the logs in the machine have been cleaned from signs of intrusion. If you ssh into the system and run any check, the result is probably "nothing is wrong here".

If you could boot from LiveCD/USB, then you could compare contents of the disk to known good files. If.


Kernel update on CentOS (my way):

yum update --exclude=kernel
yum update
reboot

To be fair, just today I saw a Scientific Linux machine, where the installation had omitted something and I had to do a bit of magic before the expected update routine worked.

Technically, "yum update kernel" does:
1. Unpack files from kernel package. They don't overwrite previous kernels.
2. Create a /boot/initrd*.img
3. Add stanza for this kernel version into /boot/grub/grub.conf

 

[Red Squirrel]

Would "yum update" ensure everything gets fully updated, and if by chance it got replaced by a hacked version, replace that too? I might run that one night when I have lot of time to deal with any issues.

 

[mv2devnull]

Would "yum update" ensure everything gets fully updated, and if by chance it got replaced by a hacked version, replace that too?

A hacked yum that pretends to update packages but doesn't, or simply refuses to update some packages, ...

 

[Red Squirrel]

What I wonder is how a hacker could have gotten in without brute forcing. The firewall blocks all ports but the actual services like http, ssh, dns, mail etc. Those are normally fairly secure and don't have privilege escalation stuff like is seen in Windows all the time.

Is there something I'm missing here? I'm still not convinced I got hacked, but I can't rule it out at this point. A service does not just restart on it's own.

I'm guessing it's probably pointless to start looking at file dates right? Most hackers would probably reset the clock before changing anything.