I have a server at OVH that hosts all my websites as well as other people's. Nothing super serious but it's basically my main online presence. Long story short my web server was running fairly old software and was due to be updated and I was procrastinating. Got hacked. So without any choice I order a new server and it gets hacked again. This time it was running new software, Debian 11 and everything was vanilla since I was still in the process of setting it up. Both attacks appear to be similar in nature.
When I look at the apache logs, I see something like this:
In the syslog:
I'm guessing these are special characters and vim is just representing them that way. It looks like some kind of injection of sorts.
I figured the first one was simply due to it being a known exploit and fact that I had not updated the OS in a while, but now that it happened to the 2nd one with newer everything, I'm really starting to wonder what in the world is going on. Clearly somebody is attacking me and this is targetted and a rather sophisticated attack.
Anyone ever see anything like this before?
After this attack the server basically becomes inaccessible. I think they did something to SSH so the keys don't work to login to the server, and changed the password. (still had the password auth enabled temporarily). I do not see any signs of brute force attempts. This really seems like some kind of auth bypass attack.
Oh and this bit of log in /var/secure is probably what did the first server in, looks like they were able to load something malicious in /usr/sbin and run it. It also seems to have done somewhere where SSH failed, and so did apache and pretty much everything, except DNS kept working and server was still pinging at that point.
I kept a backup of that file so I can try running it in a VM to see what it does, but have not gotten to it yet.
I had the 2nd server in rescue mode, which is a mode that the data centre offers where it boots into a rescue image and then I can SSH in with a provided long password. That got hacked too. Somebody is clearly attacking me right now with some kind of sophisticated attack. I booted it back up in rescue mode and I'm watching /var/log/syslog now just to see if I spot anything weird. I don't think this is a SSH brute force attack unless that weird stuff I'm seeing in the logs is a way to erase the logs. (maybe issuing a bunch of backspaces as part of the username?)
Live, I would be seeing the attempts though. So I will let this run overnight again to see what happens.
They have not touched the original server which is now in rescue mode though... this whole thing is just so strange. At this point I just want to try to figure out how they got in, I don't think it's just the thing of it being out of date considering the same attack worked on the new server.
When I look at the apache logs, I see something like this:
Code:
x.x.x.x - - [27/Nov/2021:17:52:20 +0000] "GET /?level=picture&id=1052 HTTP/1.1" 200 623 "-" "Mozilla/5.0 (compatible; Barkrowler/0.9; +https://babbar.tech/crawler)"
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@x.x.x.x - - [27/Nov/2021:17:57:03 +0000] "GET /shard.php?id=1334&act=vote HTTP/1.1" 404 740 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36"In the syslog:
Code:
Nov 27 17:56:53 server04 dovecot: pop3-login: Disconnected (no auth attempts in 2 secs): user=<>, rip=x.x.x.x, lip=x.x.x.x, session=<UL3Q7sjRwlorgSFj>
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
Followed by many white spacesI'm guessing these are special characters and vim is just representing them that way. It looks like some kind of injection of sorts.
I figured the first one was simply due to it being a known exploit and fact that I had not updated the OS in a while, but now that it happened to the 2nd one with newer everything, I'm really starting to wonder what in the world is going on. Clearly somebody is attacking me and this is targetted and a rather sophisticated attack.
Anyone ever see anything like this before?
After this attack the server basically becomes inaccessible. I think they did something to SSH so the keys don't work to login to the server, and changed the password. (still had the password auth enabled temporarily). I do not see any signs of brute force attempts. This really seems like some kind of auth bypass attack.
Oh and this bit of log in /var/secure is probably what did the first server in, looks like they were able to load something malicious in /usr/sbin and run it. It also seems to have done somewhere where SSH failed, and so did apache and pretty much everything, except DNS kept working and server was still pinging at that point.
Code:
Nov 24 09:52:02 server03 runuser: pam_unix(runuser:session): session opened for user root by (uid=0)
Nov 24 09:52:02 server03 runuser: pam_unix(runuser:session): session closed for user root
Nov 24 09:52:03 server03 sshd[2009]: error: Bind to port 87 on 192.95.14.97 failed: Permission denied.
Nov 24 09:52:03 server03 sshd[2009]: fatal: Cannot bind any address.
Nov 24 09:52:43 server03 userhelper[2506]: running '/usr/sbin/setup ' with root privileges on behalf of 'root'
Nov 25 21:40:50 server03 runuser: pam_unix(runuser:session): session opened for user root by (uid=0)
Nov 25 21:40:50 server03 runuser: pam_unix(runuser:session): session closed for user root
Nov 25 21:40:52 server03 sshd[2077]: error: Bind to port 87 on 192.95.14.97 failed: Permission denied.
Nov 25 21:40:52 server03 sshd[2077]: fatal: Cannot bind any address.
Nov 25 21:41:01 server03 userhelper[2574]: running '/usr/sbin/setup ' with root privileges on behalf of 'root'I kept a backup of that file so I can try running it in a VM to see what it does, but have not gotten to it yet.
I had the 2nd server in rescue mode, which is a mode that the data centre offers where it boots into a rescue image and then I can SSH in with a provided long password. That got hacked too. Somebody is clearly attacking me right now with some kind of sophisticated attack. I booted it back up in rescue mode and I'm watching /var/log/syslog now just to see if I spot anything weird. I don't think this is a SSH brute force attack unless that weird stuff I'm seeing in the logs is a way to erase the logs. (maybe issuing a bunch of backspaces as part of the username?)
Live, I would be seeing the attempts though. So I will let this run overnight again to see what happens.
They have not touched the original server which is now in rescue mode though... this whole thing is just so strange. At this point I just want to try to figure out how they got in, I don't think it's just the thing of it being out of date considering the same attack worked on the new server.
Last edited:
Facebook
Twitter