Comprimised Linux server

[GT1999]

So me and a new guy at work (we just started a couple of months ago) were poking around our main Linux web server. We pulled a .tar backup from the server to look around, and it was 12GB. Well, as it turns out 11GB of that is movies, porn, applications / general warez -- all in GERMAN. Also, many of the php file seem to be infected with malicious "trojan code" as Symantec has it labeled.

It looks like the entire server has basically been comprimised.

I would like to trace the intruders, if possible, which have all of the files placed in an "invisible" folder (to Linux, but viewable via NTFS/Windows) under a cgi-bin directory of a virtual website (which I will not name).

Anyone have experiene with situations like these? I don't want to delete the files right off the bat, because I'd like to catch the IP of these guys and catch them in the act and find the security breaches of the server.

From there, who knows -- probably a new server / fresh install of a new OS.

Might I also add, we just found out today that telnet is also enabled on the server (!!!) though we have been ssh'ing to it and direct root access is ENABLED...

 

[doornail]

It could be something as simple as a dictionary attack on your SSH connection, especially if you're loggin in as root.

Do you have a firewall that keeps logs? You could also set up SNORT or ethereal to watch traffic.

 

[drag]

If you haven't kept your PHP stuff up to date then that's pretty likely.

PHP has had a VERY crappy record when it comes to security, also most PHP web applications being sold or distributed have VERY crappy defaults setup. (I don't want to say crappy.. I am aiming for more along the lines of 'utter sh**')

You absolutely NEED to make sure that this stuff is very up to date. You need to subscribe to security related mailing lists and whatnot so that you can get avanced notifications of stuff like this. Absolutely critical.

Either that or use something with a much better security track record.. such as python or perl or whatnot.. and keep that very up to date also.


Other then that the next likely way that they hacked your machine is if your using crappy ssh passwords. They should be over eight characters.. the longer the better. Literally. A 16 character password is better then a 12 character password. Also should use mixed case with numbers and some of those special !@#%$$^%&$%*^{}[]\/? characters.

As for tracking these guys down it's probably pointless. For 2 reasons:
1. They are going to operate through hacked machines. They will have 3 or 4 levels of hacked machines.. often coming through countries that don't cooperate well with US authorities like Denmark or N/S Korea.

on a side note they probably used yours to attack other people's machines also... So it's likely that your ISP and other people have records of people from your IP Address attempted to hijack other people's machines. As in YOU trying to hack other people's boxes. Lovely thought that is.

2. When you discover a machine has been compromised the generally correct way to operate is to automaticly pull the power plug of that machine to shut it down violently. There will probably be scripts and other things installed on your machine that goes into effect when you perform a shutdown or reboot to manipulate the log files and other items to cover a attacker's tracks.

You generally want to pull the plug from the machine, make a image of the harddrive and use that image to perform your analysis. The reason you do this is that you don't want to destroy any evidence.

Generally you'd want to hire a professional security person or company or something along those lines to perform forensics if you want to persue a criminal conviction.

If you don't want to perform a criminal conviction, then it's very very pointless to track them down. You want to hack them back or something? Remember that it's very likely that it appears from web logs and such that people have been trying to hack from your machine for a long time now.. Not pretty.

Although it is probably a very nice thing to go and notify the people whose ip addresses and such were these files are coming and going to that they've probably been hacked also and should fix their stuff. I try to do that time to time, if I see scans comming in from script kiddies trolling for bad ssh passwords I sometimes email the administrators of those machines to tell them that they've been pooched.

Also it's nice to double check and do audits on any other of your machines and make sure that they are secure also. Especially if they are using the same passwords and web-related stuff. It's very likely that they are comprimised also.

When you want to get around actually fixing your machine the best way, the only realy proper way, is to reformat and install from scratch.

If you have backups you can use those... but you have to absolutely make sure that those are clean also. You don't want to be backing up any modified web scripts or trojens or whatnot to your new server. Any scripts or data that your plan on reusing you need to go through and double check by hand before using it.

I am not a expert on this security stuff. If your serious about it, you should contact somebody who is. They'd have better advice then what I can give you.

 

[GT1999]

Thanks guys, drag... awesome advice.

I was thinking we needed to get a new server anyway, but what I'm worried about is porting the old websites back over to a new linux distro, and like you said, having bad scripts / viruses and such brought right back with it. Doing this by hand would mean downtime for many sites and lost money and unhapy customers.

 

[drag]

Well you may be able to get away with a temporary fix of cleaning out everything as much as possible, perform a few nmap scans on it and such, then setting up a second new machine to monitor network traffic. Do it like a typical windows admin would. 😛

as for a the image of the harddrive.. I mispoke a bit. Make a image of the drive, but only mount it read-only or work on copies of that image.. That's the safest way to preserve any information that I know of.

Then work on getting everything ported over as fast as you can to the new server. (as well as auditing that data/scripts and such you plain to reuse, of course) Put in lots of overtime or whatever. If you do it properly then this will likely be the only hacked machine you'd have to deal with in a long long time.

 

[GT1999]

I'm wishing I'm not on salary right now...

nmap scans... is that similar to packet sniffing with Ethereal?

 

[drag]

Nmap is a tool that performs port scanning. It's job is to go and try to access various services on a machine based on it's IP address. That way you can find what open ports that machine has and then using that information it makes it easier to find out what exaclty is running.

Nmap is like most security tools. It's used by crackers to help attack machines and by admins to help secure them. It's aviable by default on most Linux distros.

It's home page is aviable
http://www.insecure.org/nmap/

There are lots of little things like that that can make your life easier.

http://www.linuxsecurity.com/
This page is kinda handy. Lots of links to other sites and such. Also keeps track of security advisories from bunches of distros.

 

[GT1999]

I'll give nmap a shot. Unfortunately with all the things I know are already enabled... this is just a bad, bad server setup.

I'm thinking RHEL v4, Fedora Core 5 (just released) or FreeBSD 6 for the new server OS. I'm not sure if the shell scripts and such setup on the current linux distro will even work on FreeBSD, but I know it has the potential to be much more stable than a lot of linux distros while sacrificing functionality.

 

[Nothinman]

nmap will only tell you what's listening on the box, for a real security test you need too look at things like nessus, nikto, etc.

I'm not sure if the shell scripts and such setup on the current linux distro will even work on FreeBSD, but I know it has the potential to be much more stable than a lot of linux distros while sacrificing functionality.

The shell scripts working is probably 50/50, it depends on whether or not the authors uses any bash-isms. You could install bash on the FreeBSD box but that might not fix all of the problems if the scripts were only tested on Linux.

Also the stability thing is mostly BS, either can be just as secure and stable as the other. There might be more potential for problems with a Linux distribution simply because more is installed out of the box, but if you do a server/expert install you can have a fully functional box with as few packages as necessary.

 

[GT1999]

Good lord. The man page for nmap is like the bible.

Well, I'm back at work now and logged into the server... trying to find a public IP address so I can do this from home.

 

[GT1999]

[root@server cgi-bin]# nmap -v 10.2.2.21

Starting nmap 3.81 ( [url="http://www.insecure.org/nmap/"]http://www.insecure.org/nmap/[/url] ) at 2006-03-20 19:36 CST
Initiating SYN Stealth Scan against host-10-2-2-21.domain.com (10.2.2.21) [1663 ports] at 19:36
Discovered open port 23/tcp on 10.2.2.21
Discovered open port 443/tcp on 10.2.2.21
Discovered open port 80/tcp on 10.2.2.21
Discovered open port 22/tcp on 10.2.2.21
Discovered open port 25/tcp on 10.2.2.21
Discovered open port 139/tcp on 10.2.2.21
Discovered open port 802/tcp on 10.2.2.21
Discovered open port 917/tcp on 10.2.2.21
Discovered open port 81/tcp on 10.2.2.21
Discovered open port 111/tcp on 10.2.2.21
Discovered open port 6969/tcp on 10.2.2.21
The SYN Stealth Scan took 0.12s to scan 1663 total ports.
Host host-10-2-2-21.domain.com (10.2.2.21) appears to be up ... good.
Interesting ports on host-10-2-2-21.domain.com (10.2.2.21):
(The 1652 ports scanned but not shown below are in state: closed)
PORT     STATE SERVICE
22/tcp   open  ssh
23/tcp   open  telnet
25/tcp   open  smtp
80/tcp   open  http
81/tcp   open  hosts2-ns
111/tcp  open  rpcbind
139/tcp  open  netbios-ssn
443/tcp  open  https
802/tcp  open  unknown
917/tcp  open  unknown
6969/tcp open  acmsoda

Nmap finished: 1 IP address (1 host up) scanned in 0.258 seconds
               Raw packets sent: 1665 (66.6KB) | Rcvd: 3339 (134KB)

 

[GT1999]

[root@server cgi-bin]# netstat -lnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:32768 0.0.0.0:* LISTEN 1348/rpc.statd
tcp 0 0 0.0.0.0:32769 0.0.0.0:* LISTEN 1606/rpc.mountd
tcp 0 0 0.0.0.0:802 0.0.0.0:* LISTEN 1469/ypbind
tcp 0 0 0.0.0.0:3654 0.0.0.0:* LISTEN 1974/httpd
tcp 0 0 10.2.2.21:139 0.0.0.0:* LISTEN 1724/smbd
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1320/portmap
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 1974/httpd
tcp 0 0 0.0.0.0:81 0.0.0.0:* LISTEN 1974/httpd
tcp 0 0 xxx.90.65.x:52658 0.0.0.0:* LISTEN 20852/pure-ftpd (ID
tcp 0 0 xxx.90.65.x:21 0.0.0.0:* LISTEN 1654/pure-ftpd (SER
tcp 0 0 xxx.90.65.xx:21 0.0.0.0:* LISTEN 1651/pure-ftpd (SER
tcp 0 0 0.0.0.0:917 0.0.0.0:* LISTEN 1585/rpc.rquotad
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1528/sshd
tcp 0 0 0.0.0.0:23 0.0.0.0:* LISTEN 1562/xinetd
tcp 0 0 0.0.0.0:6969 0.0.0.0:* LISTEN 26155/
tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN 1634/sendmail: acce
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 1974/httpd
udp 0 0 0.0.0.0:32768 0.0.0.0:* 1348/rpc.statd
udp 0 0 0.0.0.0:2049 0.0.0.0:* -
udp 0 0 0.0.0.0:514 0.0.0.0:* 1989/syslogd
udp 0 0 0.0.0.0:32771 0.0.0.0:* -
udp 0 0 0.0.0.0:32772 0.0.0.0:* 1606/rpc.mountd
udp 0 0 10.2.2.21:137 0.0.0.0:* 1729/nmbd
udp 0 0 0.0.0.0:137 0.0.0.0:* 1729/nmbd
udp 0 0 10.2.2.21:138 0.0.0.0:* 1729/nmbd
udp 0 0 0.0.0.0:138 0.0.0.0:* 1729/nmbd
udp 0 0 0.0.0.0:914 0.0.0.0:* 1585/rpc.rquotad
udp 0 0 0.0.0.0:40094 0.0.0.0:* 13369/bash
udp 0 0 0.0.0.0:799 0.0.0.0:* 1469/ypbind
udp 0 0 0.0.0.0:800 0.0.0.0:* -
udp 0 0 127.0.0.1:56614 0.0.0.0:* 17839/smbd
udp 0 0 0.0.0.0:623 0.0.0.0:* 1469/ypbind
udp 0 0 0.0.0.0:111 0.0.0.0:* 1320/portmap
Active UNIX domain sockets (only servers)
Proto RefCnt Flags Type State I-Node PID/Program name Path
unix 2 [ ACC ] STREAM LISTENING 2329 1706/xfs /tmp/.font-unix/fs7100


------------

[root@server messages]# uname -a
Linux web1 2.4.20-28.7smp #1 SMP Thu Dec 18 11:18:31 EST 2003 i686 unknown
[root@server messages]#


What OS?


Looks like we have a RedHat box. An old one at that:

Sep 1 19:00:56 web kernel: Linux version 2.4.20-20.7smp (bhcompile@porky.devel.redhat.com) (gcc version 2.96 20000731 (Red Hat Linux 7.3 2.96-113)) #1 SMP Mon Aug 18 14:46:14 EDT 2003

 

[n0cmonkey]

Just replace the box. Make sure someone does a good audit on the website scripts.