Linux box was rooted

[flippinfleck]

So someone was able to log into one of my friend's linux boxes (he's got several sitting on a T1) as root.

They originaly started out by trying to log in with FTP and were unsucsesfull. Next, they got in with SSH. Logged in and set up their own user. How can we find out what else was compromised? Any way to see what it was that was done while they were logged on?

Should we even try to recover the box or just scrap it and start fresh? We changed all his passwords and they were still able to log right in two days later.

 

[Electrode]

They installed a root kit. Reinstall all software from trusted media.

Edit: And then, update OpenSSH to 3.7.1 IMMEDIATELY.

After these measures are taken, do what you can to verify that your files weren't modified. Restore suspect files from backups. They probably just wanted to use the T1 for DoS attacks or a warez FTP, but they might have messed with your data just for "fun".

If you had sensitive personal information on there, keep an eye out for signs of ID theft. If you had sensitive business information on there, I sure hope you've got a procedure for data leaks...

 

[flippinfleck]

Originally posted by: Electrode
They installed a root kit. Reinstall all software from trusted media.

Edit: And then, update OpenSSH to 3.7.1 IMMEDIATELY.

Yeah, we heard about the OpenSSH vulnerability today. What files (this was a production server with some websites and other such things) are safe to backup? Should we only worry about the programs?

 

[Electrode]

I'd notify the website owners (is this a server for customers, or your business' site?) of the intrusion and tell them to verify their site, and hold on to your latest trusted backup until you are sure there are no virii, defacements, etc.

 

[flippinfleck]

Yeah, he's got customers on there. I'm not sure how many but it shouldn't be more than a dozen; he uses the system mainly for backups.

 

[Barnaby W. Füi]

It wasn't necessarily due to the SSH issue from the past few days. So far I have seen zero confirmation that an exploit is even possible from that bug. Have I missed something?

 

[cleverhandle]

Originally posted by: BingBongWongFooey
Have I missed something?

Not that I'm aware of, and I'd think that between misc@ and bugtraq something would've shown up if it were really out there.

 

[flippinfleck]

Originally posted by: BingBongWongFooey
It wasn't necessarily due to the SSH issue from the past few days. So far I have seen zero confirmation that an exploit is even possible from that bug. Have I missed something?

To be honest, it was mostly lack of a decent password on his part. Nothing more.

 

[Barnaby W. Füi]

Originally posted by: cleverhandle

Originally posted by: BingBongWongFooey
Have I missed something?

Not that I'm aware of, and I'd think that between misc@ and bugtraq something would've shown up if it were really out there.

Yeah, everything I've seen is speculation and rumor. Even the NetBSD security advisory contains:

No evidence to support remote root exploitability has been provided by
any source.

.....

There is a lot of commotion over this buffer issue. Individuals have
mentioned an increased occurrence of port scans searching for open sshd
services. Since hard facts are not available yet, individuals
will have to decide whether to believe the rumours, and apply patches to
protect against this possible issue, or to use workarounds provided
below, where appropriate, and await further information.

.....

Thanks To
=========
....
The Full-Disclosure rumour mill.

For the meantime I've just shut off ssh access to the outside.

edit: From cert:

While the full impact of this issues are unclear, the most likely result is heap corruption, which could lead to a denial of service.

If it is possible for an attacker to execute arbitrary code, then they may be able to so with the privileges of the user running the sshd process, typically root. This impact may be limited on systems using the privilege separation (privsep) feature available in OpenSSH.

http://www.cert.org/advisories/CA-2003-24.html

Note the "if it is possible ... to execute arbitrary code."

Debian advisories all say "It is not known if these bugs are exploitable."

bleh, this sucks.

 

[Electrode]

Originally posted by: BingBongWongFooey
It wasn't necessarily due to the SSH issue from the past few days.

But it may have been due to an SSH issue from longer ago. We don't know what version of OpenSSH he was using...

 

[Barnaby W. Füi]

Originally posted by: Electrode

Originally posted by: BingBongWongFooey
It wasn't necessarily due to the SSH issue from the past few days.

But it may have been due to an SSH issue from longer ago. We don't know what version of OpenSSH he was using...

It could have also had nothing to do with ssh itself, it could have just been a horrible password or something.

 

[flippinfleck]

Originally posted by: BingBongWongFooey


It could have also had nothing to do with ssh itself, it could have just been a horrible password or something.

Ding Ding Ding!

All that's left to do now is pick up the mess.

Would it be better to just format and start over?

 

[cleverhandle]

Originally posted by: flippinfleck
Would it be better to just format and start over?

Yes, absolutely. I thought that had been established. Once a box is compromised, you can't trust it. Period. Save your data, giving an especially close eye to PHP scripts and their ilk, and start over.

 

[flippinfleck]

Originally posted by: cleverhandle

Originally posted by: flippinfleck
Would it be better to just format and start over?

Yes, absolutely. I thought that had been established. Once a box is compromised, you can't trust it. Period. Save your data, giving an especially close eye to PHP scripts and their ilk, and start over.

All that was said was to "Reinstall all software from trusted media". Just wanted to clarify that for myself.

 

[cleverhandle]

Originally posted by: flippinfleck
All that was said was to "Reinstall all software from trusted media". Just wanted to clarify that for myself.

OK, fair enough - I may be reading too much into your post. If you 1) have backups, and 2) are certain those backups date before any intrusion, then go ahead and restore. But that second point is often pretty iffy.

 

[flippinfleck]

The only things that will be brought from backups are his users sites. They aren't allowed to use custom scripts so it shouldn't take too long to verify those.

 

[drag]

More then likely you aren't going to find out how he got in. Unless you want to take the computer off line completely and snoop around using a boot CD or something.

More then likely the cracker simply erased the log file entries that would show him breaking in. The fact that he left traces in the FTP logs and the SSH logs shows that he is either a complete amature, a script kiddie, or thought so little of your security that he figured you wouldn't notice.

Think of it as a learning experiance.... Boot up with a live-linux CD and compare the binaries of key files to see if they've been changed from the normal binaries that come with whatever distro you've been using. Like login... Chroot to the OS on the harddrive and then use "which login" then md5sum to check the checksums of the file.

Also check the bash history files and see if you can see what he was up to. Stuff like that.

You may be able to find out what other computers he may have comprimised on your network and what attacks he was trying. Or at least get a idea of what the hell he was trying to do with your computer. read here about this stuff

Then maybe you would want to do a security audit of your company or something. Usually its a 1000 times easier to call someone up and trick them into giving you a password then it is to spend months trying to crack a single machine. If he was a scprit kiddie it could of been dumb luck that he stumbled opon your server. But you can never tell.

Then if you realy what to figure out what happenned. Set up this server back to it's original condition and see if the guy tries to comprimise it again. Of course you go thru extra steps to protect your network, but it would be a interesting to run a honeypot.

For all you know the guy could of just been drunk or something the last time he logged in on your server and could of been careless. Who knows what he's been up to once he got a hold of it.