Man deletes entire company with one line of code

[OutHouse]

i should send this to my management to make them feel good about approving my un-budgeted 200K Purchase order for a Veeam and Exagrid backup/replication project.

 

[HamburgerBoy]

He claimed a couple days later that he managed to recover nearly everything and that it was no biggie. How do we know this isn't just a troll looking to make a headline somewhere? Like, what is the name of this Marco Marsala's company?

 

[Elixer]

I am not buying this story, this is more like a troll post on that forum.
If he was dumb enough to be root, and execute rm -rf / then, don't come crying, instead, spend the thousands of $$$ to a recovery service.

 

[Carson Dyle]

He claimed a couple days later that he managed to recover nearly everything and that it was no biggie. How do we know this isn't just a troll looking to make a headline somewhere? Like, what is the name of this Marco Marsala's company?

You mean like "There's no such thing as bad news"?

If you're a hosting company, there is such a thing. And this would be just about as bad as it gets. Whether or not it's a true story, though, is another thing.

 

[thesmokingman]

You mean like "There's no such thing as bad news"?

If you're a hosting company, there is such a thing. And this would be just about as bad as it gets. Whether or not it's a true story, though, is another thing.

How's this the hosting co's fault or bad news when the client burnt themselves? That said, yea it doesn't seem genuine with the post comments.

 

[Carson Dyle]

How's this the hosting co's fault or bad news when the client burnt themselves? That said, yea it doesn't seem genuine with the post comments.

His was a hosting company, with some 1535 customers, according to his post.

 

[thesmokingman]

His was a hosting company, with some 1535 customers, according to his post.

This really seems fake then. :sneaky:

 

[Carson Dyle]

Looking at his posting history over there, it really does look like he was doing some hosting. Saving grace is that he doesn't mention the company name.

I'm back to 'true story', although he seems dumb enough that his original assessment of having deleted everything, including backups, was probably incorrect.

 

[flexy]

If he *really* had his entire hosting company on one server/disk...and then for whatever reason ran a rm -rf there it's actually better he "deleted" his company. (Even if I doubt this story).

If this is true, think about the affected 1500 customers as well. How could you run an entire business plus the businesses of fricking 1500 clients without fool proof, remote backups?

 

[BoberFett]

Also, am I the only one who is annoyed at that being referred to as "a line of code"?

Going to a command line and typing a command is not "code".

 

[Ns1]

Going to a command line and typing a command is not "code".

but it was supposedly part of a script

 

[thesmokingman]

If this is true, think about the affected 1500 customers as well. How could you run an entire business plus the businesses of fricking 1500 clients without fool proof, remote backups?

If I had a company with 1.5K clients, I can tell you for sure that I would NOT be working on the front lines day in day out. I'd be golfing or whatever something liek that.

 

[Childs]

but it was supposedly part of a script

Yeah, he had variables for the file path that werent set ($var1/$var2), which resulted in removing /. Someone mentioned in the comments on serverfault that this technically shouldnt have even been possible if he didnt pass --no-preserve-root unless he had a wildcard in the second variable ($var2) resulting in /*. I get that someone can make a mistake, even a series of mistakes, but at least try it once on a test machine to verify the script does what you think it should do.

 

[Ns1]

Yeah, he had variables for the file path that werent set ($var1/$var2), which resulted in removing /. Someone mentioned in the comments on serverfault that this technically shouldnt have even been possible if he didnt pass --no-preserve-root unless he had a wildcard in the second variable ($var2) resulting in /*. I get that someone can make a mistake, even a series of mistakes, but at least try it once on a test machine to verify the script does what you think it should do.

 

[master_shake_]

Worse than Linus with his 3 raid 5's striped in Windows

 

[TwiceOver]

That's why you don't set your variables empty...

 

[Carson Dyle]

If I had a company with 1.5K clients, I can tell you for sure that I would NOT be working on the front lines day in day out. I'd be golfing or whatever something liek that.

1500 shared hosting customers is like having 1500 customers on a paper route. He probably makes pennies a month off of most of them.

 

[Exterous]

I'd like to say I'm surprised but its amazing how many people\companies ignore best practices and ignore giant holes in backup\DR plans

no he didnt.

All servers got deleted and the offsite backups too because the remote storage was mounted just before by the same script (that is a backup maintenance script).

 

[werepossum]

no he didnt.

Can one actually mount a drive that is offsite? If so, how does one keep malware from wiping out the backup at the same time?

1500 shared hosting customers is like having 1500 customers on a paper route. He probably makes pennies a month off of most of them.

No doubt. But if all those pennies go away at the same time . . .

No GI Joe with the kung fu grip, no happy child, no Christmas nookie.

 

[Carson Dyle]

Can one actually mount a drive that is offsite? If so, how does one keep malware from wiping out the backup at the same time?

Who needs malware when you're doing it yourself?

No doubt. But if all those pennies go away at the same time . . .

Nobody wants to lose any income. That this would be a business large enough to have enough employees that the owner could just sit back and watch isn't how that game is played.

 

[Mark R]

Can one actually mount a drive that is offsite? If so, how does one keep malware from wiping out the backup at the same time?

Yup. You just access a shared drive over a WAN link, e.g. an NFS shared volume. No reason why this can't be done over VPN.

Of course, you'd be nuts to do this with a backup. If you have the shared drive mounted read-write, then any malware on the system could toast the backup.

That's why you don't backup to a network share. You either use a different type of protocol, e.g. rsync, FTP, etc. Ideally, you would use a snapshotting file system. Trigger a snapshot, and then rsync or FTP the snapshots off to the backup server. Alternatively, you run the backup software on the backup server, and on your production machine you expose the data drive or snapshot files as a read-only share. The backup server connects to the share read-only and copies all the data off.

For the truly paranoid, the backup server can be firewalled off, except for when a backup is due. When a backup is due a cron job on the backup system temporarily lowers the firewall, initiates the backup, and when the backup is complete the firewall is brought back up. Taking this to extremes, you can use multiple backup servers which operate in rotation, but where only one has an open path through the firewall at any time.

 

[dave_the_nerd]

SAN Snapshots, people.

Anyway, that was incredibly boneheaded. His livelihood is gone, and now that this has been publicized, he'll never pass the Google test again either.

I suppose some private elementary school needs somebody to maintain their computer lab and help the 85 year old secretary figure out how to use MS Word.

 

[Exterous]

Can one actually mount a drive that is offsite? If so, how does one keep malware from wiping out the backup at the same time?

Yes and don't use your admin\server\storage machine(s) for web browsing

SAN Snapshots, people.

Anyway, that was incredibly boneheaded. His livelihood is gone, and now that this has been publicized, he'll never pass the Google test again either.

I'll bet he can just change his company's name and continue on. Shady contractors do that all the time

 

[Kaido]

How do you not have a CDP, or at least offline backups? An 8TB USB drive is $225 on Amazon, c'mon! Backblaze is what, five bucks a month for unlimited cloud storage? Reminds me of that horror story from Pixar, when they almost lost the ENTIRE MOVIE:

https://www.youtube.com/watch?v=8dhp_20j0Ys

 

[DigDog]

No idea how somebody who runs that many sites would be so idiotic to remain in business this long.