fubar mail at work

[Jeff7181]

Originally posted by: OdiN

Originally posted by: RagingBITCH

Originally posted by: BoberFett

Originally posted by: OdiN

Originally posted by: TheEarthWillShake
There's no backup because management felt it cost too much.

That is no excuse.

Sure. Try convincing stubborn management at a small company to part with the money required for a system capable of doing regular 1.5TB backups. That'll work. :roll:

Have you ever tried selling disaster recovery to a company who has never had to recover from a disaster? "Why the hell do we need backups? We've got this expensive RAID system?"

It isn't an excuse. It's just a very expensive and harsh lesson for the management of the company. 1.5TB backups isn't that expensive anymore nowadays.

Especially with compression.

And there are cheaper methods out there - even though they aren't as good of a solution. Like rotating external hard drives. Sure not the greatest, but it's SOMETHING and it's not THAT expensive.

For $1,000 you can buy 4 quality external hard drives and a nice padded brief case. Two get used for backup, at the end of the week they're taken off site (have an admin take them home) in the nifty padded brief case, and the other two are attached for the next week's backups.

Even if the drives on site are corrupted, you still have the disks off site that you can restore from and lose just a week's worth of data as opposed to all of it.

It's not ideal, but I think most companies can find $1,000 in their IT budget for something like that. If they can spend more, use something like Iron Mountain to keep your off site backups safe.

 

[OdiN]

Originally posted by: Jeff7181

Originally posted by: OdiN

Originally posted by: RagingBITCH

Originally posted by: BoberFett

Originally posted by: OdiN

Originally posted by: TheEarthWillShake
There's no backup because management felt it cost too much.

That is no excuse.

Sure. Try convincing stubborn management at a small company to part with the money required for a system capable of doing regular 1.5TB backups. That'll work. :roll:

Have you ever tried selling disaster recovery to a company who has never had to recover from a disaster? "Why the hell do we need backups? We've got this expensive RAID system?"

It isn't an excuse. It's just a very expensive and harsh lesson for the management of the company. 1.5TB backups isn't that expensive anymore nowadays.

Especially with compression.

And there are cheaper methods out there - even though they aren't as good of a solution. Like rotating external hard drives. Sure not the greatest, but it's SOMETHING and it's not THAT expensive.

For $1,000 you can buy 4 quality external hard drives and a nice padded brief case. Two get used for backup, at the end of the week they're taken off site (have an admin take them home) in the nifty padded brief case, and the other two are attached for the next week's backups.

Even if the drives on site are corrupted, you still have the disks off site that you can restore from and lose just a week's worth of data as opposed to all of it.

It's not ideal, but I think most companies can find $1,000 in their IT budget for something like that. If they can spend more, use something like Iron Mountain to keep your off site backups safe.

Exactly. It's doable, and that's why I say there is no excuse.

Hell if they don't want to use a real software like Veritas I could AT LEAST code SOMETHING up that would keep critical data backed up.

You could even have the drives stored at an actual data vault. We have bunches of tapes at work stored off site and it costs like $25/month for the service.

 

[Jeff7181]

Originally posted by: OdiN

Originally posted by: Jeff7181

Originally posted by: OdiN

Originally posted by: RagingBITCH

Originally posted by: BoberFett

Originally posted by: OdiN

Originally posted by: TheEarthWillShake
There's no backup because management felt it cost too much.

That is no excuse.

Sure. Try convincing stubborn management at a small company to part with the money required for a system capable of doing regular 1.5TB backups. That'll work. :roll:

Have you ever tried selling disaster recovery to a company who has never had to recover from a disaster? "Why the hell do we need backups? We've got this expensive RAID system?"

It isn't an excuse. It's just a very expensive and harsh lesson for the management of the company. 1.5TB backups isn't that expensive anymore nowadays.

Especially with compression.

And there are cheaper methods out there - even though they aren't as good of a solution. Like rotating external hard drives. Sure not the greatest, but it's SOMETHING and it's not THAT expensive.

For $1,000 you can buy 4 quality external hard drives and a nice padded brief case. Two get used for backup, at the end of the week they're taken off site (have an admin take them home) in the nifty padded brief case, and the other two are attached for the next week's backups.

Even if the drives on site are corrupted, you still have the disks off site that you can restore from and lose just a week's worth of data as opposed to all of it.

It's not ideal, but I think most companies can find $1,000 in their IT budget for something like that. If they can spend more, use something like Iron Mountain to keep your off site backups safe.

Exactly. It's doable, and that's why I say there is no excuse.

Hell if they don't want to use a real software like Veritas I could AT LEAST code SOMETHING up that would keep critical data backed up.

You could even have the drives stored at an actual data vault. We have bunches of tapes at work stored off site and it costs like $25/month for the service.

Even using Windows' built in backup overnight would be better than nothing. Full backup once a week and incremental backups every night until the next week starts.

 

[Scarpozzi]

What kind of array was it?

Most 30 disk arrays would be in SAN territory. If it was just 2 direct attached SCSI shelves RAIDed somehow, I would go into the RAID controller and try forcing all but one of the 'failed' disks back online and set the last for a rebuild.

What you run into a lot of times is the RAID array getting out of sync somehow....once this happens, the data isn't always corrupt, but you may definitely have some coruption on the Exchange side. Those stupid database-driven Email systems are weak that way. (ie. Groupwise, Exchange etc) I'm a much bigger fan of the more utilitarian systems like Sendmail and M+Netmail.

 

[Goosemaster]

This smells fishy.

Even the most basic array has redundancy options. I could go pee all over my servers and it'd recover. Someone obviously screwed up royally.


Also, there is no excuse for a lack of a backup. No excuse.
<--had to a raise a storm to get it prioritized at work and even then...urgh..I end up doing it all myself but whatever. Shit needs to get done.

 

[wetcat007]

I've never had a "bump" damage a hard drive before. Did they fall 20 feet or something?

 

[OdiN]

Just a little bump....

http://www.youtube.com/watch?v=UIzQzocv-TI

 

[Goosemaster]

Originally posted by: wetcat007
I've never had a "bump" damage a hard drive before. Did they fall 20 feet or something?

 

[Mahaguru]

So what will be the outcome?

 

[MagnusTheBrewer]

Originally posted by: Goosemaster
This smells fishy.

Shit needs to get done.

Ya mean like updating the whistle blower thread?

 

[darkxshade]

bump it again, you might fix a few

 

[child of wonder]

Nasty. I can't believe there were no backups for that much data.

We've been moving to a virtual environment at work and so far have 15 VMs running off a 2TB SAN. A couple months ago I went to my boss and said we need to find money to do bare metal backups of the VM images both locally and remotely. He says "isn't that why we have the SAN?"

*forehead slap*

 

[StinkyPinky]

Originally posted by: WarhammerUC
someone bumped the disk array, cause 8 disk ! out of 30 to go out of whack... lost 1.5tb data so far... and no backup...

i might not be around tomorrow, im sure they'll just get rid of the group..

Wtf? No backup? To me that's about as strange as waking up to find yourself getting probed anally by Fred Flintstone. How can you have no backup? Shit, we backup our backups at work. Lack of budget is no excuse.

 

[evident]

update!?!?!?!

 

[Goosemaster]

Originally posted by: StinkyPinky

Originally posted by: WarhammerUC
someone bumped the disk array, cause 8 disk ! out of 30 to go out of whack... lost 1.5tb data so far... and no backup...

i might not be around tomorrow, im sure they'll just get rid of the group..

Wtf? No backup? To me that's about as strange as waking up to find yourself getting probed anally by Fred Flintstone. How can you have no backup? Shit, we backup our backups at work. Lack of budget is no excuse.

 

[mugs]

Originally posted by: zoiks
We had a dumbass admin who blew away a vm by issuing a 'rm -rf'. We got our data back somehow but the admin was outta there.

Every time I have to delete files on a customer server my testicles shrink.

 

[Shawn]

Originally posted by: wetcat007
I've never had a "bump" damage a hard drive before. Did they fall 20 feet or something?

sounds like someone knocked over the server rack. :laugh:

 

[Goosemaster]

Originally posted by: Shawn

Originally posted by: wetcat007
I've never had a "bump" damage a hard drive before. Did they fall 20 feet or something?

sounds like someone knocked over the server rack. :laugh:

the pics would be glorious....

 

[All I Need]

Originally posted by: child of wonder
Nasty. I can't believe there were no backups for that much data.

We've been moving to a virtual environment at work and so far have 15 VMs running off a 2TB SAN. A couple months ago I went to my boss and said we need to find money to do bare metal backups of the VM images both locally and remotely. He says "isn't that why we have the SAN?"

*forehead slap*

That's why he's your boss.

 

[SoulAssassin]

Originally posted by: Goosemaster
Even the most basic array has redundancy options. I could go pee all over my servers and it'd recover. Someone obviously screwed up royally.

Right, but even if it's doing RAID 6 and he has 8 of 30 disks shit the bed you are hosed.

 

[Ns1]

NO UPDATES!?

ARE WE ABOUT TO GET GOOSED?!

 

[Homerboy]

IF thats how your company runs, I'm guessing NOBODY will be around very long. Sounds like a pure shit operation and/or an IT department with no balls to tell "management" what needs to be done.
Horrible failure on all levels.

 

[WarhammerUC]

basically Im just another senior systems engineer. The jr guy hasn't been doing his work so i had to pick up his slack. So they assign him BACKUP and POWER.

While replacing ups, he pulls the power to the array. 8 disk fell out of the raid and the array was not visible, storage toasted..

2nd issue was he was assign to build a new backup tape server and an anti-virus server. Both have not been done for over 1 month.

its okay now, i been backing up the server to disk, secretly on usb... so it kinda look like i saved the day but i didn't.. took a while to restore the data and we had to use back fill from our external vendor..

 

[aigomorla]

Originally posted by: her209
Why are you smiling OP?

thats what im thinkn as well.

:\

even for my home i have a dedicated server that does weekly backups.

However i expect my raid array to die one day.

You guys need a backup. :

If you dont have one, theres no point in you having servers if theres a possibility it can all die and go poof at any moment.

 

[RightIsWrong]

Originally posted by: WarhammerUC
basically Im just another senior systems engineer. The jr guy hasn't been doing his work so i had to pick up his slack. So they assign him BACKUP and POWER.

While replacing ups, he pulls the power to the array. 8 disk fell out of the raid and the array was not visible, storage toasted..

2nd issue was he was assign to build a new backup tape server and an anti-virus server. Both have not been done for over 1 month.

its okay now, i been backing up the server to disk, secretly on usb... so it kinda look like i saved the day but i didn't.. took a while to restore the data and we had to use back fill from our external vendor..

So you backed up a 1.5TB Information Store onto a USB drive without anyone knowing?

The time it would take to copy that much data alone would cause your server to slow enough to be noticed. Not to mention that any company that has a 1.5TB Info Store has got to be big enough to have had disaster recovery plans implemented and would have ensured that they could be utilized by doing routine testing.

I'm gonna have to call shens.