This next story didn't affect me at all but has our head DBA ready to down a fifth of Jack Daniels. A few months back we partnered with a distributor on the west coast and we needed a way to link our pricing, their pricing, inventory, leadtimes, etc. Our database dude comes up with a kickass frontend and a backend build on ColdFusion and SQL Server. The OS is Windows 2000 Server running IIS. All this sits on a server I built (Pretty kickass too). Supermicro mobo (440BX), dual PIII-850s, 512MB ECC SDRAM, 2 x 9.1GB SCSI HDD, Adaptec Hardware RAID Adapter (16MB), 10/20GB Tape Backup, 400W PS, oversized APC Smart-UPS, hugeass case, and tons of fans. Did I forget to mention the 2MB Diamond Stealth video card in there? Anyway, back to the story....
We wanted to make it as idiotproof as possible. The backup runs everynight automatically, the UPS is large enough to hold the system for 90 minutes (yup 90) in the event of a power outage, the mirrored hard drives protect against physical failure, and to top it off, we included an emergency pack. In the pack was wrote detailed instructions for every contingincy short of someone dousing it gasoline and setting it on fire. We also included bootdisks for emergency recovery etc. should anything go horribly wrong. For a month we stress tested the make sure it took over 200% more traffic than we anticipated. The hardware passed with flying colors and his database chugged along with no problems. He then personally few to the distributor's site and installed everything.
So what happens? Some idiot decides, "I'm gonna upgrade the software and load on tons of patches." Apparently something went really wrong (mild understatement) and the server would just go straight into a memory dump when rebooted. We thought, "OK, managable problem, pop in the emergency disks and they would be up and running in about 1 hour." No such luck, the guy actually opened up the server and played around with the hard drives. The RAID card didn't like it and insisted on rebuilding the mirror. The DBA told me to go home and he could handle it on his own.
This morning he looked terrible. Seems after I left and the mirror rebuilt (2 hours), the guy finally confesses that no one knows where the rescue pack was stored. Uh oh.... inside was everything including original copies of Windows 2000. They had run out and get a copy of Windows 2000. I was told walking the guy through the reinstall was like eating glass. Iritating as hell and so painful you wish death would come. The dork kept on jumping the gun and butting in with all sorts of dumb comments. One of the toughest parts was getting the guy to download the drivers for the RAID card. He barely knew how to use IE5 and yet he started messing with a mission critical server?
Sometime around midnight they finally managed to get a backup restored. 24 hours worth of work was lost and there were a lot of ticked off people today. Millions in orders, quotes, RMA, reports etc need to be reentered. Our DBA filed a formal complaint with this moron's supervisor.
Unfortunatly, it get worse for Database Dude. About ten minutes later, he gets a call that his girlfriend was in a serious car accident. She is ok but the car is a total loss. Makes my bad days seem like a summerday camp. Poor guy, we need to get him some booze quick.
Windogg
We wanted to make it as idiotproof as possible. The backup runs everynight automatically, the UPS is large enough to hold the system for 90 minutes (yup 90) in the event of a power outage, the mirrored hard drives protect against physical failure, and to top it off, we included an emergency pack. In the pack was wrote detailed instructions for every contingincy short of someone dousing it gasoline and setting it on fire. We also included bootdisks for emergency recovery etc. should anything go horribly wrong. For a month we stress tested the make sure it took over 200% more traffic than we anticipated. The hardware passed with flying colors and his database chugged along with no problems. He then personally few to the distributor's site and installed everything.
So what happens? Some idiot decides, "I'm gonna upgrade the software and load on tons of patches." Apparently something went really wrong (mild understatement) and the server would just go straight into a memory dump when rebooted. We thought, "OK, managable problem, pop in the emergency disks and they would be up and running in about 1 hour." No such luck, the guy actually opened up the server and played around with the hard drives. The RAID card didn't like it and insisted on rebuilding the mirror. The DBA told me to go home and he could handle it on his own.
This morning he looked terrible. Seems after I left and the mirror rebuilt (2 hours), the guy finally confesses that no one knows where the rescue pack was stored. Uh oh.... inside was everything including original copies of Windows 2000. They had run out and get a copy of Windows 2000. I was told walking the guy through the reinstall was like eating glass. Iritating as hell and so painful you wish death would come. The dork kept on jumping the gun and butting in with all sorts of dumb comments. One of the toughest parts was getting the guy to download the drivers for the RAID card. He barely knew how to use IE5 and yet he started messing with a mission critical server?
Sometime around midnight they finally managed to get a backup restored. 24 hours worth of work was lost and there were a lot of ticked off people today. Millions in orders, quotes, RMA, reports etc need to be reentered. Our DBA filed a formal complaint with this moron's supervisor.
Unfortunatly, it get worse for Database Dude. About ten minutes later, he gets a call that his girlfriend was in a serious car accident. She is ok but the car is a total loss. Makes my bad days seem like a summerday camp. Poor guy, we need to get him some booze quick.
Windogg
Facebook
Twitter