My employee updated all records in a database table...

[KingGheedora]

What would you do in this situation? I know enough about the sql inner workings to figure out who ran the query that caused the problem, and when, and i even have the text of the query that updated all the records.

Anyways, one of the main tables we use as part of our operation is not part of our production environment, and we aren't losing any money (this is exactly why we don't allow developers access to our production db) as a result of this mistake. I can even recover >99% of the three columns that were updated because luckily i created a backup of the table last night for some unrelated tests I was running. Our true nightly backup already overwrote the previous night's backup with the fucked up version of the table so that backup is useless. The data in this table is very usefuland we report off of it all the time.

I'm mainly pissed that this happened and the employee didn't fucking tell me he did this. If he had and this had been any other day, I wouldn't have the personal backup, but I would have been able to recover from the nightly backup. His not telling me would have fucked us over and made all the historical data basically useless.

Should I just fix it and send a message directly to him, what would I say in that case? "I noticed that all the records in the table were updated because you ran an update statement last night without a proper where condition. Please be more careful with this in the future."

Should i send a message to the team and not really single him out at all?

Probably either of the above two options would also include an office-politically-correct reprise of paragraph #3 of this post. I plan to let my boss know this happened in a private message.

 

[imported_Devine]

This looks like a pretty big issue but if nobody else knew about it I'd kinda keep it on the down low as much as possible unless I wanted to get the guy fired. Just pull him aside and talk to him about and see what his side of the story is. Maybe he didnt know wtf he was doing and didnt realize he fucked the stuff up. Either way you need to train his ass and/or lock down permissions to safe guard your data even more and maybe you should look into having a more thorough backup system in place to prevent something like this from happening again.

 

[KingGheedora]

Please give more opinions, I need to act first thing tomorrow morning. Should I really keep this from my boss?

 

[her209]

I thought your were his boss.

 

[KingGheedora]

I plan to put a trigger on the table that will throw an error if someone tries to update more than 1,000 records at once. Fuck i don't want to have to deal with managing permissions. I have to now deny everyone's dbo access to the db now, and they will have to submit scripts to me to be run on this server from now on to make changes. They can still update data but no more permissions to make modifications because that would mean they could bypass the restrictions on the table.

 

[KingGheedora]

I am. So are you saying my boss doesn't need to know about this? I couldn't fire him without my boss' permission anyways, and my boss would not fire him unless I wanted to do so (I don't really, but I'm pretty pissed about this).

 

[akshatp]

You dont keep backups past one night??? Poor planning if you ask me. You should be at minimum on a weekly rotation of tapes. Are you telling me you use one tape for backup and over write it every night?

 

[imported_Devine]

Well what good will come of you letting your boss know? Would he understand anything you tell him other then "blah blah blah Tim blah blah blah broke it blah blah..." ? I understand your anger but thankfully you fixed the issue without much problem. Take a little bit to cool off so you can talk to the guy rationally and not emotionally. Yelling at the guy will only put him on the defensive and you might not figure out wtf he was thinking. I know I try to keep everything in house as possible, if it doesnt help me or how my shop looks then I dont think there's any reason to air your dirty laundry. Talk to the guy and note everything that happened just incase it comes up in the future and higher ups want details.

 

[cubby1223]

Originally posted by: KingGheedora
Our true nightly backup already overwrote the previous night's backup with the fucked up version of the table so that backup is useless.

There's the problem.

Whoever is in charge of the backup system screwed up royally when setting it up.

Make sure your boss knows that the person in charge of backups should be fired.

If you do not anticipate mistakes, then that is your own damn fault. :roll: Stop blaming someone else.

 

[skace]

He should have told you, and I would let him know that, however these 2 things mean his ass is covered:
It's not a production database.
A backup system that can't go back 2 nights is effectively broken. (restore from tape?)

Oh and if the data is important to you then it shouldn't be in a non-production database....

 

[biggestmuff]

I'm not quite sure what you position is over him. If you're his shift supervisor or section lead/head, I'd do the following:

-Pull him aside and tell him how bad he fucked up, only more eloquently
-Have a shift/team meeting telling everyone what happened, but DO NOT single him out; this will provide a 'lessons learned' to your group.


- And, of course, your backup system needs to be re-evaluated.

 

[Homerboy]

Originally posted by: cubby1223

Originally posted by: KingGheedora
Our true nightly backup already overwrote the previous night's backup with the fucked up version of the table so that backup is useless.

There's the problem.

Whoever is in charge of the backup system screwed up royally when setting it up.

Make sure your boss knows that the person in charge of backups should be fired.

If you do not anticipate mistakes, then that is your own damn fault. :roll: Stop blaming someone else.

I have to agree that this is a much bigger problem. ESPECIALLY if it works this way in your production too.

As for the idiot that caused the problem. Serious written warning. 1 mistake doesnt warrant a firing... but you MUST record the mistake in detail, have them read the report and sign off on it.

 

[cubby1223]

Originally posted by: skace
He should have told you

It's a very simple mistake to make, and I can just about guarantee the person did not realize the mistake was even made. Can't report something you don't know about yourself.



That's why the majority of the blame goes to whoever devised the backup procedure. Whenever anyone has access to submit their own SQL statements, mistakes have to be anticipated, and you cannot rely on the person making the mistake to notice it right away.

I made a similar mistake a month ago, caused a large number of web pages to display the wrong text. I didn't notice it, nor did anyone else notice the mistake for over 8 days. But we have the backups in place where it was no trouble at all to go back that far.

 

[pontifex]

you only keep 1 day's worth of backups? wow...

our system backs up every day (except Sat & Sun). i have like 2 weeks worth of daily tapes. the last backup of the month is kept for 1 year and the last backup of the year is kept for 7 years.

 

[JD50]

Only 1 day of backups??

 

[tranceport]

Sounds like the real problem is your backup and recovery solution design.

 

[Blieb]

I've done this before to a production database. I think I did it 3 times actually ...

Semi before the where clause got me ...

Anyways ... I immediately reported the incident and we devised a way to fix from backup.

If it's DEV ... no biggie ... just maybe let the person know. It's possible that they were just blazin' along and didn't even realize that they did this. IF on the other hand, they know they fucked up and thought by not saying anything that they would get away with it ... I'd fire them.

 

[sdifox]

my standard operating procedure if I am doing any sort of direct manipulation of the data. I do a manual backup first. Be it the whole db if it is small enough or just the tables related to the update if not.
I also keep 3 layers of backup, daily on hdd (2 locations), weekly (3 locations) and monthly (offsite)

But to be truthful, in some operations, db backup past a day is useless.

 

[nageov3t]

fix it and give him a warning to put the fear of god into him.

 

[Vette73]

Has anybody brought up that the backup design is the problem? just a thought.

 

[Bulk Beef]

Deal with it at the lowest level.

And fix your backup plan. One night is just fucking stupid.

 

[Injury]

I agree with the written warning. It would be a different story if he did it accidentally then told you ASAP, but he either didn't know it happened or was hoping nobody would notice, which is incompetence either way.

 

[xSauronx]

Originally posted by: akshatp
You dont keep backups past one night??? Poor planning if you ask me. You should be at minimum on a weekly rotation of tapes. Are you telling me you use one tape for backup and over write it every night?

this is what hit me first. who wouldnt keep separate daily backups for this kind of thing?

 

[ViviTheMage]

you need to keep daily backups for as long as, if not longer then a week...not just daily, then overwrite that daily with the current daily.

inform the user directly imo, as well tell your supervisor.

 

[Ramma2]

First, yes, fix your backup schedule. Thats messed up.

Second, I don't think we need to be talking about write ups or firing or anything like that. It sounds like it was an honest mistake, as mentioned he may not even know what happened. Sit with the poor lad, talk to him about what happened, and make changes so that it doesn't happen again.