How exactly does Lockheed Martin screw up ACTS this bad?

[Elixer]

The database of the Air Force's Automated Case Tracking System (ACTS)—which is used by the Air Force Inspector General's Office to manage investigations into complaints from whistleblowers of waste, fraud, and abuse; Freedom Of Information Act requests; and congressional inquiries—has become corrupted, rendering over 100,000 case files dating back to 2004 unreadable. And because of the way the database was backed up, an Air Force spokesperson said that neither the service nor Lockheed Martin—the contractor that operates the ATCS system for the Air Force—can recover the data.

"The database crashed and there is no data," Ann Stefanak of Air Force Media Operations said in a statement to press. "We’ve kind of exhausted everything we can to recover [the data internally]... and now we’re going to outside experts to see if they can help." Efforts are being made to see if the data was backed up in other locations, and the Air Force has begun asking for help from other organizations within the Department of Defense and from outside experts in recovering the database's contents.

I don't get it, what kind of a back up plan were they doing? Using the same tape over and over again with only 1 copy?
They were mandated to keep all records, so... the dog ate my database?

This smells like a inside job, trying to hide something.

*edit, http://www.airforcetimes.com/story/...eports-limbo-after-systemwide-crash/85865726/

UPDATE: The Air Force released this statement on June 15, 2016:

Through extensive data recovery efforts over the weekend and this week, the Air Force has been able to regain access to the data in the Air Force Inspector General Automated Case Tracking System. Once we ensure full database stability, we'll bring the ACTS database back online for users to access. The Air Force is still investigating the incident and will implement all required corrective measures. The secretary of the Air Force has directed an independent review of the incident.

Air Force spokeswoman Ann Stefanek said all 100,000 reports, including pending or incomplete IG investigations, have been recovered. The service and contractor Lockheed Martin are still working to stabilize the system before users can begin to use the database, she told Air Force Times.

Earlier on Wednesday, Chief of Staff of the Air Force Gen. Mark Welsh said that in additional to Lockheed, the 24th Air Force, the cyber division of the service, and Oracle, another defense contractor, were all working to restore the database.

“Hopefully, we’ll be able to get this data recovered and there won’t be a long-term impact, other than making sure we understand exactly what happened, how it happened and how we keep it from ever happening again,” Welsh said during a defense writers group breakfast in Washington, D.C.

 

[sdifox]

wtf how does Lockheed even stay in business if that is their backup plan?

 

[GeekDrew]

It wouldn't surprise me if they were backing up database files while they were in-use, without asking the database engine to prepare them for backup. It's entirely possible that they've had backups misconfigured the entire time, and never tried to test them.

 

[Darwin333]

FFS my companies server is backed up better than that.

 

[pauldun170]

Update
http://www.airforcetimes.com/story/...eports-limbo-after-systemwide-crash/85865726/

UPDATE: The Air Force released this statement on June 15, 2016:

Through extensive data recovery efforts over the weekend and this week, the Air Force has been able to regain access to the data in the Air Force Inspector General Automated Case Tracking System. Once we ensure full database stability, we'll bring the ACTS database back online for users to access. The Air Force is still investigating the incident and will implement all required corrective measures. The secretary of the Air Force has directed an independent review of the incident.

Air Force spokeswoman Ann Stefanek said all 100,000 reports, including pending or incomplete IG investigations, have been recovered. The service and contractor Lockheed Martin are still working to stabilize the system before users can begin to use the database, she told Air Force Times.

Earlier on Wednesday, Chief of Staff of the Air Force Gen. Mark Welsh said that in additional to Lockheed, the 24th Air Force, the cyber division of the service, and Oracle, another defense contractor, were all working to restore the database.

“Hopefully, we’ll be able to get this data recovered and there won’t be a long-term impact, other than making sure we understand exactly what happened, how it happened and how we keep it from ever happening again,” Welsh said during a defense writers group breakfast in Washington, D.C.

The story below originally published on June 14.

 

[Elixer]

Update
http://www.airforcetimes.com/story/...eports-limbo-after-systemwide-crash/85865726/

Yeah, was updating my post when you posted this.

I would terminate Lockheed Martin's contract, and fine them $40 million for this fiasco.

 

[PottedMeat]

wtf how does Lockheed even stay in business if that is their backup plan?

because boeing fucks up worse

 

[Brovane]

Well Lockheed Martin was also responsible for this.

https://en.wikipedia.org/wiki/NOAA-19

 

[sdifox]

Well Lockheed Martin was also responsible for this.

https://en.wikipedia.org/wiki/NOAA-19

Talk about corporate welfare.

 

[HeXen]

Shit happens. We've all had shit like this happen. Hell some members here probably don't have the best backup solutions either.

 

[twinrider1]

 

[Markbnj]

I don't get it, what kind of a back up plan were they doing? Using the same tape over and over again with only 1 copy?

Doesn't matter how many copies you keep if you ignore the error long enough for the bad data to overwrite them all.

 

[glenn1]

Too bad Steve Jobs isn't still around to say how iCloud would have prevented all this.

 

[Red Squirrel]

Lol, so turns out my MMO game server's backup routine is better than something Lockheed Martin came up with? My backup solution actually communicates with the game server, tells it "I'm about to do a backup", the game server will finish any writes it was doing and say "ok I'm ready" and the backup job will start to backup what is now an atomic copy of the database. (ex: say a player used gold to buy an item, that copy of the database will not have the new item AND the gold present, ex: duping) When the backup is done it will then tell the server "ok you can keep saving" and the server will catch up with all the writes.

I don't bother getting that fancy for forums and low traffic web stuff though. The game server is continuously writing delta updates to the DB so it's important to take an atomic backup.


The thing with big organizations though is that their priority is not to prevent bad stuff from happening, their priority is having someone to blame if something bad happens. That's why companies tend to contract out lot of stuff to vendors, use very expensive proprietary solutions supported by a vendor, etc... You can build a nice SAN, backup solution and entire infrastructure using stuff like Supermicro and enterpriseish hard drives and Linux and with some effort and custom code it will be just as good as something from Dell or IBM and 1/10th the cost. But if it fails you have nobody to blame.

 

[Homerboy]

n/m

 

[Homerboy]

Shit happens. We've all had shit like this happen. Hell some members here probably don't have the best backup solutions either.

Well see, they are this multi-billion company spending billions/trillions of tax payers dollars. So I'd hope their processes are a little more in-depth and fail proof than Jedi backing up his porn collection.

 

[pauldun170]

Shit happens. We've all had shit like this happen. Hell some members here probably don't have the best backup solutions either.

I think anyone who has worked at any large organization that's been around longer than a fart understands that sometimes "shit happens".

Looks like its an Oracle database. Who knows what release (since this is federal...it could be anything from version 8 to version 12)...

Even Oracle is involved.
All the due diligence in the world isn't going to help you when you discover you are impacted by an "Oracle bug" during recovery operation.

Wonder what the details are of what happened and how they system is designed.
Since it Lockheed, I'd imagine is all H2b contractors and offshore running the day to day stuff like supporting databases.

 

[tcsenter]

Through extensive data recovery efforts over the weekend and this week, the Air Force has been able to regain access to the data in the Air Force Inspector General Automated Case Tracking System. Once we ensure full database stability, we'll bring the ACTS database back online for users to access.

Well....minus the damning data for certain number of reports that could really harm some big defense contractor, of course. "Oopsie!"

 

[pauldun170]

Looks like they converted from an older system to the current system in 2004.
Wonder if the issue is data preconversion or data post conversion or plain old issue with oracle files

I'm sure DBA's across the US would just LOVE for every single screw-up that occurs to make it into the news media.

 

[lupi]

Too bad Steve Jobs isn't still around to say how iCloud would have prevented all this.

Jlaw would approve this sentiment.

 

[sdifox]

Looks like they converted from an older system to the current system in 2004.
Wonder if the issue is data preconversion or data post conversion or plain old issue with oracle files

I'm sure DBA's across the US would just LOVE for every single screw-up that occurs to make it into the news media.

Idiots probably just upgraded as opposed to ETL

 

[Homerboy]

Jlaw would approve this sentiment.

All over her face.
()

 

[BurnItDwn]

It wouldn't surprise me if they were backing up database files while they were in-use, without asking the database engine to prepare them for backup. It's entirely possible that they've had backups misconfigured the entire time, and never tried to test them.

Even if they claim to do DR testing, and claim to have Disaster Recovery processes, they might be so useless that they do nothing more than serve as lip service to shareholders.

 

[HeXen]

Well see, they are this multi-billion company spending billions/trillions of tax payers dollars. So I'd hope their processes are a little more in-depth and fail proof than Jedi backing up his porn collection.

That's funny considering every govt funded agency in the world, every large corporation in the world has had "shit happen" before. Whether it's you or an entire country, if people have any involvement then anything unintended and sometimes intended can and will happen at some point.

Perhaps you missed Chernobyl or BP..err, well any oil company for that matter.

 

[shimpster]

as tragic as this is, i dont think a database with information to manage investigations into complaints from whistleblowers of waste, fraud, and abuse; Freedom Of Information Act requests; and congressional inquiries, was ever going to be used.