Toes up RAID, 6TB to recover, what a PITA

[mikeford]

Long post ahead, mostly a blog of events, although help very welcome. I will update as I go and post final results as well.

HTPC and media server started out 3 years ago with a Biostar 760G Win7 64 and AMD SB710 RAID1 (2x2TB Seagates mirrored). Worked pretty well eventually, a year later MB video died, and I replaced MB with MSI 880GM P51 due to the same SB710 raid controller. MB video and ancient Svideo card I need for ancient TV would not function together, gave up on the TV output and focused on music, til 1/1/13 when the system beeped while nothing was going on, and restarted. Over maybe 4 restarts the system got progressively flakier, no SMART errors or error messages from the RAID, but some kind of corruption was going on.

*** Great mistake ***

System rebooted and started running chkdsk just before windows opened. I should not have allowed it to run, but this was all very quick, maybe 20 minutes from the first beep, and I was hoping for quick recovery.

I should have disconnected the RAID array drives, restored Win7 on a different drive, imaged ONE of the raid mirror pair and attempted recovery from the image.

*** What I did do.

I think I may have let chkdsk run 3 times, tried the Win7 install disc repair options, and it saw no OS. The final attempts to boot from the array failed with "missing boot manager", but it did boot a few times prior to that all the way to windows, just flakier each time.

I think Win7 and the AMD RAID have some serious, and given the number of years the product has been out, ridiculous driver issues. I did a year of Windows Update two weeks prior to the meltdown, and I think that may have precipitated the trouble.

*** Recovery

A few stumbles, but I have ditched the AMD onboard raid and set MB to IDE. Removed all drives and reinstalled Win7 64 Pro on a Intel 330 120GB SSD and applied all updates. Purchased 2x WD RED 2TB drives, upped ram to 8GB from 4GB, bought Seagate Disk Recovery Software and downloaded the free Recuva software.

Working from the clean Win7 64 Pro on the SSD, I hooked up the two WD Red 2TB drives as J and K, and used Seagate recovery to take a peek at a 2TB Seagate that dropped out my array about a year ago, that I quick formatted prior to RMA two weeks before the whole array failed. It dropped out of the array with no error, I could still read any of the files I tried, but it would not pass Seatools long test. At the time it was easiest to buy a 3rd same model drive and restore the array, which happened pretty easily.

Seagate recovery was predicting 30 hours to scan, so I stopped the scan after a few hours and looked at the results which seemed very promising. I still hoped to send the drive back within the RMA 30 day limit, so I restarted Seagate recovery in the mode to create an image file of the drive and store it on the J drive (WD Red 2TB). Despite all the drives formatting to what appeared to be exactly the same 1.819GB, Seagate recovery complained the J drive was too small for the image file and required compression, which I enabled and it reported an expected 825 or so GB file. Once started it reported 30 hrs expected completion, so I tossed the idea of timely RMA and will just fill out all the RMA forms again later this week.

27 hrs later the image file reported completion, with a size of 1.69 TB (double the prediction from the software). I restarted the Seagate recovery software and set it to deep scan the image file, with a predicted completion time of 30 hrs again, but after 22 hrs and 30 min Seagate Recovery stopped responding, with Win7 reporting 7 levels of hang. Nothing was saved, no opportunity to save any intermediate stage either, so I had no option but to restart the software again (6 1/2 hrs as of now with 24 to go).

*** Note

Seagate recovery software is NOT user friendly, its very much read the manual to perform even basic operations. Win7 griped more than half the time I tried to run the software, saying it made adjustments to the runtime, or demanding I uninstall and reinstall it, which I did. I'll be reading tonight on the Seagate web site to see if there is more I need to learn to use it.

Recuva OTOH is nice and friendly, runs quickly, but functionally appears worthless for a large recovery. It puts all recovered files in the same location, discarding file structure, which can be ok for a few files, not so ok for 270k files. Reliability of the recovered file was close to zero confidence, but many seem ok, I'm not spending any more time with it.

More tomorrow when hopefully the deep scan completes, then I plan to buy one more WD Red and start in on recovery of one of the recent raid mirror drives (this first drive dropped out of array maybe 10 months ago, but may have the most recoverable files since it wasn't part of the raid chkdsk meltdown, so I am starting with it).

 

[Dstoop]

Wow man, that sucks It's also exactly why I never use RAID outside of business servers. That "Oh shit" moment always far outweighs the benefits when compared to just backing up to another hard drive in the middle of the night via scheduled task and a bat file.

 

[Anteaus]

Good luck. I'm sorry you got bit so hard but unfortunately that's the risk especially when the volumes get so large. I hope you can recovery most of it. The catch is that even with all of its redundancies, raid isn't failsafe and still requires an independent backup. You're taking it reasonably well. If my file server took and a dump and I didn't have a backup, I'm not sure how I would act. I have hundreds of hours worth of work invested for much of what's stored on it.

 

[It's Not Lupus]

Just replace the data on the RAID array with your backup. Simple.

 

[imagoon]

Wow man, that sucks It's also exactly why I never use RAID outside of business servers. That "Oh shit" moment always far outweighs the benefits when compared to just backing up to another hard drive in the middle of the night via scheduled task and a bat file.

Good idea considering that RAID is not a backup solution. It is a performance / capacity solution / resilience.

 

[mikeford]

What did not meet expectations for me was that I assumed two smart drives in a mirror would be fairly resilient, and I think they were, no SMART errors reported on any of the drives. A failure related to the RAID driver software never occurred to me, and IMHO is beyond lame on the parts of both AMD and MS. That was a big part of how it got worse, I didn't believe the failure was really happening.

*** Update

23 hrs 48 min into deep scan, 27,652,362 specific file documents found, AMD system monitor reports 5.285GB of memory in use, with one core maxed, one almost idle, other two around 10% each or less. Makes me think the first run hang may be memory related as I only had 4GB then.

7 hrs to go, so it will finish after I am asleep. I picked up another 2TB drive at Best buy ($50 off $100+ coupon), WD Green, and plan to restore next drive to it.

 

[VirtualLarry]

Clue me in on what this Seagate recovery software is. Is it free file-recovery software from Seagate? Have a link?

 

[Torn Mind]

RAID 1 prevents downtime. RAID 1 is not a backup because if an error occurs or you get a virus, both disks get the same treatment. The backup is frozen in time from when you finished making it. RAID 1 is updated in real time. Your file state is not safe from changes in RAID 1 while with a backup, it is, provided nothing does not destroy your backup media or corrupt your files on the backup media.

 

[mikeford]

Clue me in on what this Seagate recovery software is. Is it free file-recovery software from Seagate? Have a link?

\\http://www.seagate.com/services-software/data-recovery-services/consumers/file-recovery-software/

Free in demo mode to scan, maybe some other simple functions, but won't recover without a $99 key.

Seagate actually has a number of recovery options, the $99 DIY is the cheapest, next is a "assisted" DIY via phone/chat and remote software I think, then I believe two more levels of professional recovery services, which I think "start" at around $500.

OTOH some data is worth the expense of having a trained tech in a clean room open the drive and use special firmware etc. to extract and reassemble as much data as possible.

Right now I am several days time and a bit less than $500, with the expectation of maybe another $300 (replacement drives and drives to recover to or make images to). I don't know yet what I might try if the results from the Seagate software are less than desired. I "might" consider a service depending on what I can't get back.

The impression I get is that quickly it becomes an issue of technical expertise in fully exploiting the recovery software.

 

[Emulex]

raid is for nearline drives and the environment to support it. not crashing, five 9 uptime, battery controlled shutdown,temperature control. It is cheap and works great when you set it up like that.

hell SAS versions of SATA drives are only 10% more now (4tb SAS versus SATA) and there are huge gains in reliability and even performance. no reason to cheap out.

 

[mikeford]

raid is for nearline drives and the environment to support it. not crashing, five 9 uptime, battery controlled shutdown,temperature control. It is cheap and works great when you set it up like that.

hell SAS versions of SATA drives are only 10% more now (4tb SAS versus SATA) and there are huge gains in reliability and even performance. no reason to cheap out.

None of the drives failed with hardware or media issues AFAIK. All show no SMART errors. My best guess is that issues that have been around many years between the AMD RAID firmware and windows drivers are the heart of the problem. Both drives in the current array pass all diagnostics.

For now I am DONE with Raid, especially AMD onboard. Critical files are going to DVD, and rest a rotating offline backup. OS and apps now live on a Intel 330 120GB SSD.

I had one drive drop from the array a year ago with no error message or change in SMART health, put in a new drive and array rebuilt. The dropped drive won't pass Seatools long test so its going back on RMA (I will send it back as soon as I get what data I can recovered from it later this week.). The only reason I am running recovery software on it is that I quick formatted the drive in preparation to the pending RMA return, and RAID failed before I got it in the mail during the holidays. The data is old, but high recovery rate of good data is expected, then I will working on the array drives one at a time, and keep best data from the three sets.

 

[mikeford]

*** Update on the POS Seagate recovery software.

When the 30hr deep scan completed Tuesday morning, I checked it as soon as I got up and AMD system monitor was showing about 6GB of my newly installed 8GB in use. I clicked the next button and selected what looked like the correct partition for my old drive, and began selecting folders to be recovered within the partition. Memory use peaked at 96% of the 8GB during analysis of the deep scan and showing partitions, showed 94% at completion.

At the page where the destination and some options are selected, I had a question about one of the option settings and pressed the help button, and as MS put it, Seagate Recovery software quit unexpectedly.

Never an option to save any intermediary step, and it looks again like it bumped into the limit of physical memory and croaked. What a POS. I think its a bunch of code from the DOS era running in some kind of shell environment, but wow, its the poorest excuse for a commercial software product I've ever seen. I still think it may be powerful stuff, just not suitable for non technical consumer use.

I bought 2x 4GB of the same memory I put in the other day, no real option to trying it again with 16GB of ram. I am doing some online research, then starting the 30 hr deep scan before heading off to bed, expecting it to complete by Thursday morning.

*** The good news, before it croaked everything was pointing at high chance of recovering most of the data on this drive of interest to me.

 

[mikeford]

I've been searching more, still no good how to info. Clearly I don't know how to use this software well. I hate to spend money, but I think the smart move might have been to jump directly to the assisted recovery with a experienced tech answering questions and guiding through the process, but maybe that is why the software is so difficult?

I have stumbled on a couple things. Loading and Saving files seems to be setup through right clicks on data windows, so if I had known, I could have saved my 30 hr deep scan. I decided to try recovering from the partition instead of the logical drive since this drive should have much of the directory intact, just marked as cleared in the quick format. This brought up a window with the file structure in a few minutes, and is saying 8 hrs to recover the 481,000 files of 1.8TB data I want the most. More in the morning as they say.

 

[mikeford]

I have hundreds of hours worth of work invested for much of what's stored on it.

Two points to consider;

How damaged is the drive and the data you want?

What is it worth to recover it, and is time a factor or can you pick at it for weeks?

My first action was to attempt recovery at no cost and low effort.

Revised plan, try to manage costs, but ZERO skimping, which meant buying several WD Red high quality drives, more ram, SSD for the OS, and paying for recovery software. If needed I will replace the motherboard etc, whatever up to about $1k, then it starts being stupid not to directly go to professionals.

Bare minimum, plenty of ram, known good drives large enough to write image files to, with a second to write the actual recovered files to, and a clean OS drive for application and misc files. Image the bad drive and get it out of the system and offline.

If you think the drive "MAY" have physical damage consider immediately consulting with a professional recovery service as any action you take might further degrade the data on the drive, where a recovery service can open the drive in a clean room, physically clean the surface of the discs and/or transfer them into a mechanically safer drive. They can read tracks off center and combine multiple read attempts to improve accuracy. They can clean the disc between reads if the surface is damage and in turn is damaging read heads, they can can also replace heads as needed. Sound expensive, it is, but a good place will quote an estimate, and charge based on actual recovery.

 

[mikeford]

Seagate called me today to see how the recovery was going, and got an earful back from me. Assisted recovery appears to start at $199, I think I can skip that now, but its what I would advise most users to do if the data isn't REALLY easy to recover. Assisted recovery not only gives access to somebody that hopefully is technically knowledgeable, but also they have better software they can run remotely on your system. OTOH pretty sure that price is per drive, and I have at least two, plus now I own the software to use anytime in the future.

*** I think I have recovered everything I want off this first drive (which dropped from the array several months ago, missing last 4 or so months of changes). Almost 500k files and 1.69 TB of data. Later tonight I will start work on one of the fresh array drives, then a MASSIVE job ahead of merging best versions and checking for errors, followed by recreating the HTPC arrangement.

Thats all for me on this one unless something dramatic happens later.

 

[cantholdanymore]

The only reason I am running recovery software on it is that I quick formatted the drive in preparation to the pending RMA return, and RAID failed before I got it in the mail during the holidays.

That's some bad karma.

 

[mikeford]

That's some bad karma.

If I had shipped it out it looks like I would be SOL, testing the first drive from the active until 1/1/13 array is showing NO DATA. Quick scan found a single NTFS partition of 1.82 TB with 4 GB of garbage system files, deep scan found the same only two region blocks with anything, the rest of the drive it says is unused.

I'm running the full set of Seatools diagnostics right now, but so far first drive has no errors.

How does a mirror drive in an array in use for 2.5 years with all but 25GB of the 1.82 TB filled with data just poof? Wouldn't that require every block to be erased, or am I missing something dumb about taking the drives out of the RAID1 (mirror)?

 

[mikeford]

*** Totally bizarre.

Both of the drives that were up and running, booting Win7 64 and 98% full of 1.82TB of data, then showing errors and refusing to boot, appear to have been hard formatted, ZERO data I have been able to recover from either one. Not one sector is showing any files except those related to a format. I am 99% certain I never clicked an OK format my drive message. CHKDSK ran a few times and I tried to repair with Win7 install disk, then pulled both drives physically out of the system to work on the drive I was about to RMA.

I'm running a deep scan on the second drive now, but its looking exactly like the first one, a single NTFS partition with 7 text files related to formatting, and 717 other files with odd character names that contain nothing I recognize.

I wonder if this could have been some kind of virus or something, system was running Avast and kept updated, but the first thing I "noticed" when the system beeped and rebooted, started acting flaky, is that Avast would not run.

 

[murphyc]

*** Great mistake ***
*** What I did do.

But no mention of what you didn't do, which apparently was do backups. That was your exit strategy to completely blow away everything, and start from scratch.

All show no SMART errors.

If the entire list of over a dozen attributes show zero indication of what's wrong, I'd be surprised. I'm not surprised if the single pass/fail health status would continue to say "pass" while all of this was happening, however.

Critical files are going to DVD

Files that are completely, utterly worthless to you, go on DVD. If you're ready for another data loss experience, depend on DVD.

 

[beginner99]

Files that are completely, utterly worthless to you, go on DVD. If you're ready for another data loss experience, depend on DVD.

Yeah the best place for critical documents in some online storage. So you sure to not lose the in case of fire or any similar catastrophe.

Besides that this thread outlines why I never bothered with RAID and believe it is pretty much useless for home-use.

maybe RAID 0 using platter drives before SSD were available was an option if you did regular OS backup but now I don't see how it is useful.

 

[nk215]

Raid 5 saved my behind once and it's more than enough to pay for itself already. The key is backup. My policy is that I don't trust HDs that are older than 3-4 years old.

 

[murphyc]

The problem with dye based optical media is the time before data loss is a huge range, and there's no practical way for a consumer to learn what that number is; most typical though is it's less than that of HDDs.

The issue with home user storage needs is the incompatible need for pricing not much higher than the cost of consumer grade bare storage, but at a capacity that's at or near enterprise levels. Home user photo albums are better off recorded to film, and tossed into the freezer.

 

[Emulex]

Yeah the best place for critical documents in some online storage. So you sure to not lose the in case of fire or any similar catastrophe.

Besides that this thread outlines why I never bothered with RAID and believe it is pretty much useless for home-use.

maybe RAID 0 using platter drives before SSD were available was an option if you did regular OS backup but now I don't see how it is useful.

^^ This. Raid is another responsibility. It is fragile.

 

[tweakboy]

Toes Up ,,, you cant post sexual stuff on this forum.

 

[Zxian]

Besides that this thread outlines why I never bothered with RAID and believe it is pretty much useless for home-use.

This is why you never bother with on-board RAID solutions. Either you go for an enterprise level hardware controller or you do it in software. I've had my LSI 9650SE-8LPML for over 4 years now. I've thankfully never had to RMA any of my drives, but I've also never had to wait to get access to my files.

Important documents can go on the cloud (and for me, they are), but when you've got over 200GB of irreplacable pictures and music, cloud-based backup solutions become less appealing with the limited upload bandwidth we've got in North America.