Looking for ideas on Data storage

[JediJeb]

My company produces over a Gig of data per week from analytical instruments in an Environmental Testing Lab. This is not what goes into our database, that is a whole nother situation, this is raw data, which is not very compressible. We have to store it for 5 years and have quick access to it if an auditor shows up or a client needs us to reproduce our work somewhere down the line.

Right now I use CD-R, burning at least 2 per week one for onsite and one for offsite. This is just to backup what is being stored on hard drives scattered all over the lab. I am lobbying for a data server to centralize where that data goes, right now each piece of equipment stores its own data and I pull it down over the network and put it on the CD-Rs which takes forever. I am hoping to at least push through getting a SATA RAID card for RAID1 on some 200 or 300GB drives just for a little security and putting them in the computer I use to burn the CD-Rs. I hope to soon go to DVD-R so as not to use so many disks.

Any ideas about what else I can do to help protect our data that won't cost an arm and a leg. I need all this data live and backed up.

Another thing we want to do is to be able to scan all of our paperwork and store it as .PDF files which involves setting up a document server. Any ideas about this also would be appreciated.

Thanks in advance.

 

[SickBeast]

Sounds like you need to build a server. The more IDE/SATA headers you can find on a new motherboard, the better.

What is the total size of the data you will need to back up?

You will also need a 10/100 switch with enough outlets for the computers in your company, along with a hardware firewall if you want to give internet access to all the computers.

From what you said in my thread, I would propose an ATA RAID-1 solution and back up to a USB hard drive. Go to DVD-R on a set schedule. Again this all depends on how much space you'll need. If you can get a motherboard with tons of headers you'll be in good shape. You'll want excellent case cooling. Tape backup is slow and expensive, but is yet another option.

You need a viable off-site solution for your data to be truly safe. From what people have told me I'm not sure an external USB drive classifies as that. If you're extremely careful with it, your risk will be minimized.

Good luck, you seem to share my pain.

 

[JediJeb]

We already have the internet connectivity firewall ect setup with the building network. The office part of the lab has the SQL server for the client data and such, that is backed up with DAT tapes but they only hold maybe 20GB and are run in a rotation. That is nice when only backing up 20gb of database, but our raw data would fill that in a few weeks and would then have to be replaced with another tape. The total data for the 5 year holding time would easily be 250Gb, and with each upgrade of the instrument software the file sizes get larger, I believe in a few years we could be looking at 500 or 600 GB total data which will be a nightmare to backup at least on our budget.

Also does anyone know what the expected life of data on tape is when just left on the tape and not overwritten. I know when we used to use reel to reel it was advised that at least once a year we copy if off and back on to refresh the tape. ( yea we used to use reel to reel when the instruments were connected to a HP 1000 mini computer, oh how glad I am those days are over) Could we store data reliably for 5 years on a tape?

 

[JediJeb]

Looks like everyone is still discussing this in your post SickBeast lol.

 

[SickBeast]

Originally posted by: JediJeb
Looks like everyone is still discussing this in your post SickBeast lol.

Yeah thanks for sparing me a highjacking.

How many users will be on your network? You may need to go SCSI depending on the load you'll be placing on your hard drives.

My advice would be to archive old and rarely needed stuff and focus your backup efforts on a smaller more active data set.

As for a stable long term (5 year+) archival backup solution...I'm not sure one exists. CDRs, DVD-Rs, tapes, and hard drives all decay over time (although a good CDR should last you more than 5 years). Properly stored you could probably get 10 years out of most mediums, but beyond that is anyone's guess really. Hopefully someone will be able to definitively answer this.

 

[funggorgor]

if you want to be more secure , just backup the data in both harddisk and cd-r

 

[JediJeb]

Originally posted by: SickBeast

Originally posted by: JediJeb
Looks like everyone is still discussing this in your post SickBeast lol.

Yeah thanks for sparing me a highjacking.

How many users will be on your network? You may need to go SCSI depending on the load you'll be placing on your hard drives.

My advice would be to archive old and rarely needed stuff and focus your backup efforts on a smaller more active data set.

As for a stable long term (5 year+) archival backup solution...I'm not sure one exists. CDRs, DVD-Rs, tapes, and hard drives all decay over time (although a good CDR should last you more than 5 years). Properly stored you could probably get 10 years out of most mediums, but beyond that is anyone's guess really. Hopefully someone will be able to definitively answer this.

In terms of users only a few, but we all run several pieces of equipment which aquire data constantly. Right now the plan is to send that data to the server at then end of a run, usually about once a day, so there shouldn't be a constant stream of traffic. I'm slowly educating the new IT guy on the differences between what we do and what the office does, they have all the normal database, mail, web, file servers you need in an office but ours is just archival of large files. After a couple days the data is probably not going to be accessed ever again but we have to keep it. Speed really isn't a factor because some of the equipment still runs on 10Mb networking.( actually just got rid of the last bit of coax networking when we moved to our new building this summer ) Now all the wireing is new and fast CAT6

 

[Gigantopithecus]

If you're generating a bit more than a gig a week, I would assume you probably generate about 60gbs a year. A month's worth of data will probably fit on one DVDr.
I would simply set up a cheap but solid data server. Make the hard drive as big as you need the data to be kept live - if you want the past year live, get an 60gb drive. Two years? Go 120gb. And so on. Back up your data however often you want onto an external USB hard drive.
Depending on how much paperwork you generate, you might need a high-volume scanner and lots of hard drive space, but this could be set up on a completely separate machine.
Good luck!

Edit: Didn't notice you said, "After a couple days the data is probably not going to be accessed ever again but we have to keep it."

In that case, a server with a smaller hdd will be fine. As will be a smaller external hdd. You can just buy a new external hdd every year and write it off instead of buying a lot now and not being able to write it all off.

 

[DanDaMan315]

get an 8 sata mobo and hook 16 300gb hard drives to it. that should do the trick.

 

[DanDaMan315]

although it would be unnecessary.

 

[Dough1397]

and friggin expensive.....

 

[crimson117]

and succeptible to a single point of failure (disaster like fire or flood takes out the server).

I think a redundant raid with 2x 250mb SATA drives, combined with monthly or bi-weekly DVD-R backups would do the trick.

Once a year, for spring cleaning, you should copy the server's hard disk to a 250gb external drive.

You can easily upgrade the drives as you need rises - however at ~60gb per year, you'd have to quadruple you storage needs before 250gb wouldn't be enough.

 

[JediJeb]

These are the types of ideas I am thinking of, probably about 300GB in Raid 1, Want to go DVD instead of CD for backup like we have now. Just have to convince the powers that be now, but after today I just might be able to since we lost a computer, luckily I was able to salvage the data on the drive but now I have to find a new computer with an ISA slot . LOL they always want to buy Dell, but doubt they will find one with that . Maybe I will finally get my chance to build what I want. Spent all day trying to load NT4 on an old PII to get the instruments back online. The software just refuses to work on XP for some reason, and the newer versions of the control software won't run the old hardware.

 

[jose]

Checkout the 3ware 12 port sata raid card, very nice to pair w/ 250g wd raid ed. drives. Add this to a server mobo of your choice w/ 64bit pci slots.
I did a system a while back which ran about $3K for the server.

Regards,
Jose