Need large, cheap data archival (min. 8TB)

Ken g6

Programming Moderator, Elite Member
Moderator
Dec 11, 1999
16,632
4,562
75
I record way too many TV shows, and I never get around to either editing out commercials and compressing them or deleting them. So I wind up with lots of archived data filling my drives, that I do want to access, but I don't need to access quickly. A Seagate SMR drive would seem ideal, except that I want the data stored reasonably reliably, and I see a lot of failure reports with this drive. I've also had too many bad experiences with external HDDs to trust them.

So, experts, what do you suggest? Are tape drives in a similar price range? Should I just get a BD-RW drive and a bunch of disks? Or is the above Seagate more reliable than it looks?

Edit: Oh, I suppose I could do online storage, but my Internet's fairly slow.
 

aigomorla

CPU, Cases&Cooling Mod PC Gaming Mod Elite Member
Super Moderator
Sep 28, 2005
21,044
3,524
126
Raid Z using refurbished HGST's on FreeNAS or any free NAS Software on old hardware.

Just boot the system when you need to do your backups, and turn it off when your done.

66 dollar per drive:
https://www.amazon.com/HGST-Ultrast...=UTF8&qid=1497033879&sr=8-2&keywords=3TB+HGST

You would need at least 5 for Raid-Z. (well you can do Raid-Z with 3, but u will not meet your 8TB quota, and 4 drives is limited to Raid-Z2.)

330 dollars in drives... with 1 redundancy for failure for approx 12TB of storage.

Get old Hardware your not using for FreeNAS.
You will probably need about 4-8GB of Ram as well....

I will never trust Seagate again... ive lost 5 "NAS" drives from them. each of them dying 1-2 months after each other upon expiration of warrenty.

Im a full supporter of HGST and WD now...
Even then i will always recommend redundancy, and backups with redundancy.
 
  • Like
Reactions: harobikes333

Elixer

Lifer
May 7, 2002
10,371
762
126
I haven't seen a cheap consumer tape drive unit, so, that is out.
BD-RW have come down in price, but, those have there own issues, and the storage per disc is still too low, so, I pass on that as well.

So, we are back to HDs. You could go to target and find clearanced drives, and shuck the case, or, I suppose https://www.amazon.com/Seagate-Expansion-Desktop-External-STEB8000100/dp/B01HAPGEIE/ which seems to be close to the lowest price it has been https://camelcamelcamel.com/Seagate-Expansion-Desktop-External-STEB8000100/product/B01HAPGEIE but, it is a Seagate and I really, really don't like their warranty BS they are pulling these days, not to mention the overall product reliability experience I had in the past with them. I haven't yet used this specific 8TB line though, so, jury is still out (http://www.storagereview.com/seagate_archive_hdd_review_8tb). Oh, and these are supposed to be the same internal drive that you were looking at before (SMR tech).

I have been wanting to get the HGST Helium drives, but, those are more expensive, $100 more.
Anyway, to make this rambling of mine stop, I would look for a HGST drive if I was gonna buy a HD today as reliability as the #1 concern, though, keep in mind that to really have true reliability, you need more than 1 unit, and backup to that as well..
 

Shmee

Memory & Storage, Graphics Cards Mod Elite Member
Super Moderator
Sep 13, 2008
8,136
3,074
146
I like aigo's idea, but would go with WD reds. Not as cheap, but then I would not want to cheap out on this.
 

seagate_surfer

Junior Member
Mar 31, 2017
21
7
51
Hi Ken,

we would like to add one important thing that hasn't been mentioned yet:
the choice of your backup devices should be made in regards to the scenario you would like to use them in. It is a bit unclear if you are considering a NAS device for your backups or not...

Independent from picking a certain manufacturer, in a 24/7 environment you should use a NAS HDD ("IronWolf" in the Seagate world) whereas in a regular PC/Backup scenario a desktop drive ("BarraCuda" in the Seagate world) is sufficient.

Hope that helps...
 

Ken g6

Programming Moderator, Elite Member
Moderator
Dec 11, 1999
16,632
4,562
75
Yeah, I think this thread kinda went off the rails. I have a 5TB fast drive and a 4TB green drive in my computer currently. The goal was to transfer data from the 4TB to an 8TB drive, and have more space. I wasn't thinking about RAID or backups. The video I record often isn't worth backing up - either I compress it before backing it up, or it'll be rebroadcast. A lot of this is probably stuff I'll never watch but can't quite throw away either.
 

tcsenter

Lifer
Sep 7, 2001
18,893
547
126
Yep, find Grade A off-lease/used/refurbed 2nd or 3rd Gen Core SFF or slim tower PC with Gigabit Ethernet, USB3.0, that will house two 3.5" drives (even if you have to buy one of those adapters to mount in a 5.25" bay) for ~$150 delivered (+/- $20). Use whatever OS it comes with (W7, 8.1, 10), or install open source alternative, create RAID1 mirror using two HDD. Way more CPU/RAM than you'd get from spending $250+ on a diskless NAS box or RAID enclosure.
 

VirtualLarry

No Lifer
Aug 25, 2001
56,572
10,208
126
Yes, but unless he wants his "server" to actually do trans-coding, then he would most likely be better off with a NAS. Less power, less heat, less noise, less hassle, and physically smaller.

Not necessarily cheaper, but you can get decent 4-bay QNAP NAS units (TS-431) for under $300 on sale or on ebay.

Sure, performance is likely not going to be better than a dedicated PC, but just for file-storage, that's kind of overkill, unless you're building a "real" server (unRAID, etc.), and filling a tower case with drives. The bottleneck is likely to be the network anyways.
 

Charlie98

Diamond Member
Nov 6, 2011
6,298
64
91
Yeah, I think this thread kinda went off the rails. I have a 5TB fast drive and a 4TB green drive in my computer currently. The goal was to transfer data from the 4TB to an 8TB drive, and have more space. I wasn't thinking about RAID or backups. The video I record often isn't worth backing up - either I compress it before backing it up, or it'll be rebroadcast. A lot of this is probably stuff I'll never watch but can't quite throw away either.

I don't think any sort of NAS, RAID, etc is required... what you want is quite simple. Why don't you just add another 4-5TB drive to the 4TB one (assuming you have the room?) Or else it's just spend your money and take your chances with either a less expensive drive, or pony up the money for the HGST.

I will agree with a backup of some sort, however. I've got about 4.5TB of video stored on my HTPC's main 5TB drive, and while it wouldn't be earth shattering if it failed, it would cost me more in my time to re-rip and reload a new drive... so I have what were the old main drives (2x 3TB drives) as manual backup. You have to decide whether or not you are willing to lose that data... no matter which single drive you would purchase... without a backup.
 
  • Like
Reactions: Ken g6

Malogeek

Golden Member
Mar 5, 2017
1,390
778
136
yaktribe.org
IMO you should focus more on offline nightly processing of these recorded shows. Most likely they're being recorded in ancient MPEG2 which will give you very large files unnecessarily. Even if they're h264 you're still wasting space. You should think about setting up something like Handbrake scripts to run at night and compress everything to h265. Goodbye capacity problems.
 

sdifox

No Lifer
Sep 30, 2005
99,386
17,555
126
get two external hdds, duplicate copy of the content and call it a day.

or just torrent whatever show you want to watch instead of record and convert. internet is your backup
 

WhoBeDaPlaya

Diamond Member
Sep 15, 2000
7,414
402
126
Get a WD MyBook / EasyStore 8TB for ~$180 at BestBuy.
Drive inside is a 5400rpm version of the HGST Ultrastar He8, so it's pretty much bulletproof from a reliability standpoint.
Bonus points for being PMR rather than sh*** SMR.

Dude, 6-8 TB HD and an enclosure. Simple. People and their NAS and Raids. pishaw,
When I moved from IA to TX, I FedEx-ed my HTPC sans HDDs, and drove the HDDs down.
What's that old saying again? Never underestimate the bandwidth of a car filled with HDDs :p
(latency is a killer though, and that is one JUMBO frame!)
 
Last edited:
  • Like
Reactions: Yakk

Ken g6

Programming Moderator, Elite Member
Moderator
Dec 11, 1999
16,632
4,562
75
IMO you should focus more on offline nightly processing of these recorded shows. Most likely they're being recorded in ancient MPEG2 which will give you very large files unnecessarily. Even if they're h264 you're still wasting space. You should think about setting up something like Handbrake scripts to run at night and compress everything to h265. Goodbye capacity problems.
That potentially sounds like a very good plan. The main problem with it is commercials. I have a couple of programs to edit MPEG2 losslessly to remove commercials. This is off-topic, but got anything like that for h264 or h265?

Oh, and h265 isn't a great option because almost nothing I have will play it back.
 

Malogeek

Golden Member
Mar 5, 2017
1,390
778
136
yaktribe.org
That potentially sounds like a very good plan. The main problem with it is commercials. I have a couple of programs to edit MPEG2 losslessly to remove commercials. This is off-topic, but got anything like that for h264 or h265?
Why not just process the files after they've been edited for commercials then? Use Handbrake CLI to convert from MPEG2 to h264 and you'll have huge benefits for file size.

Oh, and h265 isn't a great option because almost nothing I have will play it back.
Fair enough. Even h264 will be of great benefit for you though. I don't know what you're using for playback so can't comment regarding h265.
 

Ken g6

Programming Moderator, Elite Member
Moderator
Dec 11, 1999
16,632
4,562
75
Why not just process the files after they've been edited for commercials then?
I do. But I don't edit commercials out frequently enough, so files build up, and I'm running out of space.
 

sdifox

No Lifer
Sep 30, 2005
99,386
17,555
126
I do. But I don't edit commercials out frequently enough, so files build up, and I'm running out of space.
Any particular reason you cannot torrent the shows already compressed and sans commercials?
 

Ken g6

Programming Moderator, Elite Member
Moderator
Dec 11, 1999
16,632
4,562
75
Any particular reason you cannot torrent the shows already compressed and sans commercials?
For starters, it's illegal. I could report you for just suggesting it. :triumph:

This is what I use for cutting commercials from my dish network captures.. Everything gets saved to h264 mkv.
Requires Windows. Did I mention I run Linux? I could use it in my Windows VM...but this is getting complicated.
Why don't you just add another 4-5TB drive to the 4TB one (assuming you have the room?)
This is probably the most practical answer.

Get a WD MyBook / EasyStore 8TB for ~$180 at BestBuy.
Another potentially good answer. Especially for backups. Although a drive that requires helium makes me nervous that it will lose it at some point. Not, I suppose, within its practical lifetime, though.
 

WhoBeDaPlaya

Diamond Member
Sep 15, 2000
7,414
402
126
Another potentially good answer. Especially for backups. Although a drive that requires helium makes me nervous that it will lose it at some point. Not, I suppose, within its practical lifetime, though.
I was apprehensive at first for the same reason, but with other options being SMR, it was Hobson's choice for me.
*knock on wood* My 32x He8 drives have been purring along just fine 24/7 for a good, long while now.
 
Last edited:
  • Like
Reactions: Ken g6

Malogeek

Golden Member
Mar 5, 2017
1,390
778
136
yaktribe.org
I do. But I don't edit commercials out frequently enough, so files build up, and I'm running out of space.
Irrespective of whether they've been edited or not, you can still process them automatically via CLI in linux on a nightly basis to at least get them in h264 format.
 

sdifox

No Lifer
Sep 30, 2005
99,386
17,555
126
For starters, it's illegal. I could report you for just suggesting it. :triumph:


Requires Windows. Did I mention I run Linux? I could use it in my Windows VM...but this is getting complicated.

This is probably the most practical answer.


Another potentially good answer. Especially for backups. Although a drive that requires helium makes me nervous that it will lose it at some point. Not, I suppose, within its practical lifetime, though.


How is any more illegal when you are just skipping the editing and encoding step?