Question Help with a media storage strategy

[TWGF1572]

It's been a long while since I've posted, but I need some help figuring out a strategy for picture/video storage and was hoping I could get some good ideas here.

Current System - 5 years old, but working fine for what I need it to do. For storage, it's set up with -

1. C Drive: A 2TB Samsung Evo 860 SSD for OS/applications. It's currently around 400GB, but my son has just discovered how fun it is to play games on the computer vs. the XBox, so I expect that to shrink.
2. D Drive: Two 1TB WD Black Drives currently used for document/picture/video storage. They are currently set up as RAID 1 using Windows 10, because I thought RAID1 was a good idea at the time. I think there is about 850GB on them right now.
3. Windows file history backs up to an external USB drive connected to the computer. This drive is 10+ years old, and probably should be replaced just because.
4. I have a Carbonite subscription to backup the D drive. I think there are cheaper and better options out there, I've been using it forever and apathy/inertia has set in on investigating other options.

My Goal: The files on the D drive are priceless. Clichéd I know, but its about 15 years of memories including my wedding, kids being born, first steps, games, etc. Money matters, but I'm not looking for a budget build. I'm trying to follow the 3-2-1 strategy, etc.

I was ready to make a plunge and buy a NAS over BF. Well I actually did, the Synology DS920+. I'd planned on doing RAID1 or 5 on that and using it as the primary storage for these files. Then I started reading about limitations of NAS over WIFI and decided to cancel it. I don't have wired ethernet. I have a Google Nest router, which I could plug the NAS into. But the computer will have to stay on WIFI in the short term. My house has Cat5 cable for phone lines, so I'm seriously thinking about changing it over to ethernet, but that's not a given. I would not be surprised if my house is wired in one long daisy chain.

During BF, I also purchased two 10TB Red Plus drives when NewEgg had them for $190. They are due to arrive today. I can either return, or incorporate them into the system.

So the ideas I'm kicking around are as follows - I'm curious what thoughts people have...

1. Stop using RAID1 on on my D drive, which would free up another 1TB of space. I would still be 3-2-1 compliant, I think. I'm also assuming it's possible to un-RAID in Windows. I haven't checked. Return the 10TB HDs to NewEgg
2. Swap out the Blacks for the Reds in my system, and leave everything the same. I'd have storage space for another 10 years.
   1. As part of #2 I could buy a NAS, put the old Blacks in it and use it for my file history drive. The USB drive I'm currently using could be shelved.
3. Buy the NAS, throw the Reds in and see how bad the performance actually is.

Lots of moving parts here, I'm just struggling to get some clarity so was hoping someone who isn't so close to the issue can chime in. And feel free to tell me I'm crazy and come up with a new recommendation. Thank you!

 

[damian101]

The problem with accessing a NAS over Wi-Fi is that the normally used SMB protocol has extremely bad performance over high-latency connections.

 

[TWGF1572]

So the best plan then is to probably shelve any thought of trying the NAS until/if I can get my house wiring setup properly.

 

[damian101]

@TWGF1572 Maybe. With a protocol like WebDAV, SFTP or FTP bandwidth for sequential transfers isn't affected by latency anymore. NFS should also perform mich better than SMB over high-latency connections, but mounting NFS shares under Windows is a bit complicated afaik.

 

[damian101]

I'm actually just in the process of building my own file server. I think I'm going to use some software like NextCloud, Seafile or ownCloud for that. Already set up a Btrfs RAID 6 Btrfs. Btrfs RAID5/6 is only safe if you choose RAID 1 for the metadata. For the operating system I went with openSUSE.

 

[TWGF1572]

Doing that may be a bit out of my comfort zone. I do see where you can plug a NAS into the ethernet port of a computer, and with some provisioning of IP addresses you can get the two talking. That may be an option.

At this point, I've decided to buy a new external drive. That will be needed regardless of what way I go now or in the future. What to do with the 10TB Red Plus drives remains the question. The price was definitely right on them.

 

[damian101]

Doing that may be a bit out of my comfort zone. I do see where you can plug a NAS into the ethernet port of a computer, and with some provisioning of IP addresses you can get the two talking. That may be an option.

At this point, I've decided to buy a new external drive. That will be needed regardless of what way I go now or in the future. What to do with the 10TB Red Plus drives remains the question. The price was definitely right on them.

You could put your 10TB hard drives in something like this: https://www.aliexpress.com/item/4000183772408.html
Btw, it's not very complicated to create an SMB share in Windows. So you don't need a NAS to try out SMB performance over your Wi-Fi. You could even use some old Windows machine as a NAS, just automatic restarts to install updates are a bit annoying, but there are ways to disable that.

 

[TWGF1572]

That enclosure may be just the ticket for what I need. Thanks for your help on this!

 

[MalVeauX]

Goal: The files on the D drive are priceless. Clichéd I know, but its about 15 years of memories including my wedding, kids being born, first steps, games, etc. Money matters, but I'm not looking for a budget build. I'm trying to follow the 3-2-1 strategy, etc.

So the ideas I'm kicking around are as follows - I'm curious what thoughts people have...

1. Stop using RAID1 on on my D drive, which would free up another 1TB of space. I would still be 3-2-1 compliant, I think. I'm also assuming it's possible to un-RAID in Windows. I haven't checked. Return the 10TB HDs to NewEgg
2. Swap out the Blacks for the Reds in my system, and leave everything the same. I'd have storage space for another 10 years.
   1. As part of #2 I could buy a NAS, put the old Blacks in it and use it for my file history drive. The USB drive I'm currently using could be shelved.
3. Buy the NAS, throw the Reds in and see how bad the performance actually is.

Heya,

So, getting onto the 3-2-1 means three physical copies, separated. Let's focus on that and connectivity.

Your primary daily use machine, the PC, needs to offload all of this storage. This is the working environment. One set of living data, but it's the most volatile. So trim your PC to be what it needs to be to run. You don't need RAID1 in your working environment system. Also, near max capacity SSD perform worse than less capacity. So you want to move data out of there that isn't needed to store there. Living data will be in this working environment, and so that's a physical copy. Have your imporatnt data set to backup to one of your backup solutions at a frequency of your choice (to your NAS, or an external drive). So for this, I would use an external hard drive or SSD (USB3Gen2) to do your common backups to, powered separately from your primary working environment (not all your data, just the stuff you need for up time or that is important to you). There's 1 (of your 3).

Use your new 10TB drives in a NAS in Mirror (avoiding saying RAID1, implying using RAID, there's no real backup using RAID, so don't use it). The key to a real backup solution is being able to actually recover it. When you use RAID and RAID controllers, this gets complicated if something fails, not just the drive. So I suggest avoiding RAID. Instead, use basic mirroring (same thing, minus the controller, etc). Hardware RAID is dead, a thing of the past, for this application. File systems are simply superior now. You can build a ZFS file system and run mirroring and you have yourself a very hearty NAS. This serves as 1 of your 3 physical copies. The two 10TB drives give you 10TB of redundant 1:1 mirroring. You can access it however you wish, over network. Recovery is easy. You can access the individual drives without a controller or RAID array. Rebuilding is easy. No parity to fool with. I highly suggest you avoid the idea of parity on important data. Parity really is for a working environment that needs capacity, not for storage and not for redundancy. Rebuilding a parity RAID array takes forever, and its dangerous and doesn't always work. If you have precious irreplaceable data, do not use RAID5 or other kinds of implmentations if you want to recover the data easily and swiftly. Mirroring is ideal now. 1:1 redundancy. Super easy access. We have massive capacity drives now so we do not need "redundant array of inexpensive drives." This system gives you that extra physical copy, that is 1:1 redundant on top of that, and has a file system that can heal and repair and is very hearty. Don't bother with a pre-build NAS solution, they're junk. Just take any PC you want and install FreeNAS (latest version) on there. That's ZFS, mirroring and easy to use interface (GUI) over network, wireless or wired. Super easy to install and use (lots of tutorials, active community forum). This is 2 (of your 3) and is the most robust of the choices as it has the most expansion and most redundancy and lifespan with monitoring.

DVD-R or BD-R (including M-Disc) is one of the three strategies you can use. All hard drives fail. SSD has too low of capacity. Flash/ND is not long term. You can have lots of them with physical copies spread out, and all of them will fail before a single DVD-R or BD-R write-once disc fails from age. These things last 20+ years without anything special. So you burn your priceless data on two separate discs and store them in separate fireproof/waterproof cases. One on location. One off location. There's your 3rd physical copy. And it's far more protected and hearty than any hard drive solution even with redundancy. Optical is still one of the best and most accessible and easy to recover mediums to use for storage. I highly recommend the M-disc variety 25Gb BD-R write-once discs and a basic BD-R drive. $65 for the drive. The discs are peanuts cheap. They will outlive you. Or just use basic standard every day 25GB BD-R discs. Optical isn't going away. It's alive and well. This is where your priceless stuff goes as a 3rd copy. Update and add discs over time. Keep some at another physical location in a fireproof/waterproof case. No gear needed. Super small and easy to store. This is 3 (of your 3). And this is the longevity saving grace of your setup so that if everything fails, these out live everything.

You can then splice in some cloud storage of anything small and that's 4 (out of 3).

Now, access. Networking is how most of it is done these days. You can easily go wired (recommended) or you can go wireless (less recommended, but doable). If you want to go wireless, get a new wireless router and you can have it operate with your existing router, etc. The wifi needs to be at least 802.11AC operating on 5Ghz and get very good thermally regulated wifi NICs that are also 802.11AC on 5Ghz for your machine(s). It's quite fast for wireless and affordable. Otherwise, basic wired gigabit LAN will do the job. You can easily drop wires in walls, or have them in corners and along base boards, etc. You don't need an overhaul. You can have your existing router feed internet to your wifi router and the wifi router connects all the machines and handles traffic.

Very best,

 

[TWGF1572]

This is excellent! Thank you so much! I'm going to save this offline for reference because it's well written, very clear, and makes it simple. That was exactly what I needed to get me heading in the right direction. Its very easy to get lost in the forest for the trees, there are so many options. Thanks again.

Heya,

So, getting onto the 3-2-1 means three physical copies, separated. Let's focus on that and connectivity.

Your primary daily use machine, the PC, needs to offload all of this storage. This is the working environment. One set of living data, but it's the most volatile. So trim your PC to be what it needs to be to run. You don't need RAID1 in your working environment system. Also, near max capacity SSD perform worse than less capacity. So you want to move data out of there that isn't needed to store there. Living data will be in this working environment, and so that's a physical copy. Have your imporatnt data set to backup to one of your backup solutions at a frequency of your choice (to your NAS, or an external drive). So for this, I would use an external hard drive or SSD (USB3Gen2) to do your common backups to, powered separately from your primary working environment (not all your data, just the stuff you need for up time or that is important to you). There's 1 (of your 3).

 

[KentState]

Snip....

I agree with 99.99% of what you suggested. However, a mirrored copy in ZFS should not be considered as any type of backup. If the data changes on one disk, it instantly changes on the other. Unless you setup snapshots, it's not possible to get the original version back without going to the other backups. That means going to DVD-R, BD-R or tape to restore a lost or corrupted file. In reality, running in a mirror in ZFS just covers the hot backup from lost downtime if a drive fails. So the copies are 1) mirrored set 2) physical media 3) cloud.

For me personally, I have a dedicated ZFS server (4 vdevs in raid-z2) for the hot data, a secondary one (4 x mirrored vdevs) for snapshot replication, dual HDD external copies done from another computer, cloud (Backblaze and G-Drive) and a rotation of hard drives stored at the bank. The most valuable of data gets replicated everywhere, while things that would be annoying to recover are replicated on site.