Question Is the cost of fixed storage always going to come down

[mikeymikec]

One thing I often say to my customers (with basic needs) in general is not to spend time going through your personal data and making arduous decisions about what to keep and what to get rid of, because the cost of storage will keep coming down (and partly because I know that the vast majority of them will say that they should go through such a process then either never start or give up halfway through). Generally speaking I think this continues to hold true but I think there are exceptions. For example:

A major user of space on my 4TB HDD (pure data drive, I have a boot SSD) is my film collection; it mostly comprises of DVDs ripped to NVENC H.264 (I started a few years ago on older hardware) and some Blu-Rays ripped to H.265 in software. As a result of a quick experiment on my new build I found that re-ripping the DVDs to H.265 in software resulted in a reduction in file size of between 0.5GB and 1GB per film with the same quality settings (RF16 for DVDs), and from what I've seen so far the visual result is identical. I have approximately 200 films on disc so the saving when I've finished the task is not insignificant (at least 97GB freed up).

The 4TB HDD has been around the 50-60% capacity mark for quite some time, and given that I'm also doing things like scanning my parents' old photos in as 600dpi TIFFs (600dpi in case someone wants to print one in full photo quality from an inkjet, and TIFF to preserve as much quality as possible especially if a given photo is to be improved in some way), I expect more space will get used during my long-term use of the drive. Judging by past usage, I expect that this drive will get full enough to get something bigger eventually, especially if I carried on storing everything on this drive and not optimising usage with say my film re-ripping work.

My reason for the question in the thread title is that historically storage has always become cheaper, but the fact of the matter is that the cloud is a game-changer. In my line of work I'm fairly sure I'm seeing fewer customers who store lots of data on their computers because these days they take pictures on their phone and the content is backed up to the cloud, and they're happy with that. I trust the cloud with as little as possible let alone all my data and I doubt that's ever going to change. Logically though, I would expect that as fewer people purchase 'very high' capacity drives that the prices won't just keep coming down as they always have done. Cloud providers probably will buy enterprise-class drives so IMO their increased usage won't inherently drive mainstream prices down either.

I'm not one of these people who will happily drop hundreds of pounds on a bigger drive because "I want more"; the 4TB HDD I have is one of a pair that a customer gave to me when they upgraded their NAS! (I have the second one as a backup drive) Buying say an 8TB HDD (I'm going for NAS class drives in future because those seem a lot more guaranteed to have the APM function that I rely on so Linux will auto-power down the drive) is pretty pricey, I think it was just shy of £200 UKP last I looked.

The other thing I'm doing also breaks a general rule I recommend to customers whereby any data that they want to back up ought to be on their main PC and then having a simple backup system that copies it to external media (otherwise they would have to keep track of what's on each external media), is that I've archived some data off my PC to two spare HDDs partly to reduce the usage of my main 4TB HDD, partly to speed up some of my backups, and partly to reduce the need to replace my army of lower capacity HDDs which are almost exclusively cast-offs from customers; for example my 'archive disks' are two 2.5" 500GB SATA HDDs; I copy the data to be archived to both archive drives then delete from the main PC. I also have a spreadsheet that details everything that's been archived and when all my different types of backup have last been run and what they include.

 

[manly]

Do we know that AWS/Azure/GCP primarily buy enterprise drives for "bulk storage?" Backblaze does not. Consumers more or less don't buy HDD anymore (yes they exist, but regular people aren't major buyers). Anyway, data "needs" are still growing robustly overall and ironically, cloud storage has probably amplified that. So quite obviously long-term, storage is getting cheaper and there's little reason to believe that is ending anytime soon. Short term, you see occasional anomalies. HDD prices spiked due to 2011 Thailand flooding and although capacities have since jumped, the gradual transition to SSD means price points remain elevated.* The NAND glut is now over, and SSDs are way more expensive than they were 6 months ago.

I don't know how it is in the UK but you can get used enterprise drives here pretty cheap (i.e. server pulls). I've been thinking about upgrading my 4TB NAS to 12TB, which runs about $90 each. However, I'd need two drives for redundancy and probably a third for backup. Like you, my 4TB isn't actually running out soon but that's partly because I've never consolidated some older Downloads onto it.

What is your workflow for scanning old photos like? At one point I thought about doing the same, but it's a huge time sink and I'm not happy about the legacy scanners I have on hand.

Ripping DVDs again just to save 100GB out of 4TB sounds like way too much trouble unless the whole process was automated.

* Prices aren't actually going down, i.e. can you buy a consumer 8TB or 12TB HDD for cheap? Not really, because the manufacturers would rather sell you a 16TB drive for > $220. Mainly the cheap external HDD you see for sale are portable drives (2.5") with small capacities. Sadly some of these 2TB drives are roughly the same price they were 4 years ago when I last bought one.

 

[mindless1]

If those old 2nd hand HDDs work for you, great, but personally I will buy a lot of different things used, but not HDDs. My bulk storage HDDs are just whatever WD or Seagate where cheapest/capacity at the time I wanted more, and then anything important is backed up onto offline externals, and then anything really important, gets backed up more often in multiple different ways depending on data size.

Making decisions what to keep and what to get rid of, for me it depends on what it is. Video in particular since it eats up so much storage space, if I didn't get rid of some, I'd have to consider myself a data hoarder and be wasting space. Will I ever watch old episodes of Miami Vice for example? Doubtful, but I haven't deleted them yet either, probably because they only take up 40GB of space, not much compared to modern 1080p or 4K stuff. Except for some of my favorites, most things that I watch, get deleted afterwards. I'm not the type of person who watches something over and over.

Storage prices drop, but the amount of data expands faster than it drops, if you aren't particular what you collect and what you delete.

I don't see cloud storage as a game changer at all. Upload speed is too slow unless you have an expensive ISP plan and if you're willing to spend more for such a tech oriented service, you most likely also have the disposable income to put towards more/new HDDs too. If talking about less data, I trust my ability to access a cloud that still has the data years from now, far less than having it backed up both on and off-site, and with the on-site access at least an order of magnitude faster.

My only use for clouds is to share files, though I could see setting up an automated cloud backup for someone that isn't tech savvy or is forgetful or lazy, so unlikely to make backups if it has to happen any other way.

 

[mikeymikec]

@manly My scanning workflow is nothing special, I've got Epson Scan in a Win7 VM (I run Linux and while scanning works fine on Linux, I thought I might need the whole armament of options that ES gives). It's tedious work but I've optimised the process a bit in ES by switching off the file options confirmation page so it just saves straight to TIFF once I click on Scan. The 'Zoom' option can also be utilised to half the time of a pre-scan and get a closer view which works quite well if you're scanning in a load of same-size 6x4 landscape shots, though one thing that really surprised me is just how common it was for photos not to be a set 6x4 size and I bet that the person doing the exposure onto photo paper cut the photos either by hand or used some process that doesn't result in purely rectangular sheets.

I've got a preset set up in Handbrake to make DVD ripping workflow easier so I just have it scan a disc, set the preset, do the subtitles, name the file and queue it up. I have two optical drives so I queue two jobs and set it off while I go do something else.

@mindless1 Bear in mind that 99% of the time I know where those HDDs have come from, ie. I retired them myself when say a computer was scrapped or a HDD was replaced with a SSD.

---

Right now my main supplier is selling a Seagate Ironwolf NAS 8TB drive for £180 UKP, perhaps I'll remember to check in on this thread in a year or two's time to see what's happened to the price!

I suppose the other thing I dread is the lottery of how noisy / whiny the new drive will be. The Seagate 7200.12 500GB wasn't too bad before I found it had a bad sector on it, I replaced it with a WD Black 1TB which was a little more noisy but its noise type wasn't irritating and it's actually of some help to be (barely) able to discern it accessing data. The WD Blue 4TB I replaced that with was a whiner and I had migrated to Linux at that point and that drive's inability to do APM meant that it was constantly spinning in Linux. The Seagate 4TB Ironwolf NAS drive I replaced it with is probably similar to the other predecessors, but I've heard some drives that really surprised me, like hearing a WD Black 2TB in the era of my WD 1TB drive and the 2TB drive sounded like it stomped around in its container when seeking data in comparison to the 1TB drive, almost like the obvious clunking sounds that drives used to make say pre UDMA era.

 

[dr1337]

Can yall please grow some more brains and get bigger hard drives so you don't have to waste time re-encoding things and then also you get redundancy in the form of having multiple disks with the same content. srsly blowing my mind out here

Hard drives are getting so massively cheap now that I treat them like CDs or DVDs. Hotswapping them around if I want to find something ancient that I don't already have on NAS is way more convenient than fussing about with a tiny amount of storage and juggling files.

Like really, I spent $120 in 2010 for 2tb, 16tb is ~$150 on sale now in 2024. Cloud storage isn't free in of itself either ya know, and even then has its own uses. I might like my phone backing up notes and pictures, but its not like I watch my movie rips on the go ever. And generally if I am travelling then I am smart enough to pack entertainment with me instead of needing to summon it on the spot.