Question How would you store vast amounts of data in your basement?

[cytg111]

Rather just a spit balling question actually, hope its ok.

Suppose you are producing 30-50G of h264 video pr. day, and you want to archive it all, dont throw anything away.

Ill keep like, maybe 2 months online on spinning discs or ssds .. that is like 3TB give and take, from there it has to go to some sort of archive.
*what* sort of archive?
I have been contemplating a NAS, that could give me another 10 months maybe, ill probably do that, still, from there it still has to go somewhere.
Tape? BluRay discs? Another NAS where I downsample the videos to lesser quality and reencode to h265.
I could keep only those segments that registers motion, but i'd like to keep it all in case I need to run another motion detection algorithm(or other algos) on days past.

How would you store vast amounts of data in your basement? .

Moved to the storage forum for you. Consider changing the title to reflect the question: you want to archive large amounts of video. AT Mod DAPUNISHER

 

[Muadib]

A NAS whether you build or buy, will do what you want. You will still need a backup solution for the NAS though.

 

[cytg111]

A NAS whether you build or buy, will do what you want. You will still need a backup solution for the NAS though.

And for even longer time "on the shelve" - if online is not an option?
Optical disc?
hdd-drives? (xxT nas drives)?
Tape?

I could bend towards making a setup where I just put it on large and slow regular hdds and file em in a box somewhere.

 

[Muadib]

Hard drive gets my vote. I’ve had both of the others fail on me. Yes hard drive can fall too, but I’ve never had two fail at the same time. As long as you are using two or more for backup, I think you should be fine.

 

[sdifox]

BB pods, but you need to have a few of them to be fault tolerant.

Open Source Storage Server: 60 Hard Drives 480TB Storage

Get the details on Storage Pod 6.0 and learn how to build your own data storage server.

www.backblaze.com

 

[Arx Allemand]

30-50GB a day is nothing!
Our 10 camera NVR saves 300GB per day.
We just use Seagate Exos 16TB drives.
FreeNAS on a repurposed server is cheap and ZFS works well.

 

[killster1]

i would just encode to h265 that should save you 5x the space then id keep adding to the array of spinners. 30gb per day isnt bad. why not invest in h265 recorder.

 

[Muadib]

30-50GB a day is nothing!
Our 10 camera NVR saves 300GB per day.
We just use Seagate Exos 16TB drives.
FreeNAS on a repurposed server is cheap and ZFS works well.

I just got Exos drive to put in my NAS. How do you like them? Do you find them to be noisy?

 

[cytg111]

i would just encode to h265 that should save you 5x the space then id keep adding to the array of spinners. 30gb per day isnt bad. why not invest in h265 recorder.

Well, its my own cigar and so far I have only gotten h264 to work properly... further down the line i'll get into 265 but it's not plug'n play 264->265, wish it was. For now h264 is a constraint I must live with.. plus I imagine 264 is much faster to unpack for analysis.

 

[cytg111]

30-50GB a day is nothing!
Our 10 camera NVR saves 300GB per day.
We just use Seagate Exos 16TB drives.
FreeNAS on a repurposed server is cheap and ZFS works well.

Damn, that is some data alright ... but you are like spending 10T/month, getting a new disk every 1½ months? You just store all those disks in a box somewhere?

 

[killster1]

Well, its my own cigar and so far I have only gotten h264 to work properly... further down the line i'll get into 265 but it's not plug'n play 264->265, wish it was. For now h264 is a constraint I must live with.. plus I imagine 264 is much faster to unpack for analysis.

whats your own cigar mean!? 264 faster to unpack? im pretty confused, what is it you are doing? also they should store them in a cool humidity controlled safe most likely thats where all mine are stored (onsite and offsite)

 

[Arx Allemand]

The 16TB Exos and Ironwolf Pro are very similar drives. There is noise but the drive(s) are in a rack full of POE switches and 1U servers so TBH I can't hear anything over that racket!

As for data, yeah there's typically a 60 day retention policy and then it gets purged. Footage that's been tagged/locked won't get purged ever.

Think of it like a septic tank, eventually the solids build up and it needs to be pumped. We run freeNAS across several servers and those host a plethora of stuff (ex. Plex) and SMB shares too.

It's amazing how fast both storage capacity AND demand grows.

In 1995 my main PC had 64MB RAM. Today it has 64GB. A thousandfold increase or approximately so.

My first IBM AT (286) had 2MB RAM and my colleagues teased me always asking what are you going to do with all that memory?

Now even the bottom feeder CPUs have more than that in L3 cache!

 

[cytg111]

whats your own cigar mean!? 264 faster to unpack? im pretty confused, what is it you are doing? also they should store them in a cool humidity controlled safe most likely thats where all mine are stored (onsite and offsite)

My own code, connects to rtsp camera, gets feed, unpacks, analyses, repacks and stores. Using ffmpeg libs for the encodig decoding parts but its not straight forward just switching to 265 plus when I re-run motion detection again later, decoding it will be faster for 264 over 265 (I would think)

 

[Arx Allemand]

My own code, connects to rtsp camera, gets feed, unpacks, analyses, repacks and stores. Using ffmpeg libs for the encodig decoding parts but its not straight forward just switching to 265 plus when I re-run motion detection again later, decoding it will be faster for 264 over 265 (I would think)

Interesting!

Familiar with this?

Shinobi

shinobi.video

 

[mxnerd]

A nice case for storing 20 offline HDDs

Amazon.com: ORICO Hard Drive Case, 20-Bay Multi-Protection HDD Storage Box Suitcase with Foam, Anti-Static Shock Moisture Proo for WD, Seagate, Toshiba, 3.5" HDD

Buy ORICO Hard Drive Case, 20-Bay Multi-Protection HDD Storage Box Suitcase with Foam, Anti-Static Shock Moisture Proo for WD, Seagate, Toshiba, 3.5" HDD: Hard Drive Bags & Cases - Amazon.com ✓ FREE DELIVERY possible on eligible purchases

www.amazon.com

 

[KentState]

This is what I use in my basement.

 

[cytg111]

Interesting!

Familiar with this?

Shinobi

shinobi.video

Yes been getting inspiration from around, I know I am reinventing the wheel, I like it like that, I know Im weird .

 

[cytg111]

This is what I use in my basement.

View attachment 22286

Beastmode! How many watts does that thing pull at the wall? (I have just built a 16/32 core 2p xeon (sandy) from used parts, its sitting naked on a desk all parts out lol, running esxi)

 

[KentState]

Beastmode! How many watts does that thing pull at the wall? (I have just built a 16/32 core 2p xeon (sandy) from used parts, its sitting naked on a desk all parts out lol, running esxi)

Both of the servers seem to pull around 300w each. The jbod array is much lower since it's only SSD, but still shows a few hundred. I had 6 * 20amp circuits installed in that area so I can handle a lot more equipment and luckily our power is pretty cheap. $.07 kwh for 9 months of the year and while there are peak charges over the summer, it's only for the first usage tier which we hit anyways and then goes to $.07kwh for anything above it. In actuality, my 3970X desktop with monitors draws 600w+ at the wall when using it during the day so the servers are not a big deal.

 

[Fallen Kell]

I build a system in a Supermicro 846. The reason I went with this case is that you can easily swap out the internal fans/fan wall with 3x120mm fans to quiet the system down. I currently have 50TB of usable space with raidz2 (equivilent to RAID6), but can grow it to 200TB using the same sized disks that I have just in this one chassis (and can easily get an expansion chassis). With the proper power supplies (the super quiet line (model numbers end with "SQ") from super micro), it will be very quiet for a server (I have had plenty of desktops louder than this system).

Hard to beat 24 hot swap 3.5" disks plus room for 2x2.5" internal disks and a DVD drive on a server. I have 6x14TB drives in mine. But as stated, can easily grow that to 24 using groups of 6 in a RAID6 or (raidz2 in my case) for fault tolerance, especially when you can see these chassis/bare-bones systems come up from time to time in the $200-800 range. I think mine was $600 with dual 6core/12thread E5-2630v2 Xeon's and 192GB RAM.