Best options for backup

[Brian Stirling]

There's two kinds of backup I'm looking at and I'd like some feedback on how best to go about it.

The first kind of backup is an image of your HD/SSD at some point in time, likely pretty early, that can be loaded onto a new drive if the original fails. In my case, I had a 500GB HD on a laptop that I recently upgraded to Win 10 and then cloned the HD to a SSD (mSATA SSD). So, I guess I have the image backup covered with the original HD, but what else is possible?

The second kind of backup is ongoing data. It would be impractical to do a full drive image every time you do a data backup, but ideally, you want an organized method for recovering if the drive fails. If the drive fails you can rebuild from the original image but you still need to cover restoring the data that isn't part of the original image. So, how are folks handling this task these days and what automated/program options are best.

Ideally, one would like an automated way of building a new drive by taking the original image and then adding the new data, but what about adding the programs that have been installed since the original image? Years ago when people used tape drives you had complete backups and incremental backups but that was when HD's were less than 100GB and not many people use tape drives anymore.

Perhaps having a few images would be wise -- an original image and then perhaps two alternating images when the system changes like when new software is installed. Other than image and video data the amount of storage needed for pretty much all other data is fairly low so backing up ALL such non-image and non-video data periodically should be relatively easy.

For image and video storage, given the huge data requirements, it's not practical to frequently do a full backup of all that data as it could easily be many TB. What I do now is maintain two 6TB HD's in my desktop and about 8 external HD's so I have at least 3 copies of all the image and video data. When new image or video is captured, usually away from home, I download to my laptop then copy to the externals. When I get home I then copy to my desktop.

If that is enough I have at least 6 active PC's, all but one laptops so managing backups for 6 PC's would make the whole backup thing all the more complicated.


Brian

 

[frowertr]

There's two kinds of backup I'm looking at and I'd like some feedback on how best to go about it.

The first kind of backup is an image of your HD/SSD at some point in time, likely pretty early, that can be loaded onto a new drive if the original fails. In my case, I had a 500GB HD on a laptop that I recently upgraded to Win 10 and then cloned the HD to a SSD (mSATA SSD). So, I guess I have the image backup covered with the original HD, but what else is possible?

This is exactly how most business do it with AD. They will create 'virgin image' by adding the needed drivers, programs, and security patches and then save that image. It can then be deployed via PXE boot to the computers within their AD. So if an employee quits or gets fired, it makes it real easy to wipe and repurpose that computer for the next guy. Also makes dealing with Malware easier. Employees are typically told (and made to sign a document proving they were told) not to keep any important files on the local computer as their local machines aren't backed up at all. All of their files should be saved to network drives (NAS/Fileservers) which are backed up by the IT department according to their practices.

The second kind of backup is ongoing data. It would be impractical to do a full drive image every time you do a data backup, but ideally, you want an organized method for recovering if the drive fails. If the drive fails you can rebuild from the original image but you still need to cover restoring the data that isn't part of the original image. So, how are folks handling this task these days and what automated/program options are best.

Ideally, one would like an automated way of building a new drive by taking the original image and then adding the new data, but what about adding the programs that have been installed since the original image? Years ago when people used tape drives you had complete backups and incremental backups but that was when HD's were less than 100GB and not many people use tape drives anymore.

Most methods of backup today use a block tracking system. So only changes are backed up after the initial first full backup.

But the answer to your question really hinges on how and where your data is stored. If you are storing your data on a NAS that is separate from your computer, then it doesn't matter if your computer crashes. Reload the OS, reconnect the NAS, and continue on. The NAS still needs backup however. You can either replicate/version to an additional NAS or backup online to EC2, Glacier, Crashplan, etc...

If you are storing your data locally (in the computer your are also working on), then you need a backup plan that takes that into account. Veeam Endpoint is great for Windows systems. It uses the Change Block Tracking I mentioned, so that the first backup is a total backup while subsequent backups are only changes. You can point Veeam Endpoint to either a local disk (internal or external via USB/eSata) or a network drive (NAS).

BTW, tape is till used heavily in Enterprise. It pretty good for cold/archival storage. The new tape speeds make it pretty damn snappy to from what I have seen been told. I don't use it but it's still available widely available.

For image and video storage, given the huge data requirements, it's not practical to frequently do a full backup of all that data as it could easily be many TB. What I do now is maintain two 6TB HD's in my desktop and about 8 external HD's so I have at least 3 copies of all the image and video data. When new image or video is captured, usually away from home, I download to my laptop then copy to the externals. When I get home I then copy to my desktop.

If that is enough I have at least 6 active PC's, all but one laptops so managing backups for 6 PC's would make the whole backup thing all the more complicated.

Backing up 6 PCs by hand sounds like work. You need a central NAS that you can backup everything to. Depending on how important that data is to you, you then would need to backup that NAS via another NAS/external drive(s)/online.

What I've found is that the average user that is storing over ~8TB worth of data at home is keeping mostly "junk". Torrents, movie rips, music, etc... I'm not sure versioning/replicating to a secondary NAS is worth the added expense for home use. I'm not saying this is your case or everyone's, it just an observation I've made.

 

[Charlie98]

The only PC I have a need to back the OS up on is my primary (business) desktop... I have a full image copy of the clean OS reload in case of some complete failure of some sort. Beyond that, I use Acronis to create a full backup image every night, across at least 3 HDDs (2 internal, at least 1 external, with a recent backup image in the safe.) I think my images run about ~120GB each, so I use 2TB drives and maintain a weeks worth of backups, plus the last 3 'first of the month' images (in case there is some recent corruption not evident.)

I backup the OS image on my HTPC, but the ~5TB of media files is simply backed up to 2 separate 3TB HDDs (files split between the two.) Although the media files represent a fair amount of time invested, if I lost them it wouldn't be the end of the world.

Backing up 6 PCs by hand sounds like work. You need a central NAS that you can backup everything to.

I would agree on that...

 

[Ketchup]

My system is pretty simple in comparison, but it seems to work (for me).

I use Windows Backup for my desktop to the server in the house. Runs once a week. After a couple months (less if there are significant changes) I update my backup drives (I have two). This would include recent backups from the desktop. When not running backup, these drives sit outside the computer. So they only run when backing up data. For home use, I am a fan of having a copy of all important data in a safe, static location 99.9% of the time.

 

[ronbo613]

Macrium Reflect to image the boot SSD periodically or after any major changes or installations to a few internal and external drives. Regular incremental backups of valuable files to NAS, internal and external hard drives. Important files are backed up in at least three different locations including offsite storage.

PITA but you absolutely have to do it.

 

[Brian Stirling]

This is exactly how most business do it with AD. They will create 'virgin image' by adding the needed drivers, programs, and security patches and then save that image. It can then be deployed via PXE boot to the computers within their AD. So if an employee quits or gets fired, it makes it real easy to wipe and repurpose that computer for the next guy. Also makes dealing with Malware easier. Employees are typically told (and made to sign a document proving they were told) not to keep any important files on the local computer as their local machines aren't backed up at all. All of their files should be saved to network drives (NAS/Fileservers) which are backed up by the IT department according to their practices.



Most methods of backup today use a block tracking system. So only changes are backed up after the initial first full backup.

But the answer to your question really hinges on how and where your data is stored. If you are storing your data on a NAS that is separate from your computer, then it doesn't matter if your computer crashes. Reload the OS, reconnect the NAS, and continue on. The NAS still needs backup however. You can either replicate/version to an additional NAS or backup online to EC2, Glacier, Crashplan, etc...

If you are storing your data locally (in the computer your are also working on), then you need a backup plan that takes that into account. Veeam Endpoint is great for Windows systems. It uses the Change Block Tracking I mentioned, so that the first backup is a total backup while subsequent backups are only changes. You can point Veeam Endpoint to either a local disk (internal or external via USB/eSata) or a network drive (NAS).

BTW, tape is till used heavily in Enterprise. It pretty good for cold/archival storage. The new tape speeds make it pretty damn snappy to from what I have seen been told. I don't use it but it's still available widely available.



Backing up 6 PCs by hand sounds like work. You need a central NAS that you can backup everything to. Depending on how important that data is to you, you then would need to backup that NAS via another NAS/external drive(s)/online.

What I've found is that the average user that is storing over ~8TB worth of data at home is keeping mostly "junk". Torrents, movie rips, music, etc... I'm not sure versioning/replicating to a secondary NAS is worth the added expense for home use. I'm not saying this is your case or everyone's, it just an observation I've made.

Man, I appreciate your thoughtful response, really appreciate it...

As far as my data is concerned the overwhelming bulk of it is image files from my various digital cameras as well as slide scans from before I shot digital. An equal data hog and growing is video. I've gotten more involved with video in the last couple years and with 4K video shot at the higher bit rates you go through a crap ton of data fast. So, while my non-image and non-video data needs are pretty modest the image and video is well into 4TB and growing quickly.

I think I have a decent handle on the image and video backup and have used multiple external HD's to maintain at least 3X redundancy. So, for the non-image and non-video backup I'd surmise a full backup would not be much more than 40GB after compression with most of that being progams and not data. Data alone probably in the 10GB-20GB range pre-compression so probably 3GB-8GB post compression.

I have dropbox and google drive but only ever use dropbox. I don't want to shell out monthly payments for massive cloud storage as the costs for that add up particularly when travelling and using cellular bandwidth.

I do have a NAS but it's a couple years old and only 4TB in size so it's already kind of maxed out. I do have much of my image and video on it so I suppose I could remove that and free up the NAS for non-image and non-video backup as well as virgin images of all my PC's. But, as you rightly point out, the NAS would then become a single failure point so it would need to be backed up as well. The always on NAS thing is nice but has the drawback of working the drives 100% of the time and if you don't have enterprise level drives your odds of a failure are pretty high.

How deep does the rabbit hole go...


Brian

 

[frowertr]

It really depends on how much your data is worth to you and if it would cripple you if you lost it. For many, backing it up to an external drive is enough. For others, they would have a panic attack if they couldn't get a copy of it offsite as well.

There isn't a right or wrong answer with backup. Just depends on your needs.

I really like Amazon Glacier. Its perfect for cold storage that never changes/gets accessed. But you are only talking about 4TB which, honestly, is a pretty modest amount. I would think keeping that on a couple of externals (and keeping one offsite) would be pretty safe.

It really sounds like you need to invest in a larger NAS so you don't have to juggle externals drives constantly to access your data. Synology makes some real good ones.

 

[Concillian]

If that is enough I have at least 6 active PC's, all but one laptops so managing backups for 6 PC's would make the whole backup thing all the more complicated.


Brian

The way I handle this with the PCs in my house is this:

A file server that anything important gets put on.
The file server has 3 drives.
1 - boot
2 - an imaged drive that spins up every night, does an incremental backup of anything on the boot drive from the last 24 hours
3 - an external drive that I image ~once a month (okay, more like about once every 3 months) and put in a "fireproof" safe.

Tools in Linux to do this are fairly easy. I forget how and then google the scripts again every ~5-6 years when I update my file server PC. Using dd to create images, these are 100% plug in and boot if the main drive fails. The low duty cycle on the 2 backup drives was a key point to me.

The other PCs are fairly easily rebuilt, so I consider them to be expendable. 90% of the applications are download and install the latest version anyway (open office, firefox, etc...) and as you note, keeping 6 PCs up to date with backups is a real chore.

You just have to get the other people in your family used to using the network drive. But once you do, having a common place for family photos / videos and like itunes folders and such becomes very convenient for everyone, because they have access to all their stuff from any PC in the house.

 

[Brian Stirling]

So, what about the available backup programs on the market? You have programs like:

Acronis Backup for PC, Shadow Protect, Macrium Reflect, Paragon Backup & Recovery, NovaBackup and many more. My problem is that I don't like adding things I don't have to as that invariably slows your system down. Obviously, the worst offenders were/are the virus scanning suites from Norton et al, but I don't see info on how much these backup programs effect your system. For me, I'd prefer as little impact on my system as possible and don't know that I'd want it running on startup every time.

Since I pretty much handle my huge data backup of image and video files more or less manually with external drives I'm wondering if the built in backup within Windows 10 is sufficient for the drive image and data backups and given it's already there on all my PC's and it wouldn't add any overhead to system operation.

I may remove my image and video storage from my current NAS and free up space to permit drive images and backup folders for some/all of my PC's and without the huge data requirements of my image and video files I'd have more than enough storage for a virgin image plus a couple backup images in addition to incremental backup data for all my PC's.

Another thing I'm thinking about is using some of my retired HD's for backup purposes. As I've gone from 500GB drives to 1TB drives to even larger drives to handle the growing storage needs these older drives could be re-purposed to do redundant backup of smaller data sets like backups. I wonder if any of the commercial backup programs handle multiple external drives as destinations for backups. I wonder how Windows 10 backup handles that.


Brian

 

[frowertr]

There are tons and tons of backup programs out there. The problem is most are junk and wading through them to find a decent one is difficult. I recommend Veeam Endpoint. So long as you have a NAS, you can backup your entire LAN directly to it via Endpoint. Makes it pretty simple. You can choose to do a system image or file/volume level backups.

It's not a system hog. It does load at startup but doesn't really do much of anything until you manually start the backup or the scheduled backup is running.

I've never used the Windows backup program so I don't have an opinion on that.

 

[Anteaus]

I really like Beyond Compare. It is inexpensive and does a great job with file level backups. Its very simple to sync drives or directories. It will even do a bit level file comparison. It runs on Windows, Linux, and Mac.

http://www.scootersoftware.com/

 

[Brian Stirling]

There are tons and tons of backup programs out there. The problem is most are junk and wading through them to find a decent one is difficult. I recommend Veeam Endpoint. So long as you have a NAS, you can backup your entire LAN directly to it via Endpoint. Makes it pretty simple. You can choose to do a system image or file/volume level backups.

It's not a system hog. It does load at startup but doesn't really do much of anything until you manually start the backup or the scheduled backup is running.

I've never used the Windows backup program so I don't have an opinion on that.

OK, downloaded Veeam Endpoint on two PC's and have made a full backup of both. Will do the same with my other PC's later.

I've ordered some thumbdrives so I can dedicate one to each PC given that the drivers and configs are not the same across the PC's.

My NAS is kind of flaky and often kills the wifi network when accessing so I made the first full backups with a retired external HD. It doesn't look like Veeam does much compression as the image size is pretty near the total size of the drive. Other than already compressed stuff like video and zip files the rest should compress by a factor of 2 and raw data usually better than a factor of 4 so I would have expected the image to be on the order of half the installed size.


Brian

 

[dave_the_nerd]

I do a full image of my C: drive every day.

Differential backups or file backups like Crashplan, Time Machine, or even the Windows Backup utility, have their advantages, but recovery is sometimes more involved and I am lazy. I just want to buy a new SSD, restore an image to it, reboot, and be done.

 

[Brian Stirling]

I don't think a full image every day is in the plan for me -- if I have a HD or SSD die or my system gets hosed by malware restoring from a more virgin config is an opportunity to start fresh and get rid of the junk that accumulates.

It does takes some time to get back to where you were, but that's the trade off.


Brian

 

[Ketchup]

I am with you on that Brian. I can also see a self-fulfilling prophecy in writing a fresh image of an entire drive every single day. Hopefully the image isn't being written to an SSD (hey, at least with a hard druve you have the hope of it lastingva few years with said setup).

 

[dave_the_nerd]

I am with you on that Brian. I can also see a self-fulfilling prophecy in writing a fresh image of an entire drive every single day. Hopefully the image isn't being written to an SSD (hey, at least with a hard druve you have the hope of it lastingva few years with said setup).

It's to an HDD array in my home server.

The only thing I'm really trying to guard against is disk failure. All my "good" data lives on a NAS that's backed up to Crashplan; the main annoyance if the SSD in my system died would be downloading all my Steam games again.

I don't use the system much except for gaming, I don't pirate stuff, I do network-level ad-blocking, don't customize my OS or install third party utilities, and I'm not particularly worried about malware. So the most likely (based on personal experience) source of issues is a drive failure.