Cost Effective backup solution for Backupexec

[chuck2002]

Hi. Here is my scenario:
At work, we are backing up about 500 gigs of data per night.
We are backing up several Windows servers via BackupExec version 11d and the Backupexec remote agent.
We are currently storing the backups on a Drobo Pro device with 8 x 2 TB drives communicating via iSCSI.
We have two backup servers. They both can't communicate to the Drobo via iSCSI, (Drobo limitation) so I have one communicating with a standalone USB drive connected directly to it a robocopy job that copies the .bkf files off of the usb drive and onto the Drobo via network share during the day when backups are not happening.

To work around the problem of both of my Backupexec servers not able to communicate directly to the Drobo device via iSCSI, I have tried creating a backup to disk folder and setting the disk to be a network share that is located on the Drobo. This appears to work in the Backupexedc config, but the backups get partially done and then fail with a generic communication error.

I am considering dumping the Drobo all together and moving to some other cost effective drive array solution that will let me talk to it directly via iSCSI from both of my backup servers. I don't want to spend the big bucks on a SAN. I think the cost of the Drobo was reasonable for my organization.

Does anyone have any good solutions that they have employed for this issue?
Thanks.

 

[imagoon]

Why do you need 2 backup servers? Roll the jobs in to one. 500GB is peanuts for a single BE server.

 

[chuck2002]

I don't have to have them both, but as is my backups just get done. Starting at 5 pm and going till 9am the next morning. If I have them on two servers, I can get the backups done overnight without running over.

 

[imagoon]

I don't have to have them both, but as is my backups just get done. Starting at 5 pm and going till 9am the next morning. If I have them on two servers, I can get the backups done overnight without running over.

Sounds like you have something really slow in the link then. I have run terabyte backups over a few hours.

 

[stlcardinals]

I made my own backup server with a server in the HP DL100 family. Added an MSA with SATA drives. Windows Server 2003 standard with BackupExec 11d. HP 1/8 Autoloader for tapes attached to it. Backups were D2D with each Server having their own D2D drive.

I believe I was backing up close to 500GB each night, then had Backup exec backup the D2D drives to tape.

Worked like a champ.

 

[chuck2002]

Sounds like you have something really slow in the link then. I have run terabyte backups over a few hours.

Wow. What are you backing up to and from?
I am going from standard Dell servers, 2950, etc, Gigabit network.

I had a home built 3ware RAID server setup with 8 drives running a RAID array and this worked too, but it was configured with 1TB drives. We needed more capacity because we are trying to keep these nightly backups for 30 days.

 

[Fardringle]

Sounds like you have something really slow in the link then. I have run terabyte backups over a few hours.

Agreed. I run ~500GB backups every night and it takes about two hours including file verification.

 

[chuck2002]

What are you going to and from for your backups??

 

[imagoon]

Dell R610, 2950 etc via gig to either a tape library or the SAN. Depends on where the backups are going. Offsite goes to tape.

 

[Fardringle]

The backups I mentioned are pulled from multiple desktops and a secondary server to a removable SCSI hard drive enclosure (drives are pulled and taken off site) attached to a SBS 2003 server. The backup jobs run one at a time so the multiple sources don't make it run any faster compared to a single source.

I read some reviews on that Drobo Pro device and it looks like it should be able to easily handle fast data transfers so if your backups really are taking 16 hours to move 500GB I suspect it is because:

1) The Drobo is malfunctioning, or one (or more) of the drives in the Drobo are malfunctioning, causing the backups to take longer to be stored on the drives.

2) The server(s) running the backups are overloaded/slow and taking a long time to compress the backup data. Try running an uncompressed backup to see what happens.

or
3) As imagoon suggested, there's something wrong with your network. Make sure all devices in the backup chain are connected at gigabit speeds and actually functioning at that speed. Test the connection with a LAN benchmarking tool (I like LANBench for simple tests) to make sure you are actually able to move data at gigabit speeds. 500GB in 16 hours is roughly 8.7 Megabytes per second (71 megabits/second) which corresponds very nicely to the average capability of a 100 megabit connection, so my guess would be that somewhere along your backup chain you are connected at 100mb and not gigabit.


edit: typo.

 

[chuck2002]

Damn. Looking at my Job History on the server that writes directly to Drobo via iSCSI here are my stats:
Elapsed time: 15:56
Byte Count: 376,971,648,895
Job Rate: 393.44 MB/min

What are you guys getting?

 

[chuck2002]

Thanks for the tips. I will check out my network speeds via the speed test.
These servers are in a rack along with two switches in the rack. They don't have far to go.

Also, under backup job Settings|Advanced do you guys have Enable single instance backup for NTFS volumes. I have that unchecked.
I am also doing software backup compression in backupexec. I will try your suggestion to disable the compression.

 

[Fardringle]

Your transfer speed fits too nicely with the speed of a 100mb connection to be a coincidence. I would check that before you do anything else. Backup compression really shouldn't be a problem unless the server doing the backups is already overloaded and/or slow. I only mentioned that as a possibility in case the other (more likely) options don't pan out.

 

[chuck2002]

Ok looking trough the job history, I think it is the Drobo that is causing my slowness.
on 3-25-10 I did a job where I just changed the existing job's destination to an internal 1TB drive and I got 793.12 MB/min.

 

[imagoon]

To give you a bit of a reference, the slowest system to backup I have (Domino, due to loading and how domino does backups) pulls about 1,500 MB/min

 

[seepy83]

chuck - As far as the speed of your backups is concerned...

you mentioned you're using Dell 2950's and iSCSI. Are you using the onboard Broadcom nics for your iSCSI Connections? If you are, you might want to look into disabling TOE on those cards, and even removing the TOE Chip from the motherboard. Google and/or call Dell for more info, but I ran into some major performance problems with 2950's attached to an EMC iSCSI SAN and it was all related to TOE not functioning properly.

The other thing I would be curious about is the content of your backups. For example, your throughput will be much slower if you're backing up thousands of small files that are 500 GB in total (vs backing up 100 files that are 5GB each).

 

[drebo]

I'd say upgrade your BE to 2010 (gets you free CPS server use) and use your server connected to the Drobo as a CPS server. Then buy an LTO3 or LTO4 drive to do a nightly snapshot of the CPS server. That way, all file changes are replicated in real time and you don't need to worry about network congestion trying to get a backup done over night.

 

[Emulex]

just make sure your drobo can handle the bandwidth and try to not overlap and fragment backups.

I do full backups (Veeam for servers/BESR for 30+ machines) every night - full - no incremental/delta's. the destination server has 12 500gb in raid-5 with dual gigabit links which are manually balanced using SMB (windows storage server).

works great. about 1.5TB a night. the key is not to overlap backups as it can cause a million file fragments versus a few hundred.

when the battery on the bbwc failed - the backups took two days to complete so make sure you tweak your 512/1024 BBWC for best performance - i'm using 25% read 75% write on CCISSP (p400 hp raid)

 

[Emulex]

p.s. lefthand VSA - after the trial period ends - it is still a single unit lefthand vsa. runs fine on free esxi. just loses all the fancy features. but create volumes/thin provision still work. doesn't have the flakiness that windows iscsi targets have.

 

[chuck2002]

Emulex, I have no idea what you just said. Thanks for the info, but too many acronyms for me to follow.

I know now my slowdown is Drobo. I am going to see about replacing it with a box that runs OpenFiler and see how it does iSCSI. Can't be any slower....
http://www.openfiler.com/

Anyone have any input on this solution? It says it will handle 60+ TB of data, so plenty of room for growth.
I am thinking if this works ok in testing, I will get a servermicro chassis for holding all of the drives.
Something like this one:
http://www.supermicro.com/products/chassis/4U/?chs=846
This will hold 24 hot swap drives in a 4u chassis.

 

[Emulex]

openfiler works much better in a vm; than bare metal.

i think my iscsi openfiler lasted two days with 750gb drives (sata non raid type) in raid-1 before the raid died. typical sata crap.

 

[chuck2002]

I am taking suggestions for some cost effective alternative for storing my backups. I am using a Drobo Pro now directly connected via iSCSI with 8 x 2TB drives.

I made a call to Backupexec to talk with them about version 2010. There is one interesting feature I noted: Advanced disk backup. As it was explained, you take 1 full backup and then the rest of the week incremental. Instead of taking another full backup at the start of the week again, Backupexec does a synthetic full backup. It smooshes the previous full and incrementals together and makes them into your starting full backup.
Anyone have any experience with this? This could save me a ton of backup time even to my slow Drobo. With it, I would be doing nothing but incremental backups pretty much.
Also, anyone have any experience with the Sharepoint option? We have MOSS 2007 and are currently backing it up with Avepoint Docave.
How about the Vmware ESX option? We are cuurrently using ESX Ranger (vRanger) to do out virtual machine backups.
It would be nice if we could consolidate these backup functions into a single interface to reduce complexity.

 

[imagoon]

VMware Option backs up the VM's from the SAN directly or via NBD if your using direct storage. It backs up all the of the VM's files. However, if your not running on a SAN you will need to build a virtual proxy because the "NBD" mode is slower than hell since VMware limited the service console performance.

The cool thing is that backupexec can open and restore single files from the virtual machine images so it backs everything up at once. You must have a Virtual Center Server and at least the 'essentials' package.

 

[Emulex]

veeam rocks. virtualize everything. their license is per socket - not per copy so you could have a veeam backup server on every machine they are cool. scsi hot-add and all.


check out veeam 5.0 it can mount and boot a backup which is made up of say 1 full and 29 incrementals - very cool