software or hardware solution? or both?

[IceBergSLiM]

Lets say you have 200 gb - 1tb of data that you need to push to from one machine to another. What is the fastest way to accomplish this and where are the bottlenecks?

Is the bottle neck the crossover cables or the hard drive write speed or something else?

If you had to do this on a massive scale(thousands of machines) and each machine must be done within say...1 hour window time frame how would you do it. Specialized software? Specialized hardware?


Do tell.

 

[effowe]

I would think that the hard drives would be the bottleneck, but it also depends on the setup. IDE, SATA, RAID? All of these can affect the transfer speeds. Also is the ethernet 10/100 or gigabit? I have no raw numbers, but if you knew the hardware setups you could come up with some estimates. Whenever I have to transfer large amounts of data, I find it easiest to just connect a USB Hard drive to the computer I'm transfering from. Don't have to deal with opening computers or setting up networks. Convenience is the key, I don't mind waiting a little longer.

 

[IceBergSLiM]

hold on let me get you some baseline specs


EDIT: Ok let use some Current Gen Dell Builds as the Baseline and assume your going to transfer from these machines to the current Gen Dell Builds in 3 years(use your imagination)

Desktop

HD - SATA
ethernet ports Integrated 10/100/1000

Laptop is the same except it uses whatever hardrive connection dell laptops use.


Transfers can take place from Laptop to laptop or laptop to desktop and vice versa

 

[spidey07]

Gig ethernet isn't going to be the bottleneck. You should be able to do around 3 gigbytes/min so it's doable. Just don't use windows for transferring the files, use FTP.

-edit-
I see what you're going after now. I'd use a backup program.

 

[IceBergSLiM]

Originally posted by: spidey07
Gig ethernet isn't going to be the bottleneck. You should be able to do around 3 gigbytes/min so it's doable. Just don't use windows for transferring the files, use FTP.

-edit-
I see what you're going after now. I'd use a backup program.

3 gb/min is too slow 200 gb would take 70 minutes

 

[IEC]

Internal P2P network.

 

[BurnItDwn]

If everything was full gigabit, and the gigabit wasn't run on a regular 33/32 pci bus, then hard drive read/write speeds would probably be your bottleneck. If you are using really fvcking fast drives (like 15K or some SSD's), then the network would be the bottleneck. as far as "how to do it" .... are thousands of machines getting the exact same thing as the source machine? cause then you would want to look into broadcasting it .... if you are upgrading 1000 machines from model X to model Y, and you need to copy all the data from model X to model Y one a 1 by 1 basis .... then I dunno what you'd want to do ....

 

[vi edit]

Part of it depends on how the data is layed out too. If you have a million little files that up up to a TB that's going to take longer from a software overhead to copy over than a couple hundred larger files. Windows in particular HATES moving lots of little files.

 

[Crusty]

There is nowhere near enough information for anyone to help you out.

 

[MixMasterTang]

The fastest I've seen large files be transferred is usually 300-400mbps but usually it was a limitation of the write speed of the desination. If this is a serious question then you're probably going to need a SAN and have all workstations have a fibre channel card.

 

[dighn]

Originally posted by: IcebergSlim

Originally posted by: spidey07
Gig ethernet isn't going to be the bottleneck. You should be able to do around 3 gigbytes/min so it's doable. Just don't use windows for transferring the files, use FTP.

-edit-
I see what you're going after now. I'd use a backup program.

3 gb/min is too slow 200 gb would take 70 minutes

3gb/min is about 50mb/s - most hard drives have maximum sustained transfers around that so the link won't be the bottleneck. unless you've got some really fast HDDs or RAID setups, gige should suffice. gige's theoretical maximum is over 100mb/s so it can go over 3gb/min. on my home network I easily get 70mb/s transferring between win7 and the file server running debian, over samba - pretty sure the bottleneck is the HD in this case.

 

[IonBlade]

If you're really talking about 1 TB, and the situation is that you want to move the data exactly from one machine to another (as opposed to backing up just relevant, specific folders, laying down a new image, and then migrating the relevant data back over via an automated process) it's not going to be possible within an hour a machine.

Even if you remove *every* bottleneck that you can by putting both the old and new drive in the same machine, your bottleneck will become the speed of the old drive. Per StorageReview's performance benchmarks (http://www.storagereview.com/Testbed4Compare.sr - see the "Maximum Transfer Rate - Read" benchmark), the best performance you can get out of a SATA drive today is 127 MB/s. Dells won't have Velociraptors in them - at 1 TB, you're looking at, at best, the E7K1000 @ 121 MB/s.

121 MB/s * 60 sec / min * 60 min / hour = 435.6 GB / hour read. The PCI Express bus is fast enough to write off to another drive at the same speed (presuming drives have improved in the next 3 years to write at 121 MB/s sustained), so you're going to be able to hope, best-case, for 2.5 hours for a full TB copy.

It sounds like you're trying to do a migration to new machines? You might be better off making an image when the time comes and using something like Ghost Solution Suite to lay down the image to as many machines as you can plug into the network at one time via multicast, then using a centralized migration tool to move documents and settings over from the old machines to the new ones, if that's an option in your environment. By using that sort of method, you'll still likely be > 1 hour per machine, but if you can work on them in the dozens at a time, you'll still finish pretty quickly.

 

[IceBergSLiM]

Originally posted by: MixMasterTang
The fastest I've seen large files be transferred is usually 300-400mbps but usually it was a limitation of the write speed of the desination. If this is a serious question then you're probably going to need a SAN and have all workstations have a fibre channel card.

what is a fiber channel card. If both the source and destination machines had one... and a standard 7200 rpm 3 Gb/s hard drives ........help me out here.

 

[IceBergSLiM]

Originally posted by: BurnItDwn
If everything was full gigabit, and the gigabit wasn't run on a regular 33/32 pci bus, then hard drive read/write speeds would probably be your bottleneck. If you are using really fvcking fast drives (like 15K or some SSD's), then the network would be the bottleneck. as far as "how to do it" .... are thousands of machines getting the exact same thing as the source machine? cause then you would want to look into broadcasting it .... if you are upgrading 1000 machines from model X to model Y, and you need to copy all the data from model X to model Y one a 1 by 1 basis .... then I dunno what you'd want to do ....

exactly what i'm talking about. Has to be done desk side

 

[IceBergSLiM]

Originally posted by: IonBlade
If you're really talking about 1 TB, and the situation is that you want to move the data exactly from one machine to another (as opposed to backing up just relevant, specific folders, laying down a new image, and then migrating the relevant data back over via an automated process) it's not going to be possible within an hour a machine.

Even if you remove *every* bottleneck that you can by putting both the old and new drive in the same machine, your bottleneck will become the speed of the old drive. Per StorageReview's performance benchmarks (http://www.storagereview.com/Testbed4Compare.sr - see the "Maximum Transfer Rate - Read" benchmark), the best performance you can get out of a SATA drive today is 127 MB/s. Dells won't have Velociraptors in them - at 1 TB, you're looking at, at best, the E7K1000 @ 121 MB/s.

121 MB/s * 60 sec / min * 60 min / hour = 435.6 GB / hour read. The PCI Express bus is fast enough to write off to another drive at the same speed (presuming drives have improved in the next 3 years to write at 121 MB/s sustained), so you're going to be able to hope, best-case, for 2.5 hours for a full TB copy.

It sounds like you're trying to do a migration to new machines? You might be better off making an image when the time comes and using something like Ghost Solution Suite to lay down the image to as many machines as you can plug into the network at one time via multicast, then using a centralized migration tool to move documents and settings over from the old machines to the new ones, if that's an option in your environment. By using that sort of method, you'll still likely be > 1 hour per machine, but if you can work on them in the dozens at a time, you'll still finish pretty quickly.

Data being transferred will be in a limited number of specific folders. This data isn't the hard drive image itself but specific files. could be any combination of office documents, engineering drawings, movies, media etc...

 

[Modelworks]

Provided all the drives are identical . I would use a diagnostic card + ic clip on the drives controller and copy the data as fast as the drive could read it. Ic clip takes about 3 seconds to attach.
They actually make jigs that fit the drive where you can slide the drive into the slot, pogo pins contact the relevant area on the pc board and copies data straight off the controller as fast as it can read.

The other way is to use a drive duplicator. It uses dedicated hardware to copy from one drive to another, or one stored image to a drive, without bus limitations getting in the way. The drives have to be identical. It is pretty much plug and play. I have seen models that can do 8 drives at once.


Looks like a 19 drive model is out
http://www.ics-iq.com/

6GB /minute to 19 drives simultaneously.