whats the best way to transfer 10+ gigs of data across a LAN?

benjamit

Senior member
Dec 22, 2000
775
0
0
i've had a hard time doing this

actually the data can vary from a couple of gigs to a little less than 30 gigs

zips, cdrs are all too small

hdds are doable but slow

all the nics are 10/100 and for some reason win2k says that the lan is 100 even though its cable modem

the router is a smc barricaid, older version with the lights on top
 

benjamit

Senior member
Dec 22, 2000
775
0
0
the amount of data bogs the systems down and sometimes not reliable for large files

looking for something faster and more reliable
 

skriefal

Golden Member
Apr 10, 2000
1,424
3
81
Copying a 10GB file over a 100mbit network will probably require you to wait about 24 minutes or so -- assuming an average transfer speed of 70mbit/sec. Upgrading to gigabit ethernet would decrease the time to perhaps 8 minutes, assuming your hard drives are quick enough.

Tranferring these files over 100mbit really shouldn't bog your system down, unless the driver for your network card is poorly written.
 

Hoober

Diamond Member
Feb 9, 2001
4,387
37
91
Yeah, Ethernet's the fastest way to move things between two computers on an ethernet.... how does it bog the systems down? Eats up the processor? Eats up the harddrives? The network should be able to handle it like a champ.
 

benjamit

Senior member
Dec 22, 2000
775
0
0
it's not really one large file

there actually many smaller files in many folders, located in folders, located in folders, etc.

win 2k says something like the path is too long or something like that to transfer
 

benjamit

Senior member
Dec 22, 2000
775
0
0
on one machine t-bird 900 the cpu is busy

on the other 2(p3 866) the machine seems ok

is it worth it to get gigabit nics just for this reason?

 

Goosemaster

Lifer
Apr 10, 2001
48,775
3
81
Itseems to me that you need to check the drivers for the NICs. If you think about it, the machines are either sending or recieving about 6-9Megabytes per second max. If your pcs cannot handle that something is definitely amiss.

I transfered an entire harddrives' contents (28.3GB) from one pc to another this weeked over my SMC barricade router(Backing up before format) and It went quickly, about half an hour, and the sytems were fine. I was even using Terminal Server to the machine receiving data the whole time(runing apps etc..heY! i WAS BOREED OKAY!?!). anyways it was quick and painless.

As for getting Gigabit eithernet...don't

basically don't buy "heavy duty speaker wire when you have a $200 stereo and $50 speakers"



Serisuly, do you know what GIGABIT costs?You'll have to forgive me for being adament on the subject, but it has become a common occurence for people to want GIGABIT without regarding the costs, and without thinking about the preformance in day to day activites.

I already reponded to a question like ths today ( look me up) but for the sake of saving a few of anand's search dollars...here we go


1.
You can buy GIGBIT switches with a GIGABIT port, however, it has 1 GIGABIT PORT, which is the uplink port. all of the other ports are 10/100. WHy spend the money? Well if you have all of the clients trying to access the server at the same time, that can equate to a theoretical-total request of a bout 40MBps or 80MBps in dublex mode. Of course, Duplex mode only handles a maximum of 200Mps or about 20MB/s max(Probably about 15 or so in real life) Anyways, I assume you see the bottleneck to the server. THe uplink aids in this providing a fast connection. Of course, you need a hell of a fast server(a GOOD one basically) to get the job done

Cost: about $150 for a kit.(Linksys, Netgear i think, others)


2.
You can also opt for a TRUE Gigabit switch that has all 10/100/1000 ports, such as a Netgear. WHat is the problem here? WEll, for starters, THe pci bus on a regular dektop machine will limit the bandwidth of any card to about 20MB/s. Second, you would need to have workstations capaple of supporting 66mhz 64-bit pci cards to take advantage of the speed? What then is the problem you ask?

SPEED..SERVERSIDE.
These Machines of your are going to be greedy little bastards. they are going to want gigabit speeds from the server, and in some cases, all at the same time. This equates to about an estimated maxium of about 80MBps x4 or 8(basically number of root clients) so If you want GIGABIT speeds, your server will have to be able to put out(heh) at a sustained maximum of about 320MB/s.(damn:Q) For that you can forget IDE or even plain SCSI. SCSI RAID is a must for such an application. A seagate 36lp shoots out about 70MB/s sustained so you get the picture....

Cost: Netgear 8port 10/100/1oo0 swtch retails for $1500
Take into account cabling, NICs $70-100 a piece for 1000T varieties)

This of course is a worst case scenario...but I just wanted to scare you out of making a "foolish;)" purchase.


I remeber the good 'ole days of setting HPs on fire in Cirsuit City to show how fammable they were, in addition to their other drawbacks...;)
 

skriefal

Golden Member
Apr 10, 2000
1,424
3
81


<< on one machine t-bird 900 the cpu is busy >>



Crappy network card or poorly written drivers, most likely. What type (brand & model) of network card is in that machine?
 

benjamit

Senior member
Dec 22, 2000
775
0
0
well i wasn't really going to get gigabit nic but i did look up the cost of a nic and it wasn't too bad, not that long ago a 10/100 costed that much

but having to get new cables and router, well that's clearly not worth it for my old setups

i think it was the way the data was setup

i did transfer 10 gigs once, fairly fast and reliable

it seems transfering, doing anything with large multimedia files is tricky
 

benjamit

Senior member
Dec 22, 2000
775
0
0
the specific models i don't have in front of me but one is a netgear, one linksys, one smc, one old isa 10 intel, one 3com

i haven't tried to transfer with all of these nics but the first time it worked for me was from the old intel to the netgear

when i tried it last it was from the linksys to the netgear
 

err

Platinum Member
Oct 11, 1999
2,121
0
76
Is this going to be a one - timer transfer or a scheduled transfer ? If this is a one time, just do it once disk to disk or something if you wish to maintain data integrity.

I also don't understand your statement on: "the amount of data bogs the systems down and sometimes not reliable for large files"

How slow can this be ? we normally transfer 40GB of database over 100mbps network and it is transfering fine.. less than two hours.

The only problem that I can see is that you have LOTS of files and folders on your Win box. This does bog performance and Windows would normally crap out on that huge of a files. I once transfered 120 mb of data with over 75,000 files and folders and yes it took a good part of the day. The only solution that I would suggest is script the transfer and zip the files before you transfer it.

Hope this helps.

eRr
 

Need4Speed

Diamond Member
Dec 27, 1999
5,383
0
0
I would setup a FTP server on the LAN. In general it uses much less overhead when transfering files than copy/pase etc. and makes more efficient use of the bandwidth. I copy large files on a regular basis from PC to PC on my LAN, and that seems to be to most effective way to do it without extra investments.

-P
 

Woodchuck2000

Golden Member
Jan 20, 2002
1,632
1
0


<< whats the best way to transfer 10+ gigs of data across a LAN? >>



Overnight :)

Well said goosemaster, I fully agree. Everyone wants to hike up their trousers and say in a manly voice: "I'm running Gigabit, what'cha got punk?" It's like when 100mbit first appeared - an irrational need appeared for new technology before any use became apparent.
 

benjamit

Senior member
Dec 22, 2000
775
0
0
thanks guys

i'll try ftp, file sharing in windows does not seem too reliable to me in several networks i've used

as for the files being in multiple folders i think that is really the problem in bogging the system


 

nightowl

Golden Member
Oct 12, 2000
1,935
0
0
In yous AMD system that is bogging down check the transfer mode of the hard drive. I had a similar problem with one of my systems and the problem was VIA's crappy chipset. It would not use UDMA, only PIO. I ended up having to install a special IDE driver from them to somewhat fix the problem, as the higher CPU usage is than my Intel based chipset systems. I was using a 3Com 905B NIC too.

While it may be a poor NIC driver, I would also check the transfer mode of your hard drive too.
 

benjamit

Senior member
Dec 22, 2000
775
0
0
yes i'll check for dma

i just installed xp pro on it recently and i saw that it had dma if possible for both controllers by default

i'll have to try to transfer again after my move
 

Gunbuster

Diamond Member
Oct 9, 1999
6,852
23
81
<<win 2k says something like the path is too long or something like that to transfer >>

Your using FAT32 right? Switch to NTFS and say goodbye to FAT's spazzy extended ASCI file name problems
 

benjamit

Senior member
Dec 22, 2000
775
0
0
yes i'm using ntfs in all my drives, even zips

funny mac os, recently fixed with X, does not recognize ntfs
 

n0cmonkey

Elite Member
Jun 10, 2001
42,936
1
0


<< yes i'm using ntfs in all my drives, even zipsfunny mac os, recently fixed with X, does not recognize ntfs >>



Does not recognize it how? (and by fixed, do you mean it works in X or was that a joke at Apple's expense? :p)
 

jteef

Golden Member
Feb 20, 2001
1,355
0
76
my solution for gigabit was 2 gigabit nic's and a crossover cable. I haven't tried it yet since i'm broke...

jt
 

benjamit

Senior member
Dec 22, 2000
775
0
0
prior to os x macs did not recognize ntfs only fat

the volume can not be read by the os

 

n0cmonkey

Elite Member
Jun 10, 2001
42,936
1
0


<< prior to os x macs did not recognize ntfs only fatthe volume can not be read by the os >>



Thank you for the explanation. I dont use Windows, but if I ever did its good to know I could possibly fix it with my Mac :D