doing a network/OS/samba benchmark *new benchmark done scroll down*

[Mucman]

I am transfering 18.7GB worth of data from my temporary storage on my win98 box to my FreeBSD fileserver... I am curious as to what I should expect from the transfer, and then I will look at how it performed in the experiment...

Files FROM : Win98, 768MB RAM, 27G 7200rpm ATA33, PIII 1000
medium : 100MBit full duplex connection
Files TO : FreeBSD 4.6, 32M RAM, 60G 7200rpm + Promise 66 card, P150 running samba

I started the transfer at 9:17:10PM my local time, when do you think it "should finish"? A `top` reading on the BSD machine shows smbd using
50% cpu usage and and the RAM usage isn't moving.

It's already 9:50 and it looks like and I think I am 1/3 through...

 

[Mucman]

Results are in!

Transfer took 2 hours and 10 minutes. Which means I was getting around 19Mbit/s. Pretty slow huh? Copying from a Debina linux box with samba,
to the same win98 machine took 2 hours and 40 minutes! Any ideas what's up with that? That almost makes me think the wiring isn't the culprit!
How slow is SMB?

 

[Nothinman]

SMB should only be slightly slower than FTP which itself should be as close to wire speed as you'll get.

Bear in mind a lot of things come into play, especially with large data sets like that. Hard disk speed is usually the main culprit, make sure one of the disks isn't slowing you down.

 

[gaidin123]

If you are transferring 18.7GB in a single file (ie a disk image or uncompressed video) it will transfer much, much faster than if you have 18.7 GB of 1KB files. The more files you have, the longer it will take over a single, large file transfer.

Still, 20Mbps is pretty slow. Even with your slower FreeBSD box I'd think that you would get better transfer rates.

Gaidin

 

[Mucman]

gaidin123 - The 18.7G consists of 2500 files... I tried transferring a 700MB avi and it went at 20Mb/s as well...

I am now moving 25G of data from my FreeBSD machine to the file server (so BSD to BSD) and the smbd daemon on
the file server is topping out at 87% CPU usage! When transferring files from the win98 machine, it never went above
50%.

Gonna do some serious testing when I get home...

 

[Nothinman]

Is that CPU time mostly process user time or interrupt/system time? It could be a shoddy NIC or NIC driver.

 

[Mucman]

Nothinman - I am not sure I know how to check that... I am just looking at my `top` results :

98212 root 56 0 5260K 1928K RUN 221:21 82.81% 82.81% smbd

 

[Nothinman]

Doesn't the second row from top have USER SYSTEM INTERRUPT, etc percentages?

I'm currently copying ~5G in 21 files from my main box (Dual 1.2G Athlon w/ SCSI160 drives running Win2K) to my Alpha test box (600Mhz SCSI-2 drives running Linux using ext3) and ~200M in the drive started slowing me down cause the box only has 256M of memory (and it's got WindowMaker and xchat loaded) and it ran out of disk cache. But when it doesn't have to hit the drive smbd takes ~30% cpu (most of which is system time which is general kernel so it could be interrupts, packet processing, misc ext3 or scsi, etc) and it gets up to ~8MB/s (according to iptraf)

 

[Mucman]

ahh!

CPU states: 15.4% user, 0.0% nice, 42.8% system, 37.3% interrupt, 4.5% idle

so NIC interrupts are causing a huge delay then?

btw, nice machines!

 

[Nothinman]

ahh!

CPU states: 15.4% user, 0.0% nice, 42.8% system, 37.3% interrupt, 4.5% idle

so NIC interrupts are causing a huge delay then?

Could be, I can't imagine it causing the poor performance you're getting though. What kind of NIC is it? I would lean more towards the hard drive, is there something like 'vmstat 2' in FreeBSD?

vmstat in Linux looks like (the 2 just means print ever 2 seconds)
The bolded part is the block in and block out columns (which are 0 cause the box is idle right now) but when the disk is constatly working it displays how many blocks (I believe they're 1K each on Linux) are going from or two the disks, respectively.

$ vmstat 2
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
0 0 0 3496 294088 15408 44896 0 0 1 5 4 7 1 1 98
0 0 0 3496 294088 15408 44896 0 0 0 0 1033 99 1 1 98

 

[n0cmonkey]

That ATA33 disk isnt helping at all.

EDIT: If you have a cheap nic it is going to use more cpu time. The best nic for OpenBSD is supposed to be fxp (Intel), I figure it would be the same for FreeBSD.

 

[Mucman]

PIII machine has a 3com 3c905B, the P150 has a 3com 3c905C

Nothinman - FreeBSD has vmstat... I will play with it when I get home.

noc... I've realized that Intel nics are the most consistent out there... I learned this while dealing with many servers at work

Who knew file transfering could be sound enteraining, and educational

 

[n0cmonkey]

Originally posted by: Mucman
PIII machine has a 3com 3c905B, the P150 has a 3com 3c905C

Nothinman - FreeBSD has vmstat... I will play with it when I get home.

noc... I've realized that Intel nics are the most consistent out there... I learned this while dealing with many servers at work

Who knew file transfering could be sound enteraining, and educational

There's your problem... 3com :Q

I think the ATA 33 is one of the biggest culprits for the slow transfers. You guys almost have me wanting to try this stuff

 

[Nothinman]

Nothinman - FreeBSD has vmstat... I will play with it when I get home.

I know the command exists, just like OS X has vm_stat, but I don't know if the output is anything similar.

noc... I've realized that Intel nics are the most consistent out there... I learned this while dealing with many servers at work

Yea Intel seem to be the most common lately, although I find DE500s (well, all tulip based cards) to work just as well and they're a good bit cheaper since they're older.

I think the ATA 33 is one of the biggest culprits for the slow transfers. You guys almost have me wanting to try this stuff

Yea I'm almost willing to bet $$ it's the hard drive slowing him down.

Doing this stuff is interesting, until you see how long it's going to take to tranfer that much data and you're like "Why am I doing this again?" =)

 

[n0cmonkey]

I havent seen Linksys cards less than $15 (Im not looking all that hard and prefer to buy local ), and I can get Intel cards for just a bit more. Ill buy the Intel. Of course, I only have 1 Intel card right now, and a hand a half full of linksys tulip based cards on my home network

 

[Mucman]

n0cmonkey - cmon, you know you want to try it . I am going to do some transfers from the ATA66 HD on my machine and see how it goes... Too bad I don't have any Intel nics on me to try ... I have a Netgear, which I believe uses the tulip drivers...

Nothingman - Here is the output of vmstat on my FreeBSD machine... I have no idea what it means so I will be reading some man pages
tonight . I am going to try playing with some Samba settings too... but I will do that after I exhaust all the hardware combos.

procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr ad4 md0 in sy cs us sy id
0 2 0 11336 4976 5 0 0 0 15 12 0 0 266 75 10 1 3 96
0 2 0 11336 4976 2 0 0 0 0 0 0 0 231 7 5 0 2 98

btw, I am going to do these tests with a 700M avi now, instead of 18Gigs of mp3s

 

[n0cmonkey]

Originally posted by: Mucman
n0cmonkey - cmon, you know you want to try it .

You mean introduce SMB to my Window-less network at home?! :Q

I am going to do some transfers from the ATA66 HD on my machine and see how it goes... Too bad I don't have any Intel nics on me to try ... I have a Netgear, which I believe uses the tulip drivers...

Tulip driver based cards are fast, just fairly cpu intensive. I doubt Netgear nics use tulip, but I have never bought/used one, so I cant be sure. I was thinking they used RealTek chipsets...

 

[Mucman]

You mean introduce SMB to my Window-less network at home?!

I gotta keep the roomies happy ... plus I am going to learn openldap and convert the machine into a fully functionaly PDC

I have a Netgear FA310TX which I believe is tulip, while the FA311 and FA312 cards are RealTek... but don't quote me on it though I think
I have a spare DLink card lying around too... I have several Linksys cards but they are 10Mbit ISA for my legacy machines

 

[Mucman]

OK, here is the result of `vmstat 2` when I started the tranfer of a 700MB avi from an ata66 HD on the win98 machine to the P150

procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr ad4 md0 in sy cs us sy id
1 2 0 14908 4284 5 0 0 0 15 12 0 0 266 75 10 1 3 96
1 2 0 14908 3992 3 0 0 0 542 2288 40 0 868 713 85 2 78 20
1 2 0 14908 3208 2 0 0 0 600 2284 43 0 910 802 96 2 83 15
1 2 0 14908 4440 3 0 0 0 598 1328 41 0 948 793 94 3 82 14
1 2 0 14908 3648 2 0 0 4 600 506 48 0 933 784 94 3 81 16
1 2 0 14908 2984 2 0 0 1 584 502 46 0 915 761 91 2 81 17
1 6 0 15652 4060 25 0 2 0 636 760 49 0 925 802 103 1 83 16
0 2 0 15304 3272 58 0 7 0 671 1401 60 0 884 826 108 3 82 15

The last part of it was :

procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr ad4 md0 in sy cs us sy id
1 2 0 14192 3288 2 0 0 0 648 505 19 0 926 865 103 3 81 16
1 2 0 13796 2712 3 0 0 0 584 512 18 0 877 788 89 6 75 20
0 2 0 13796 3928 2 0 0 0 616 768 19 0 912 818 92 4 81 16
1 2 0 13796 3032 2 0 0 0 616 505 19 0 885 816 93 4 82 14
1 2 0 13796 4312 3 0 0 0 608 768 19 0 880 821 98 4 81 15
1 2 0 13796 3416 2 0 0 0 620 505 21 0 893 836 93 2 85 13
1 2 0 13796 4568 3 0 0 0 624 768 19 0 888 832 94 4 80 16
1 2 0 13428 3668 2 0 0 0 624 512 19 0 893 829 95 3 80 18
1 2 0 13428 2776 2 0 0 0 616 505 20 0 886 817 93 4 77 19
1 2 0 13428 3992 3 0 0 0 614 766 20 0 882 821 98 3 80 17
0 2 0 13448 4560 7 0 1 0 185 256 7 0 430 275 36 2 25 73
0 2 0 13448 4560 3 0 0 0 0 0 0 0 232 13 6 0 1 99
0 2 0 13816 4592 2 0 0 0 4 0 2 0 234 18 7 0 1 99
0 2 0 13816 4564 2 0 0 0 6 0 2 0 235 8 6 0 2 98
0 2 0 13816 4564 3 0 0 0 0 0 0 0 232 15 6 0 1 99

I have "socket options = TCP_NODELAY"

 

[Nothinman]

Have you done the obvious and watched for the hard drive light to go solid?

The formatting sucks and I don't have FreeBSD handy to look at the man pages for that.

 

[Mucman]

Don't worry about it Nothinman ... I just got sidetracked... I almost got the machine working as a PDC and openldap is working as it should. So I set this stuff aside while I am on a roll with the other stuff

so many toys, so little time!

 

[manly]

I doubt the ATA33 interface is a problem; no 27GB IDE drive would max out that interface (or come close IIRC). The important thing is that the disk is running in Ultra DMA (ATA Mode 2). You can use a Winblows utility called hdtach to quickly test disk read throughput and CPU utilization.

Having said that, it does sound like your SMB throughput is significantly slower than the disk (main bottleneck) is capable of. On Linux, you can use hdparm to do a quick disk throughput test (and to query the drive's DMA mode). I'm sure there's some similar functionality on *BSD.

 

[n0cmonkey]

Originally posted by: manly
I doubt the ATA33 interface is a problem; no 27GB IDE drive would max out that interface (or come close IIRC). The important thing is that the disk is running in Ultra DMA (ATA Mode 2). You can use a Winblows utility called hdtach to quickly test disk read throughput and CPU utilization.

Having said that, it does sound like your SMB throughput is significantly slower than the disk (main bottleneck) is capable of. On Linux, you can use hdparm to do a quick disk throughput test (and to query the drive's DMA mode). I'm sure there's some similar functionality on *BSD.

dmesg to find out what DMA mode its using, bonnie is a great benchmark utility.

 

[Mucman]

PIII 1Gig : ad0: 39266MB <IC35L040AVER07-0> [79780/16/63] at ata0-master UDMA66
P150 : ad4: 57259MB <MAXTOR 6L060L3> [116336/16/63] at ata2-master UDMA66

I think manly is right though... 100Mbit/s = 12.5MB/s which even a slow HD can keep up with.

Once I get LDAP working I will put on a FTP program so I can try transfers there, that way
we can tell if it is the protocol...

 

[Sunner]

Just to add a little personal experience with 3Com NIC's to this discussion.

I use only 3Com NIC's for my boxes, and I've reached just under 100 Mbit without any problems under both Linux, OpenBSD, and FreeBSD, as well as Window 2000.
Though it usually takes a whole lot more CPU power to transfer from the Win2K box.