GiG-E Problem

[Hoober]

I'm stumped on this one and was hoping somebody might have a new idea...

I've got a dual P-III 1.4Ghz XEON server on Gig-E, 2GB RAM, with U160 10k rpm drives that won't throughput more than 340Mbps up or down. It's on a Cisco 3550 switch using one of the two built in gigabyte ports. I've tried two different onboards NICs on the server (different manufacturers) and three different client machines all using Gig-E. The patch cables are CAT-6.

With the client computer connected to the other Gig uplink, I can get 340Mbps transfer rates.

With the client computer connected across campus I can get about 60Mbps through a 10/100 switch. What bothers me, though, is that I can't get any where near 1gbps even on the same backplane.

Any ideas?

 

[spidey07]

That is completely normal.

Wintel architecture is simply too slow for GigE. One think to check is that it is running on the fastest bus possible like 66Mhz/64 bit.

Check out your processor utilization and interrupts during the tests. They'll be very high.

So just face it. You have a slow a$$ server.

 

[Hoober]

C'mon, Spidey, that's not what I wanted to hear.

It's on a 64bit bus and there really isn't much processor utilization at all.

So there isn't any way to speed it up? I mean, that's 30%!

 

[spidey07]

Even super-duper 16 way sun boxes have a hard time getting over 60%.

speed of hard drives comes into play as well. Search around for some articles on the net and maybe some independant testing.

I know the syskonnect gig card is supposed to be real good.

 

[spidey07]

http://www.veritest.com/clients/reports/intel/pro1000t.pdf

quick one (don't know the vailidity of it).

Lots of info out there though.

 

[spidey07]

Oh, and get a faster server and loose the wintel.

jab, jab.

 

[Hoober]

But I'm a wintel fanboy

So you think Sun, huh? Problem is, the front end on this server only likes Windows.

 

[Hoober]

So what you're basically saying is that my Broadcom NetXtreme Gigabit Ethernet card just isn't up to the task?

 

[spidey07]

No, I'm just messing with ya.

I haven't been keeping up with the different gig cards. But bus performance, hard drive performance, processor, etc all matter when trying to squeeze every last drop out of gig ethernet. gig E is just so darn fast the computer literally can't keep up.

You numbers seem about right though.

 

[Hoober]

I'll look into some benchmarking.

Windows is telling me that the disk access isn't that great when moving data over the gig link, so I don't know.

Thanks for your input, Spidey. I appreciate it.

 

[cmetz]

Hoober, what are you actually doing to get said transfer rate?

On two P4 1.3 / i820 / RDRAM boxes, running Red Hat Linux 7.2 with send and receive buffer sizes increased to 256k and netperf TCP, I can get about 650Mb/s from Intel Pro/1000MTs and about 700Mb/s from 3Com 3c996BT. That's network-only TCP performance. If I was actually, say, transferring a file, I'd be very lucky to get half that. If I were running Windows, I'd be very lucky to get half that again.

If for example you're doing a FTP between systems, the results you are getting are not outside the realm of what I'd expect.

Also, think about this for a moment: suppose your hard drives have a peak sequential read/write throughput of 40MB/s - not out of whack with reality. Multiply by 8B/b and you get 320Mb/s. Now that's awfully close to your 340Mb/s. If you're doing file transfers, sounds like you're bottlenecking on disk, which is not at all unexpected.

In general, though, think of gigabit to a machine as greater-than-100Mb/s. 340Mb/s is still quite good! Beats 60Mb/s, doesn't it?

As for Suns, yes, Suns do have better architectures especially for I/O. But last time I checked, the hard disk drives are still commodity SCSI U160 7200RPM or 10kRPM drives, and you can RAID them for some extra throughput but there's limits on how much faster that all can be. And meanwhile, the price tag difference for a non-crippled Sun (SMCC) box is pretty hefty.

 

[Hoober]

CMetz,

I was using QCheck to get that theoretical throughput of 300-350mbps. I updated the driver on that Broadcom and bumped it up another ~100mbps using Qcheck, but that doesn't actually transfer a file.

Transfering a file via Windows (not FTP) I'm getting ~ 250-300mbps sustained or around 45 seconds to transfer a 1GB flat file. Not too shabby, I suppose. I was under the delusion that 1Gbps was 1Gbps. So, I was rather frustrated when I wasn't seeing those kinds of speeds. And I became even more frustrated when I went through and troubleshot everything I could think of.

According to performance monitoring software, the RAID array isn't taking that much of a hit during file transfer. This server is used to image machines via PXE through a DOS application over the network and I'm seeing speeds of ~80MB per minute (don't ask me why the designers of said application used that as a measurement) over a fibre uplink from the Cisco switch the server is attached to down to other Cisco switches and into a 10/100 client on a 100Mbps full duplex connection.

Now if I use the same program to image a machine directly from the other Gig uplink port on that Cisco switch I see transfer rates of ~150-180MB per minute.

This server used to sit on a 10/100 switch and because the network won't handle multcusting (we still have old Fiddi ring gear) we made the decision to move it a gig uplink so that we were able to image more than five computers at once (necessary for computer labs). On the 100mbps uplink, we hit the 12.5MBps cap with 5 computers. Now, imaging an entire lab, the total throughput of the server over the LAN still doesn't break 12.5MBps.

Now, I'm not a network guru by any sort. I'm a DBA with a little sys admin thrown in for fun, so I don't really understand all the network topology and hierarchy stuff. I had assumed that once the server was moved to GiG-E, we would be able to image more than five computers at once. And I assumed we would be able to break 12.5MBps on a single switch because said switches are uplinked to the various MDF's via gigabit fiber. So, theoretically, each port on said switch would be able to achieve 12.5MBps and then said switch would be able to achieve (up to) 125.0MBps to the server because the server was also on gigabit.

Can either of you correct me on that theory?

 

[ScottMac]

You might poke around to see if you can configure that server to use giant/jumbo frames (make sure the switch is ALSO set for giant / jumbo frames). Crank up your MSS/ MTU sizes to max ... so whatever you can can to pump the data in the largest possible chunk sizes.

Also, how far away was the other endpoint, and what was the setup?

Good Luck

Scott

 

[Hoober]

Scott,

I'm not sure the fibre distance to the other endpoint.

From what I understand about the topology of the network, and I apologize if I butcher some terminology here, but there are two central junctions on campus. One for the north side and one for the south side. The buildings holding these junctions are roughly five hundred yards apart. Now, in each of the other buildings there is generally an MDF and an IDF. The MDF has a fibre uplink to the closest campus "junction," and the junctions are joined in whatever manner so that it makes one LAN. I hope that makes sense.

Anyway, to answer your question, the server uplinks via GIG-E to a Cisco 3550 (I'm pretty sure that's the model number, my brain isn't remembering little things so well at the moment, though) which is uplinked via fibre to the North campus "junction." The client(s) that I've been using to test are in a separate building connected to that main north campus junction. They use 100mbit connections to a cisco switch in the MDF of the building which is uplinked via fibre to that north campus junction. So it's a fibre backbone, I guess.

 

[ScottMac]

You'd be likely to see better numbers if you ran the test on the LAN (for a "Maximum" reference). Other equipment and traffic between the two locations might also be factors.

Just for reference, check out SysKonnect's site, there's a comparison of copper Gig NICs done by University of (Somplace, Iowa, Nebraska...someplace flat ....) that shows performance numbers using different MTUs, frame sizes, computer settings, etc.

In a "stock" PC with "stock" IP settings, they got a couple hundred meg; after some tweaks, the same NICs in the same computers were nearly full bandwidth.

Gigabit Ethernet is definately "Tune Up Territory" if you want max performance. You are likely to need to tweak every component from endpoint to endpoint.

Good Luck

Scott

 

[Hoober]

Thanks, Scott. I'll take a look.

I upgraded to the latest drivers this morning and coaxed another 100MBps out of the connection.

 

[Hoober]

Scott, do you have the link for that gig study?

I'm looking at SysKonnect's site and can't find anything that looks like that. I am, however, learning a whole log about Gig-E.

 

[Hoober]

Does an MTU size need to be configured on the switch? And how much of a difference does a bigger MTU size make?

 

[bigpow]

i'd do these first on phy-layer:
1. clean all fibres & connectors with proper tools
2. check optical power-meter if possible
3. if you don't have any powermeter or fibrescope, try with new set of fibres
4. what GBIC/SFP are you using? 850nm SX, 1310nm LX, ZX, etc?
5. make sure the fibre type match the GBIC wavelength
6. if all fails, then move up to the higher layer - up to app-layer

 

[Hoober]

Bigpow,

It's copper, not fiber. And the NIC is going directly into an uplink port in the switch, there's no GBIC card in between.