Jumbo Frames

[m1ldslide1]

The question: When are jumbo frames actually used?

I've never actually seen a jumbo frame register on an interface counter. I understand that support for jumbo frames has been written into the gigabit ethernet standard, but it's my understanding that they're only actually used in very specific applications. I here the term thrown around on these forums quite a bit and it seems like people think that their consumer-class switch is somehow reencapsulating their ethernet frames to combine them into these 9kB frames. I've even heard some poor souls suggesting "You can't get to gigabit speeds without jumbo frames!" which is a pretty fundamental misunderstanding of networking architecture, so those folks need not respond to this post.

From Cisco press:
"Jumbo frames- Many high data-transfer applications, such as network attached storage (NAS - not to be confused with network access server) support jumbo frames, which extend the MTU for ethernet frames from 1518Bytes up to 9216Bytes. Using jumbo frames improves data throughput for data-intensive applications such as network-attached storage."

So in other words, if you're not using an application that specifically encapsulates frames as jumbo, then you're never ever going to see a jumbo frame on your home or enterprise network. After all, TCP is only going to window up to 1518 bytes in a network with no congestion, and is usually less than that. NAS is new to me - the only time I ever heard of jumbo frames being used is with Fibre-channel networks, in other words, Storage Area Networks. I can verify that one of the networks I maintain is a very large metro-ring with over 250 individual sites and gigabit connectivity between them. There has never been a jumbo frame recorded on any gigabit interface, and these are enterprise class cisco switches that definately support jumbo frames.

It's not a huge deal overall, but I cringe every time I see someone talking about jumbo frames in their home network, or even worse making a purchase recommendation based on jumbo frame support.

Anybody have any input or quotes to add???

 

[spidey07]

Just that you're a little off base.

Jumbo frames increases throughput by sending more on the wire per layer2/3/4 header. You don't need any special application to take advantage of it as this is done at layer2.

this results in the folowing advantage:
1) less computational overhead for calculating and ack'ing TCP
2) More effiency because the header is a much smaler percentage of overall payload
3) In esoteric terms it allows you to use a higher percentage of the bandwidth because with larger frames the interframe gap that is mandatory for ethernet becomes a much smaler amount of the bandwidth actually being used to carry data.

All that is needed to take advantage of this is all end nodes to support it and the switch as well. It is frequently used on servers where maximum performance is needed and one wants to save CPU cycles. Once you hit a router it is no big deal because the router will fragment the packet or send an ICMP back to the sender that says "hey, lower your TCP max segment size (MSS) to 1460 (the standard for a 1500 byte MTU"

So TCP will indeed take advantage of a larger frame size regardless of the application being used. If one truly want the most performance, one will get and utilize jumbo frames. Frame size actually limits ethernets performance from being so small - one of the reasons why 100 megabit FDDI would run circles around ethernet was because of it very large frame sizes.

 

[m1ldslide1]

Thanks for the response. I understand how jumbo frames will increase throughput both by reducing L2/L3 header overhead and in a packets-per-second sense.

According to the Cisco quote I posted, you do in fact need an application to take advantage of this. I won't requote, but this is consistent with all that I've encountered.

The fact that ethernet encapsulation is done at layer 2 is irrelevant to the jumbo frame -- it's the payload that we're talking about here. There is no ICMP message that tells a host "go ahead and transmit jumbo" is there? I can look it up if need be. Seems like something you'd set in your application, or if you bought a fancy storage server something you'd set up locally. I guess you'd have to have two fancy storage servers though, so that they could send jumbos back and forth to each other, because my laptop NIC sure as hell doesn't support jumbo.

 

[spidey07]

TCP does it by taking the MTU (L2 length before adding L2 header and CRC) and subtracting L3 and L4 length.

no special app needed. We use jumbo frames on thousands of servers and saw a marked improment in performance, especially backups since this is what would benefit most - large/long data transfers.

It's how TCP works. The application isn't aware nor does it care what the frame or MSS is. The app just says "open connection and send this data", from there the stack and TCP/IP takes care of the turning it into packets.

In a perfect world the MTU of ethernet will be increased to 9 or 10K (the CRC looses it's effectiveness at around 12K). And I can see it being that standard here before too long. The 1518 byte frame size has been around since ethernet's inception and has been crippling it ever since.

Especially since 100 Gig is coming, 10 gig over copper right around the corner.

 

[m1ldslide1]

So then how did you implement this on your servers? Did you just plug them all together with jumbo-compliant switches and they all started transmitting in jumbo automatically? Do you have hosts with jumbo-compliant NICs that communicate with these servers, or is it only inter-server traffic?

These are things I must know.

 

[spidey07]

the servers have jumbo frames attached to switches that support jumbo frames.

the clients are normal 10/100.

so of course when the jumbo frame hits the router to be sent to the client it is fragmented or and ICMP is sent back to the sender to adjust MSS. this is called Path MTU Detection or PMTUD.

 

[m1ldslide1]

Sweet! Learn something every day. So the servers speak jumbo automatically... It's interesting to me because I don't work with any servers save for ACS and various appliances, like load balancers, PIX, content filters and so forth. Totally different group of folks handles the servers. Good thing too, because I hope to never learn windows...

 

[nweaver]

You handle those devices, and don't understand that talking frames = Layer2?

 

[FreshPrince]

all I gotta say is large packet switches work great in my environment

we have a lot of storage devices and they definitely help in data transfers.

 

[Madwand1]

Originally posted by: spidey07
We use jumbo frames on thousands of servers and saw a marked improment in performance, especially backups since this is what would benefit most - large/long data transfers.

I'd like to know what sort of rates you've gotten before & after in order to understand and position this better.

Going on a data overhead basis, I get a marginal difference between jumbo and standard frames (something like 0.4% vs 2.5%). On a packet count basis, of course the difference is huge -- 6X, so I can see this as being very important when you have a lot of cumulative traffic and the hardware / software has a problem keeping up with the number of packets.

I'm getting 110 MB/s synthetic transfer rates and up to 80 MB/s actual file transfer rates without jumbo frames, so it's hard for me to see much room for improvement with jumbo frames in my situation. Perhaps jumbo frames are simply over-rated in very small environments; perhaps I'm mistaken, and some of the missing file transfer bandwidth in my case can be attributed to jumbo frames. I'd like to know.

 

[spidey07]

It was about a 25% increase in throughput. but you'll have to talk to the server guys.

because as it stands now, the server is the bottleneck.

Try real world tests and see what you get.

 

[Madwand1]

Originally posted by: spidey07
Try real world tests and see what you get.

To be clear, some of these were real world tests from my perspective.

E.g. 158 image files in one directory (one shoot) totaling 1.25 GB. Max transfer rate around 78 MB/s.

E.g. 1 DVD ISO of 4.5 GB max around 65-70 MB/s.

E.g. 15724 image files of various sizes totalling 202 GB, 54 MB/s (1 hour). I'm guessing the file system + overhead from smaller files is the likely bottleneck here based on the other results.

 

[stimpyman77]

Originally posted by: spidey07
Just that you're a little off base.

Jumbo frames increases throughput by sending more on the wire per layer2/3/4 header. You don't need any special application to take advantage of it as this is done at layer2.

this results in the folowing advantage:
1) less computational overhead for calculating and ack'ing TCP
2) More effiency because the header is a much smaler percentage of overall payload
3) In esoteric terms it allows you to use a higher percentage of the bandwidth because with larger frames the interframe gap that is mandatory for ethernet becomes a much smaler amount of the bandwidth actually being used to carry data.

All that is needed to take advantage of this is all end nodes to support it and the switch as well. It is frequently used on servers where maximum performance is needed and one wants to save CPU cycles. Once you hit a router it is no big deal because the router will fragment the packet or send an ICMP back to the sender that says "hey, lower your TCP max segment size (MSS) to 1460 (the standard for a 1500 byte MTU"

So TCP will indeed take advantage of a larger frame size regardless of the application being used. If one truly want the most performance, one will get and utilize jumbo frames. Frame size actually limits ethernets performance from being so small - one of the reasons why 100 megabit FDDI would run circles around ethernet was because of it very large frame sizes.

Spidey, I had a question on some of this info.. Where you say all end nodes support it, does that mean that I can not have 10/100 devices plugged into the gigabit switch while there are gigabit clients on it? Are there any downfalls to doing that? I did not think it would be a problem, but would like the gospel from the master so to speak..

Thanks!

 

[spidey07]

stimpy,

you will probabably have problems. the switch will receive a jumbo frame on ingress (coming into the port) and then attempt to forward it out a port with a smaller MTU. Normally a switch will drop the frame.

but with TCP and PTMUD that I talked about earlier TCP could probably decied on a smaller MSS and hence the sender would drop it down. UDP and other protocols don't have this.

in other words, I wouldn't do it. never hurts to try though.

 

[m1ldslide1]

Originally posted by: nweaver
You handle those devices, and don't understand that talking frames = Layer2?

Are you talking to me? I doubt you read or understood my posts if you are. There was never any ambiguity about what layer framing occurs at, just at what causes your stinkin server to start transmitting jumbo frames.

 

[nweaver]

frames = layer 2

what level does your server start transmitting jumbo frames...level 2, where device drivers live (sorta)

 

[VisionxOrb]

Ive got a question, lets say you have 2 wired and one wireless client, the 2 wired are equiped with gigabit NICs ( supports jumbo frames ) and there connected to a gigaswitch which supports jumboframes ( adlegidly (sp?) its a gatway metal housed 8 port switch ) and that switch is connected to a linksys wrt54GC ( the compact one ) which handles my wireless client ( along with shareing my internet ). Could I then enable jumbo frames as the NICs would connect through the switch and the router would tell them to change there MTU when accessing the internet? ( and the wireless client )

 

[brentpresley]

Originally posted by: VisionxOrb
Ive got a question, lets say you have 2 wired and one wireless client, the 2 wired are equiped with gigabit NICs ( supports jumbo frames ) and there connected to a gigaswitch which supports jumboframes ( adlegidly (sp?) its a gatway metal housed 8 port switch ) and that switch is connected to a linksys wrt54GC ( the compact one ) which handles my wireless client ( along with shareing my internet ). Could I then enable jumbo frames as the NICs would connect through the switch and the router would tell them to change there MTU when accessing the internet? ( and the wireless client )

Great question. Anyone have an answer to this? I'm looking a similar setup.

 

[randal]

Originally posted by: VisionxOrb
Ive got a question, lets say you have 2 wired and one wireless client, the 2 wired are equiped with gigabit NICs ( supports jumbo frames ) and there connected to a gigaswitch which supports jumboframes ( adlegidly (sp?) its a gatway metal housed 8 port switch ) and that switch is connected to a linksys wrt54GC ( the compact one ) which handles my wireless client ( along with shareing my internet ). Could I then enable jumbo frames as the NICs would connect through the switch and the router would tell them to change there MTU when accessing the internet? ( and the wireless client )

If the interface on the WRT54GC that is plugged into the Gateway switch supports Jumbo-frames, then sure.

Anything in the broadcast domain of the Jumbo-frame switch has to support jumbo-frames. Layer 3 devices (routers) that break up broadcast domains will receive the jumbo frame on one interface, fragment it, re-encapsulate it at a lower MTU and then transmit the packet on another interface.

 

[spidey07]

Originally posted by: VisionxOrb
Ive got a question, lets say you have 2 wired and one wireless client, the 2 wired are equiped with gigabit NICs ( supports jumbo frames ) and there connected to a gigaswitch which supports jumboframes ( adlegidly (sp?) its a gatway metal housed 8 port switch ) and that switch is connected to a linksys wrt54GC ( the compact one ) which handles my wireless client ( along with shareing my internet ). Could I then enable jumbo frames as the NICs would connect through the switch and the router would tell them to change there MTU when accessing the internet? ( and the wireless client )

You could probably rely on PMTUD to have it work. This should be fine for TCP applications. UDP or others may run into problems.

That goes back to where in the initial 3 way tcp handshake both hosts send their MSS and they choose the lower one.

actually now that I think about it PMTUD isn't at play here since you aren't crossing a L3 boundary between wired and wireless and are staying layer2 - from here it is the MSS stuff done in the 3 way handshake.

hope this helps, you can give it a shot and post back.