Question Slow Download but Fast Upload After Enabling Jumbo Frames on a VM

[Collider]

Spent hours trying to resolve this, so definitely needing some advice. Seeing very slow internet download speed only, but fast internet upload and fast local network download & upload on guest OS / VM.

I have a 10 Gbe setup with ConnectX-2 card and a T1700G-28TQ switch. All is well on the host / bare metal OS and I'm able to successfully get 9.6 Gb / s in iPerf. Internet up / down performance is perfect as well. Jumbo Frames are enabled on the switch, physical network adapter and Hyper-V virtual switch. Router has default MTU 1500.

Problem is on a Guest VM after enabling Jumbo Frames on the virtual network adapter inside the VM (setting MTU 9014)
- internet download speed drops to low 5-10 Mbps
- internet upload is fine 500+ Mbps
- local network connections are fine, 9.6 Gb / s in iPerf

What could be causing this? Any advice would be highly appreciated.

 

[mxnerd]

If it's a VM, why not add another virtual adapter that connects to the internet without setting jumbo frame? Just don't set a gateway for virtual adapter that connects to your LAN.

The Case Against Jumbo Frames (for internet)

 

[Collider]

If it's a VM, why not add another virtual adapter that connects to the internet without setting jumbo frame?

Interesting idea. I have not considered this.

I assume then I would need to somehow set 1 adapter to default as internet and the other to be used for LAN connections - any idea how to do that?

 

[mxnerd]

Interesting idea. I have not considered this.

I assume then I would need to somehow set 1 adapter to default as internet and the other to be used for LAN connections - any idea how to do that?

See update above.

 

[Collider]

See update above.

Thanks. That worked partially - I'm able to force all internet traffic via 1 of the adapters, however same adapter will be utilized if I copy a file from a network folder. Not sure how to force 2nd adapter to be used for local network.

 

[mxnerd]

Give LAN adapter a higher priority with a lower metric number.

How to change the priority order of network adapters on Windows 10

If you use multiple network adapters on your Windows 10 PC, in this guide, we'll show you the steps to prioritize each one in two different ways.

www.windowscentral.com

 

[Collider]

Give LAN adapter a higher priority with a lower metric number.

Much appreciated! I got it working

It is a very clever workaround that you suggested - it will actually be quite useful for web server VMs, or other scenarios where default MTU is needed and Jumbo Frames could be set up on 2nd adapter for backups/etc.

 

[VirtualLarry]

I liked that NANOG presentation on "why Jumbo Frames may be harmful". Very enlightening.

I know that there are a lot of "gotchas", and I have NOT implemented Jumbo frames on my local LAN, mostly because I run a "mixed" LAN, both NAS / storage, and internet access. I'm running 3rd-party firmware on the router, perhaps it can handle jumbo frames, but I haven't tried it yet.

Perhaps in the future, if I have a dedicated "storage / NAS" LAN, as well as an "Internet" LAN, I can enable Jumbo Frames on my "storage" LAN, but not the "Internet" LAN.

 

[Collider]

Question: is there any way to not set Default Gateway but still use DHCP?

The only way I could avoid setting Default Gateway was by changing TCP/IPv4 properties and setting static IP with empty Default Gateway field. Curious if there's another way to do that in Windows.

 

[mxnerd]

Question: is there any way to not set Default Gateway but still use DHCP?

My experience is that Windows Server's DHCP let you do that very easily.

But I could be very wrong since I never used any Linux DHCP server, and some routers with 3rd party firmwares (DDWRT, TOMATO variants, OpenWRT, etc) with DNSMASQ service probably will let you do that, but you need to use special commands and it's very obscure sometimes.

The problem is that it's hard to set gateway for some machines but not to set gateway for some on the same network. Yet again you can do this using Windows DHCP server's different scope options easily by setting each device's MAC address and IP pair.

 

[Collider]

My experience is that Windows Server's DHCP let you do that very easily.

But I could be very wrong since I never used any Linux DHCP server, and some routers with 3rd party firmwares (DDWRT, TOMATO variants, OpenWRT, etc) with DNSMASQ service probably will let you do that, but you need to use special commands and it's very obscure sometimes.

The problem is that it's hard to set gateway for some machines but not to set gateway for some on the same network. Yet again you can do this using Windows DHCP server's different scope options easily by setting each device's MAC address and IP pair.

I guess then it won't be possible in my case. I'm using Orbi as my router and DHCP and it doesn't appear to have that option.

 

[thecoolnessrune]

This likely was happening because you left your Router at 1500 MTU. Any segment attached at L2 needs to have MTU accomodating Jumbo frames, otherwise more than likely it's going to drop them. You would need to change the Router port configured for that segment to support jumbo frames, and from there, the router can fragment packets to get to the next hop. That being said, consumer routers aren't usually going to give you that option, even if you have gear that can, Jumbo Frames are generally not worth the trouble they cause.

 

[Collider]

This likely was happening because you left your Router at 1500 MTU. Any segment attached at L2 needs to have MTU accomodating Jumbo frames, otherwise more than likely it's going to drop them. You would need to change the Router port configured for that segment to support jumbo frames, and from there, the router can fragment packets to get to the next hop. That being said, consumer routers aren't usually going to give you that option, even if you have gear that can, Jumbo Frames are generally not worth the trouble they cause.

Thanks for the reply. I completely understand your point, but I don't think that is happening in my case because I have other non-VM machines with Jumbo Frame MTU enabled and they do not have this issue. The issue occurs only on VMs for some odd reason.

 

[thecoolnessrune]

Did you test that? You should be able to ping back and forth with large payloads with the do not fragment flag set and on any device in that L2 Domain you should be getting responses and no drops. If you can't do that, then something in that group is not configured to run Jumbo Frames. For instance, if I connect a printer to this theoretical network, and it's internet connected, with the Router side being configured for MTU 1500 and the printer at 1500, it will work fine all the time. Say it's also an MFC and can send scans to a computer on that network. That will probably work fine too, because the large packets are being sent by an MTU 1500 device, and being put through an MTU 9000 tunnel, which has more than enough room for it. On the same side, the internet won't have a problem with the printer because the printer is sending and receiving packets at the same size as the Router interface (1500), even if the middle area is a 9000 MTU. But if the computer with a 9000 MTU tries to submit actual large packets to the printer, or the internet, it doesn't work, because they're discarded.

The low upload, normal download speeds read like an MTU mismatch because that's exactly the kind of symptoms you would normally see. Because packets coming in from WAN -> LAN are limited by your WAN interface (1500 MTU), so it's max 1500 MTU all the way to your system. But if your computer tries to send large packet,s, they get dropped by LAN interface on the router because it's max MTU is 1500. The low upload speed is indicative of TCP Slow Start repeatedly getting engaged because of packet loss (I would surmise when the computer builds up to sending >1500 MTU size packets.)

 

[Collider]

My issue is actually reverse of that, uploads are normal, but downloads are slow if Jumbo is enabled on the virtual adapter inside the VM. Host's physical adapter and virtual switch both have Jumbo enabled without any such issues.

I don't have Jumbo frames enabled for entire network, only on switch ports for machines that need it (UnRaid, Hyper-v, workstation). My switch allows setting MTU by port, so my setup is: Router (default MTU) -> Switch (allows MTU by port, some default, some set to Jumbo) -> Hyper-V + Other machines + WIFI

I'm able to successfully ping between Jumbo enabled machines with -f -l 8972 without any packets fragmented. I'm using 9014 MTU on those, so 8972 is max (switch allows up to 9216).

Pining 8.8.8.8 -f from Jumbo enabled machines works fine as well. Obviously if I ping 8.8.8.8 -f -l 8972 it fails.

 

[thecoolnessrune]

What is your Hyper-V Version? And have you Captured a trace in Wireshark while downloading to look for things like TCP Re-transmissions or other indicators of an unhealthy link?

 

[Collider]

Hyper-V version is 10.0.17763.1

Wireshark is a good idea. I'll give it a go on the problem VM.