BufferBloat - what and how

[DigDog]

i'm not sure whether to post this in general discussions, gaming, or networking .. because reasons.

i've stumbled upon a speedtest which has a feature i had not seen before: measuring buffer bloat.

http://www.dslreports.com/speedtest

and so i did a bit of reading, mostly due to getting a big, red circle with an F in it, as my test result.

now, i do quite a bit of online gaming, FPS mostly, don't laugh, it's my hobby, not my job. And i've come to suspect my connection is not as responsive as others - this just seems to confirm it.

Now, i have no idea what buffer bloat is, and while i can deduce it from the name, i'm not sure how to tackle the issue, IF it can be done, and what i've been reading about it paints a dramatic picture.



First of all, Virgin. The first page that gave me a description of Bb and an how to, was some techie's blog where he describes getting banned from the virgin forums for complaining "in technical detail" about the buffer bloat; also how Bb is linked to the SuperHub (VirginMedia router) and to the algorithms they use, how there are different solutions in form of code (which allegedly is easy enough to implement), and finally Virgin seems to be the company most affected by this.

I know that Virgin have a very high contention ration, and in fact my connection is far superior at night - not faster, but quicker to respond.


Now .. do any of you gentlemen know what this bufferbloat thing is? How to resolve it?
The aforementioned code, how is it installed? Flashing the router with a custom bios? Can this be done under Windows?

Do i need to buy a new router?

I'm kinda short these days so that solution would be best left for later, but i'm still interested for a potential fix later on.

and that's it for now. any suggestions?

 

[JackMDS]

"Bufferbloat is undesirable latency caused by routers and cable/DSL modems buffering more data than necessary. It occurs at any bottleneck in a network: the most common place is the connection between your router/modem and your ISP".

See more - http://www.dslreports.com/faq/17883

 

[Gryz]

Now .. do any of you gentlemen know what this bufferbloat thing is?

Yes.

How to resolve it?

You can't.
Buffer bloat can be a problem on any router in the path between you and the computer (server) you are communicating with. Lets split these into three cases:

1) The router at your home
2) Your ISPs routers
3) Any other router owned by another ISP

Home routers usually don't have a lot of buffers. Its unlikely bufferbloat is a problem there. And you must be aware: bufferbloat is only a problem when links are congested. Easiest solution: don't download torrent and other huge data sets while you are playing a game. I bet you already knew that.

If your ISPs routers have too much buffers (configured), tough luck. You're not gonna get them change their network configuration. Note, having a lot of buffers helps (a bit) with preventing packet loss. For the average user, packetloss is a bigger problem than slightly increased latency. Therefor some ISPs will keep configuring their routers with more buffers. You can't really blame them. If you don't like it, find another ISP. One that configures less buffers on their routers. Or uses routers with better QOS.

If the bufferbloat is happening somewhere else in the network, at other ISPs, you have even less chance to fix the issue. You'll need to change your ISP and find an ISP that has different peering and different transit-provider(s) to reach your target server. Note, ISPs do sometimes change their contacts with other ISPs. And thus paths from an ISP to particular destinations in the Internet can change any day, without notice. Imho there's not much point in changing ISPs just so you hope their routes change a little.

But again, the biggest chance of problems with high delays are in the last hop, between your ISP and your home router. Don't overuse the capacity of your link, and you should have no problems.

 

[DigDog]

why do the people at dslreports claim it's "an easy fix to implement and freely available code" then; sounds like it's an inevitable aspect of networking.
router flashing was also mentioned, but they were talking about Linux. (i'm afraid my knowledge of all things netowrking is really limited)

 

[Gryz]

Where do they claim that ?
Remember, most people know nothing about networking, except messing with the WiFi settings of their routers.

Remember, buffer bloat can only be a problem at links that are congested. Not temporarily/burst overload, but constantly overloaded. When you bursty traffic for a few hundreds of milliseconds, then having extra buffers actually helps. Only when your link is constantly overloaded (for minutes or longer) then large buffers are actually decrimental.

Also remember that your router only has control over traffic that goes from your router to your ISP. Not the other way around. So any fix for buffer bloat only helps when you have congested your local upstream traffic. Now, that might happen in some cases. But usually it is the downstream direction that is overloaded.

Did a little reading:
https://en.wikipedia.org/wiki/CoDel#Description
"The algorithm is independently computed at each hop".
There you have it. Those algorithms need to be deployed on every router on the path. But you have only control over your own router. Ergo, if you change the behaviour/software of your own router, that will only have limited effect.


Maybe I'm completely wrong here. But then somebody please explain to me how a router can have impact on queuing strategies, qos and buffer bloat on another router further away in the path.

 

[DigDog]

what you say makes sense. i was following mainly on this page here:

http://the-edge.blogspot.co.uk/2015/03/virgin-media-fixing-epidemic-of.html

that paints the problem as being a very different one. (its not a wall of text)

 

[Gryz]

No, it's the same thing.

Bufferbloat happens when 1) router vendors put too much buffering in their routers, and 2) links get congested, which means there will be buffering.

The simplest way to fix that is to put fewer buffers in routers.

Another way is to implement better queuing algorithms. The guy in your link seems to want to fix bufferbloat via new algorithms called Codel and Fair Queuing Codel. It seems those algorithms (or at least FQ Codel) treats different types of traffic differently. Give priority to things like DNS, TCP Syn packets (opening a connection), voip, maybe gaming, etc. Bulk data transfer (torrents, ftp, video) can still be buffered a lot more, because that 1) is good for achieving high bandwidth, and 2) on those applications delay doesn't matter. Note, it was Van Jacobsen who came up with Codel. He knows what he's talking about. I'm sure (fq)Codel is a good idea. The question is: in which scenarios does it help, and where does it not help ?

The problem of bufferbloat has been known for 10-15 years now. It's not new. I assume that most large provider-router vendors (cisco, Nokia, Juniper) have dealt with the problem ages ago. Or at least allow their customers to configure their buffers and queues any way they want. Maybe small home-router vendors still put in too much buffering (in relation to the bandwidth of the links attached to those routers).

If you're willing, you can always install new software on your home-router. Nobody is stopping you. And check for yourself to see if it helps.

 

[DigDog]

eh, i dont really know how to do that, but i quick google should help me out. i'll report if i get any improvement.

 

[mxnerd]

Explaining And Fixing Bufferbloat

https://www.youtube.com/watch?v=XB3w09VZn7g

===

How to reduce gaming lag, while streaming video?

http://forums.anandtech.com/showthread.php?t=2477830


R7000 & QoS - recommended settings for games/media stream

http://www.dd-wrt.com/phpBB2/viewtopic.php?t=263940

===

Get a router that can configure QoS manually.

 

[Mark R]

Bufferbloat occurs when data tries to flow from a fast link into a slower, or congested link. For example, when it tries to flow from your ISP's network onto your DSL line. Routers, rather than sending an alert back to the transmitter telling them to slow down, may just buffer the data, and transmit it when possible.

A little bit of buffering is good, and avoids temporary stalls and slowdowns. A lot of buffering adds latency - if you have a 10 Mbps downlink, but there is 120 kB of buffer between the sending server and you, then this can add potentially 100 ms of latency, which may be problematic for real-time applications.

BT did a huge amount of research into this about 15 years ago when they introduced ADSL. They solved this with a technique called IP profiling. Every time your modem resyncs, the actual speed of your line is linked to your IP address, and all the routers in BT's network will throttle traffic to your line rate. So, if you are downloading from an FTP server, as soon as the traffic comes into BT's network, the first router it hits throttles the traffic, and sends a "slow down" message (source quench) back to the server. Then because the data has already been throttled, it has a clear path straight through the BT core network, and the DSL line, so there is no buffering, and therefore no bufferbloat on downloads.

It can also occur on your uploads, when data moves from your fast LAN to the slow DSL/cable connection. Cheap home routers are often configured to buffer hugely, as this gives better scores in bandwidth tests, but it leads to bufferbloat on upload.

 

[Gryz]

Bufferbloat occurs when data tries to flow from a fast link into a slower, or congested link. For example, when it tries to flow from your ISP's network onto your DSL line.

To be precise, what you describe is generic congestion.

Lots of those articles and algorithms are about queuing-strategies (and drop strategies) when buffering because of congestion. Those problems, and proposed solutions have existed for decades. Priority queuing, custom queuing, Random Early Drop (RED), Weighted Fair Queuing (WFQ). And now CoDel and FQ-Codel.

Buffer bloat is the name for the phenomenon that in the late nineties, and early previous decade, router vendors (and some switch vendors too) we putting more and more buffer-memory in their routers. The rule of thumb, as I heard it in the late nineties was: "have as much bufferspace so you can buffer the maximum normal RTT of your traffic". Normal maximum RTT was considered 200ms world-wide. Now if "the Internet" would buffer traffic for 200ms, that wouldn't be so bad. But if all routers do that, and you got half a dozen (6) congested routers on your path, then you can see 1.2 sec of extra RTT.

Buffer bloat was about this trend of adding buffer memory. I think the "fight against buffer bloat" is just a campaign to make router vendors aware of the problem.

Having more buffers also has good properties. Esp for high-bandwidth traffic. If you just put less bufferspace in routers, your network will not start behaving better. You will see lower RTT, but more drops. What you need is better queuing algorithms.

Routers, rather than sending an alert back to the transmitter telling them to slow down, may just buffer the data, and transmit it when possible.

And what kind of alert would that be ?
In the world of TCP/IP, we don't have explicit congestion notifications. There are no such packets to tell transmitters to slow down. (Reason: sending such packets will only add to the congestion).
In TCP congestion avoidance is done by carefully watching the acknowledgements you are getting back from the receiver. If you don't receive expected acks, that indicates that your packets were dropped. Which is good. TCP's behaviour depends on packets sometimes getting dropped. Also the speed/frequency/rythm at which you receive acks indicates how your packets were being forwarded.

BT did a huge amount of research into this about 15 years ago when they introduced ADSL. They solved this with a technique called IP profiling.

Never heard of that.
IP profiling, in the context of DSL, usually means something else. In stead of configuring each DSL-connection differently, ISPs group them in profiles. And each DSL-connection is then configured according to the profile that suits them best (according to the quality of the copper-wires, etc).

Every time your modem resyncs, the actual speed of your line is linked to your IP address, and all the routers in BT's network will throttle traffic to your line rate.

I sincerely doubt that. That wouldn't scale. Do you have a link where this is explained ? I'm curious.

So, if you are downloading from an FTP server, as soon as the traffic comes into BT's network, the first router it hits throttles the traffic, and sends a "slow down" message (source quench) back to the server.

ICMP source quench has not been used since the eighties.
See: https://tools.ietf.org/html/rfc6633

Then because the data has already been throttled, it has a clear path straight through the BT core network, and the DSL line, so there is no buffering, and therefore no bufferbloat on downloads.

Again, I can't believe BT's network works that way.
Tomorrow I'll ask a friend about this. (He happens to work for the vendor of BT's core-routers. And he happens to work on QoS. What a coincedence ).

It can also occur on your uploads, when data moves from your fast LAN to the slow DSL/cable connection. Cheap home routers are often configured to buffer hugely, as this gives better scores in bandwidth tests, but it leads to bufferbloat on upload.

URL please. You might be right. But I would imagine that home routers won't have much buffers, because hardware costs money. And lots of buffers don't improve download speeds (nor upload speeds) much. Large buffers make sure that there are fewer drops when traffic is bursty. This can help with DNS-requests, web access, etc.

 

[richb-hanover]

@Mark R and @Gryz - thanks for the interesting discussion. As a member of the bufferbloat team for a number of years, I'd like to add some additional info...

TL;DR:

• Bufferbloat is undesirable latency that occurs at the bottleneck (usually your connection to your ISP).
• Unless your router at home is configured to fight it, you'll see lag/latency.
• That latency occurs when up/downloading, but also during normal web browsing: web pages today average a 1mbyte download
• (Sadly) Nearly all network equipment suffers from bufferbloat, and it all needs to be updated. The fact that you're reading this note means that you may be in a position to help yourself. The vendors have not yet seen that it would be a competitive advantage...

Citations

• www.bufferbloat.net - Research results collected by the bufferbloat team
• What to do about Bufferbloat - some hints for alleviating bufferbloat
• Tests for Bufferbloat - a list of testing resources
• Bufferbloat and the Ski Shop - a layman's introduction to bufferbloat
  

And some detailed comments:

Originally Posted by Mark R View Post
Bufferbloat occurs when data tries to flow from a fast link into a slower, or congested link. For example, when it tries to flow from your ISP's network onto your DSL line.

To be precise, what you describe is generic congestion.

Lots of those articles and algorithms are about queuing-strategies (and drop strategies) when buffering because of congestion. Those problems, and proposed solutions have existed for decades. Priority queuing, custom queuing, Random Early Drop (RED), Weighted Fair Queuing (WFQ). And now CoDel and FQ-Codel.

And up 'til CoDel and FQ-Codel, all those solutions were wildly unreliable (or even harmful) and were not often used in real world applications, or at the edge of the network, where they would help the most.

Buffer bloat is the name for the phenomenon that in the late nineties, and early previous decade, router vendors (and some switch vendors too) we putting more and more buffer-memory in their routers. The rule of thumb, as I heard it in the late nineties was: "have as much bufferspace so you can buffer the maximum normal RTT of your traffic". Normal maximum RTT was considered 200ms world-wide.

This is still a good rule. But the normal RTT today is under a couple dozen msec, so the buffering at a router needs to be smaller.

Now if "the Internet" would buffer traffic for 200ms, that wouldn't be so bad. But if all routers do that, and you got half a dozen (6) congested routers on your path, then you can see 1.2 sec of extra RTT.

This is not quite correct. There will only be one slowest (bottleneck) link any path, which will be the congestion point. The other router's buffers will tend not to be full, since they're getting rid of the data more quickly

Buffer bloat was about this trend of adding buffer memory. I think the "fight against buffer bloat" is just a campaign to make router vendors aware of the problem.

It is the Bufferbloat team's sincere desire and hope that router vendors would become aware that this, of all the aspects of their router design, is the major reason that most home router installations stink out loud. But we have not seen good results in this regard.

Having more buffers also has good properties. Esp for high-bandwidth traffic. If you just put less bufferspace in routers, your network will not start behaving better. You will see lower RTT, but more drops. What you need is better queuing algorithms.

Routers, rather than sending an alert back to the transmitter telling them to slow down, may just buffer the data, and transmit it when possible.

And what kind of alert would that be ?
In the world of TCP/IP, we don't have explicit congestion notifications. There are no such packets to tell transmitters to slow down. (Reason: sending such packets will only add to the congestion).

Actually, TCP has an Explicit Congestion Notification (ECN) bit reserved in the TCP header, so it doesn't consume bandwidth. fq_codel sets it when it detects congestion to inform the sender to slow down.

In TCP congestion avoidance is done by carefully watching the acknowledgements you are getting back from the receiver. If you don't receive expected acks, that indicates that your packets were dropped. Which is good. TCP's behaviour depends on packets sometimes getting dropped. Also the speed/frequency/rythm at which you receive acks indicates how your packets were being forwarded.

Exactly. fq_codel measures how long packets dwell in a router's queue. When packets have been in queue "too long", it marks them with ECN, or drops a random percentage, to force senders of large flows of data to slow down. This naturally gives priority to small flows that don't develop a queue. For details, see the Bufferbloat and the Ski Shop blog, or read the full documentation at the bufferbloat.net site.

BT did a huge amount of research into this about 15 years ago when they introduced ADSL. They solved this with a technique called IP profiling.

Never heard of that.
IP profiling, in the context of DSL, usually means something else. In stead of configuring each DSL-connection differently, ISPs group them in profiles. And each DSL-connection is then configured according to the profile that suits them best (according to the quality of the copper-wires, etc).

Every time your modem resyncs, the actual speed of your line is linked to your IP address, and all the routers in BT's network will throttle traffic to your line rate.

I sincerely doubt that. That wouldn't scale. Do you have a link where this is explained ? I'm curious.

So, if you are downloading from an FTP server, as soon as the traffic comes into BT's network, the first router it hits throttles the traffic, and sends a "slow down" message (source quench) back to the server.

ICMP source quench has not been used since the eighties.
See: https://tools.ietf.org/html/rfc6633

Then because the data has already been throttled, it has a clear path straight through the BT core network, and the DSL line, so there is no buffering, and therefore no bufferbloat on downloads.

Again, I can't believe BT's network works that way.
Tomorrow I'll ask a friend about this. (He happens to work for the vendor of BT's core-routers. And he happens to work on QoS. What a coincedence ).

I don't know how BT's network works - I would be intrigued to hear the outcome.

It can also occur on your uploads, when data moves from your fast LAN to the slow DSL/cable connection. Cheap home routers are often configured to buffer hugely, as this gives better scores in bandwidth tests, but it leads to bufferbloat on upload.

URL please. You might be right. But I would imagine that home routers won't have much buffers, because hardware costs money. And lots of buffers don't improve download speeds (nor upload speeds) much. Large buffers make sure that there are fewer drops when traffic is bursty. This can help with DNS-requests, web access, etc.

This is correct - home routers have megabytes of buffer available, and they're "not afraid to use them." You can measure this yourself by starting a ping and then running any speed test. If the ping times get high during the transfers, then there's bufferbloat. See the citations above.

 

[Gryz]

Hi Rich, thanks for your comments. It seems you agree with most I've said. That's always nice to hear. I got one question for you.

In normal use, the congestion is always/mostly in the downstream direction to your home. Now your home router only has control over traffic leaving your home router, going towards your ISP. How can a home-router do QoS on incoming traffic ? Imho only your ISP's router can do QoS (or any fancy queuing) on congested traffic. Can you explain me how that is supposed to work ?

The only thing I can think of is that a home router can start dropping acks intentionally, to try influence the remote host to slow down on individual tcp flows. But that would require keeping state per flow. I can see that working in small home routers. But not in core routers that have millions or more flows going through them. As the bufferbloat problem is the sum of large buffers in all routers, I see a problem fixing bufferbloat everywhere. (Assuming you keep state per flow, that is).

And then there is the problem that non-tcp problem doesn't adjust itd bandwidth ....

Anyway, I'm mostly curious to learn how you influence traffic coming to you, in stead of leaving you. TIA.

 

[DigDog]

Unless your router at home is configured to fight it, you'll see lag/latency.

ok, but how?

 

[Gryz]

So richb-hanover comes here, tells us fq-codel is awesome and solves all your congestion and bufferbloat problems.

Then I ask one simple question:
"queuing strategies have only impact on outgoing traffic. how do you control incoming traffic, when most of the congestion (and thus delay) is on the incoming (downstream) side ?".

No answer. I've asked this question a few times before. To several different people. Never got an answer. Not a theoretical one. And not an answer that explains how things actually work in today's routers, software and algorithms. Not even an attempt.

This makes me believe that all congestion algorithms today still work only on outgoing traffic. And thus, to improve most people's home situation, their ISP must enable QoS (or fq_codel) on their router connected to their home-routers. (The so-called BNG or RAS routers). And there is very little you can do yourself to improve congestion.

If somebody knows more, please tell us.

 

[DigDog]

if the issue was only congestion, then you would see an improvement outside of peak hours, right?

 

[Gryz]

if the issue was only congestion, then you would see an improvement outside of peak hours, right?

Yes.
But if you're talking about buffer bloat, that also certainly only happens when there is congestion. So suppose you see increased RTTs, you can not say: "that is because of buffer bloat, and not congestion, because there is no congestion".

 

[Makaveli]

I've used this test plenty.

To reduce buffer bloat you need to enable QOS on your router.

Another trick for this specific site is to limit download speed so its equal to your connection speed. So if you have 50 Mbps downstream set that as the cap.

I generally run with QOS off on my router so I just limit the speed in the test.

Here is the test left on default

And now with the test capped to my download speed.

 

[DigDog]

ok, i did that and, slight improvement; capping the test speed to just 50Mb (55Mb max) gives me .. a D instead of an F. from what i see, it could be that the test doesnt allow me to cap UP speed, and i can see the Bb indicator turn red only during the UP. i imagine the vote is a median between up & down.

so .. i dont understand .. why would the router, or switch, need to buffer based on MY speed. the fiberoptic line surely doesn't cap at 50Mbs. might this be due to the traffic shaping / speed limiting that the ISP imposes on my connection?

so, if i ask the switch to give me more data OR even simple more bandwidth, than 50Mbs, then the switch slows me down? And this is what causes Bb ?

in other words, if i use QoS to cap my speed to lower than what is set at the switch (Virgin cables go up to, i think, 200Mbs, you need to pay for the extra speed so they limit your connection *somewhere*, i assume at the switch), THEN i will have a faster, mroe responsive internet?
seems really dumb to have this kind of stuff going on.

 

[Makaveli]

What is your connection speed down and up?

What router are you using?

Can you post the grading part of the test?

Example.

 

[DigDog]

i'm on Virgin Media 50Mb fiber optic. (this same line can go up to 200Mb afaik, depends on how much you want to spend)

the result of the test while doing nothing else, uncapped:

capped at 45Mb:

capped at 20Mb, for shits n giggles:

i'm using a Virgin Super Hub 2ac, now in modem mode (no WiFi at all, only 1 ethernet). the firmware seems to be the latest.

 

[Makaveli]

Here is a review

http://www.expertreviews.co.uk/networks/wireless-routers/1402804/virgin-media-superhub-2ac-review

"Finally, the Advanced Settings menu lets you limit wireless speeds to improve compatibility and range, set up guest networks (two per band with their own passwords and security settings), enable port forwarding and configure DHCP settings. However, there are no Dynamic DNS settings, no QoS settings and no parental controls."

Based on this you have no QOS.

The only other option as suggested above would be to replace this router with something alittle more advanced.

Since that is a modem/router you can just set that to modem mode and add a better router behind it.

 

[VirtualLarry]

http://www.newegg.com/Product/Product.aspx?item=N82E16833320208

We are running on a 300MBps by 20MBps cable connection and I usually get on average 352MBps by 25MBps speed when not using QOS setup. However I do use it because the router gets rid of basically ALL our "Buffer Bloat" completely so even though the overall speed drops down to usually 285MBps by 22MBps because it cleans up all the buffer bloat our connection is much cleaner and in real world use is actually much snappier, pages load instantly.
My connection went from a D (Buffer Bloat) C (Line Quality) A (speed) to now its A, A+, A using the DSLReports speed test.