ACK prioritization and Traffic Shaping results.

[Balthazar]

I recently installed a default RedHat 9.0 install (selecting the "Server" option in setup) with only these packages:
Webmin
Samba 3.0
Shorewall

And some other misc. update.
And setup traffic shaping on the box, everything I needed was installed by default, and I took the machine over to my brother-in-laws house to test it out (would have liked to do it here but I couldnt at the time).

After about 15 minutes of tweaking ONE setting we ran some "real world" tests to see if there was a noticeable difference.
Here are the results.
His line is a 1.5Mbit down 768K up DSL line from Speakeasy. So the result werent AS dramatic on his setup (I ran a few really rough tests on my home machine and showed some even more drastic improvements being as I have 2Mbit down 128K up cable from Charter) but they give you a good idea. Also I lost alot less upload bandwidth than he did for some reason.

Anyway, I'm just putting it up here because I've been looking into it for some time now and all the stuff I've read online (the HOWTO's and other such things) make it out to be some nightmare task involving recompiled kernels and a bunch of nasty tasks in order to get it working.

In reality it took me 45 minutes to install redhat, 15 minutes to tweak the shaper settings, and about 15 minutes to set the box up as a gateway. Everything I needed came with RedHat 9.0 (and I understand any 2.3+ kernel distro thats not custom or cut-back should have the neccisary tools).

So if anyone wants a copy of the script I used to set it up for themselves (yay no more 30K/Sec downloads on BitTorrent when people start downloading from me) just let me know.

-Balthazar-

 

[masul0100]

I have also been researching traffic shaping for some time. I would be intersted to try your script out, would it be possible to post it on your site? Thanks for the info.

Masul

 

[Fuzznuts]

yeah i would like a look too to compare to my setup i use wondershaper and then use then set rules within shorewall to mark the packets. would be interesting to see how you did it.

 

[Balthazar]

Ok there you go I updated that page with a copy of the script.

I like this method mostly just because its ALL I have to do. Just set the values, run the script (or add it to ifup) and viola. To tweak it I just change the values, re-run the script, viola.

Thats about as easy as it gets.

Fuzznuts: What kind of results do you get with Wondershaper?

 

[Fuzznuts]

i find it works very well. i cap my downstream to 1000kbits and my up to 230kbits then mark packets as need using the tcrules and tos files within shorewall.
i can upload at 27k(max is 30k unshaped) while downloading at 120k (max is 122 -125 unshaped). only thing i havemt quite got is playing online games usually when a dls is flat out. i get pings in the 30 - 60 range but packetloss makes the game unplayable so need to work on that.

i use a lot of ssh to remote sites and give ssh priority over large dls, the ssh sessions are as responsive as if i were local so no complaints there either

also if i run a q3 server i cap at 900k and shape then i can dl at 80 ish k whilst playing on the q3 server with 3 client pings in the 30 - 40 range. so i find it works very well. i find lower you set you intial value the more succesful shaping is however when nothing else is going on its too slow. so i am constantly changing the values.

looking at the script on yours it is loks very much like the wondershaper cbq script i use the htb version. the version you have has some rules ripped from it here is my wondershaper script that i simply moved to /etc/shorewall/tcstart

as you can see though they are VERY similar

# Wonder Shaper
# please read the README before filling out these values
#
# Set the following values to somewhat less than your actual download
# and uplink speed. In kilobits. Also set the device that is to be shaped.

DOWNLINK=1000
UPLINK=230
DEV=eth1

# low priority OUTGOING traffic - you can leave this blank if you want
# low priority source netmasks
NOPRIOHOSTSRC=

# low priority destination netmasks
NOPRIOHOSTDST=

# low priority source ports
NOPRIOPORTSRC=

# low priority destination ports
NOPRIOPORTDST=


# Now remove the following two lines

#echo Please read the documentation in 'README' first
#exit

if [ "$1" = "status" ]
then
tc -s qdisc ls dev $DEV
tc -s class ls dev $DEV
exit
fi


# clean existing down- and uplink qdiscs, hide errors
tc qdisc del dev $DEV root 2> /dev/null > /dev/null
tc qdisc del dev $DEV ingress 2> /dev/null > /dev/null

if [ "$1" = "stop" ]
then
exit
fi


###### uplink

# install root HTB, point default traffic to 1:20:

tc qdisc add dev $DEV root handle 1: htb default 20

# shape everything at $UPLINK speed - this prevents huge queues in your
# DSL modem which destroy latency:

tc class add dev $DEV parent 1: classid 1:1 htb rate ${UPLINK}kbit burst 6k

# high prio class 1:10:

tc class add dev $DEV parent 1:1 classid 1:10 htb rate ${UPLINK}kbit \
burst 6k prio 1

# bulk & default class 1:20 - gets slightly less traffic,
# and a lower priority:

tc class add dev $DEV parent 1:1 classid 1:20 htb rate $[9*$UPLINK/10]kbit \
burst 6k prio 2

tc class add dev $DEV parent 1:1 classid 1:30 htb rate $[8*$UPLINK/10]kbit \
burst 6k prio 2

# all get Stochastic Fairness:
tc qdisc add dev $DEV parent 1:10 handle 10: sfq perturb 10
tc qdisc add dev $DEV parent 1:20 handle 20: sfq perturb 10
tc qdisc add dev $DEV parent 1:30 handle 30: sfq perturb 10

# TOS Minimum Delay (ssh, NOT scp) in 1:10:

tc filter add dev $DEV parent 1:0 protocol ip prio 10 u32 \
match ip tos 0x10 0xff flowid 1:10

# ICMP (ip protocol 1) in the interactive class 1:10 so we
# can do measurements & impress our friends:
tc filter add dev $DEV parent 1:0 protocol ip prio 10 u32 \
match ip protocol 1 0xff flowid 1:10

# To speed up downloads while an upload is going on, put ACK packets in
# the interactive class:

tc filter add dev $DEV parent 1: protocol ip prio 10 u32 \
match ip protocol 6 0xff \
match u8 0x05 0x0f at 0 \
match u16 0x0000 0xffc0 at 2 \
match u8 0x10 0xff at 33 \
flowid 1:10

# rest is 'non-interactive' ie 'bulk' and ends up in 1:20

# some traffic however suffers a worse fate
for a in $NOPRIOPORTDST
do
tc filter add dev $DEV parent 1: protocol ip prio 14 u32 \
match ip dport $a 0xffff flowid 1:30
done

for a in $NOPRIOPORTSRC
do
tc filter add dev $DEV parent 1: protocol ip prio 15 u32 \
match ip sport $a 0xffff flowid 1:30
done

for a in $NOPRIOHOSTSRC
do
tc filter add dev $DEV parent 1: protocol ip prio 16 u32 \
match ip src $a flowid 1:30
done

for a in $NOPRIOHOSTDST
do
tc filter add dev $DEV parent 1: protocol ip prio 17 u32 \
match ip dst $a flowid 1:30
done

# rest is 'non-interactive' ie 'bulk' and ends up in 1:20

tc filter add dev $DEV parent 1: protocol ip prio 18 u32 \
match ip dst 0.0.0.0/0 flowid 1:20


########## downlink #############
# slow downloads down to somewhat less than the real speed to prevent
# queuing at our ISP. Tune to see how high you can set it.
# ISPs tend to have *huge* queues to make sure big downloads are fast
#
# attach ingress policer:

tc qdisc add dev $DEV handle ffff: ingress

# filter *everything* to it (0.0.0.0/0), drop everything that's
# coming in too fast:

tc filter add dev $DEV parent ffff: protocol ip prio 50 u32 match ip src \
0.0.0.0/0 police rate ${DOWNLINK}kbit burst 10k drop flowid :1

 

[cleverhandle]

Nice work. I should point out, however...

Originally posted by: Balthazar
...(yay no more 30K/Sec downloads on BitTorrent when people start downloading from me)...

BT has a command-line option to limit your upload. Just set that below the point where ACK's start getting squeezed out and you should be fine. On my connection, for example, if I don't limit the upload I'll do about 25K down and 35K up. If I limit the upload to 30K, I do 80K+ down and 30K up. Clearly, that last 5K up is the ACK killer.

But yours is the more thorough solution, no doubt.

 

[Balthazar]

The problem with BitTorrent is every version I've ever used that allows you to limit traffic does a pretty poor job of it. Both the command line and experimental GUI based version, I've watched them with traffic monitors and they might get withina few K/Sec of the limits you set, MAYBE. And to get any kind of decent download from BT I usually have to either get lucky or max out my upload.

 

[Balthazar]

Fuzznuts: I saw a script like that in my many attempts and finding out more about traffic shaping, but from what I can tell the version I use is a newer simpler method (or maybe older-simpler?). And my understanding was more systems come with cbq than htb so, I kinda chose it for that reason.

Funny thing is, I was writing my own shaper script and was about halfway done with it when I came across this shaper script and was like "wow....the first half is VERY similar, and I cant really think of any other way to do the rest of it thats not going to be VERY similar to this script." so I just gave up and used the shaper script I found online, why reinvent the wheel right?

In either case I think I am going to see if tc accepts limits defined by bits and not kbits because I get to a point where literally buping the uplink up 1 kbit is like flipping a switch where I go from no effect, to VERY noticeable effect. so I want to try to fine tune it a bit more (and I cant imagine why they would let you secify kbit if it isnt capable of accepting bits too).

Either way I am VERY happy with the results, and theres no need for me to cap the downlink because downloads I have no problem with, its the upload killing my download that burns me.

 

[gaidin123]

I've found that using HTB is definitely more accurate than CBQ so it should be the preferred method of shaping if it is available to you. The newer 2.4.x kernels already come patched with HTB so at most you would just have to get a patched tc/iproute2 tool.

In my case when both the upstream and downstream of my DSL line is being heavily used, the pings definitely rise up a bit for gaming. If you back off on your upstream or downstream (or both) just a little when you want to play games, the pings drop back to normal for me. I've tested the line many times before and I think the increased pings just have something to do with both the up and down being almost saturated. If you don't, you should dump the UDP packets into the highest priority queue along with your ACKs to get the best performance out of your game.

A little trick I use to make the traffic shaping script more flexible is to give the script variables on the command line like this:
./trafficscipt 600. In the script I have a line saying UPLINK=$1. It's a lot simpler than making a bunch of copies of the script with slightly different traffic limits. You can easily cron this to change the limits based on time of day too...

I'm glad you're happy with the results and found that it really wasn't that hard. Traffic shaping has made sharing a net connection an easy experience for years now for me and it's a *lot* easier to set up than it used to be.

Edit: Oh, and do a man tc and you'll see that it can take bytes/s, kb/s, mb/s, KB/s, MB/s for bandwidth rates.

 

[Balthazar]

Good to know it accepts bytes, I plan on setting it up here at home in a few hours.
As for accuracy, I'm not sure what you mean about HTB being more accurate?
In my case I have a very simple need, I just want my download not to die when I upload.
I don't see cbq faltering at all in that regard, but maybe I'll give HTB a try sometime too and see if there is any difference.

 

[Balthazar]

I updated the page to show the results on my home cable modem, 2.5Mbit down 128K up cable line.

Comparatively its alot more dramatic a difference.

 

[VTEC01EX]

I'm trying to load balance between machines - when both are downloading, each gets roughly 50% of the pipe. Would something like this help me?

 

[Balthazar]

In the sense that you can limit the download each machin gets. By changing the DOWNLINK to 50% of your actual downstream bandwidth you could ensure that both get even amounts of bandwidth.

Not really load balancing though, load balancing would be having two machines and when one slows down a secondary server starts taking the new connections. In which case, no this wont do that.

I'm not sure I quite get what your going for.

 

[VTEC01EX]

That's basically what I'm looking for. Ideally, it would be alright if Machine A was generating no traffic that Machine B could have 100% of the pipe... but if it means a 50/50 split all the time, regardless of utilization, I guess that would be alright too. I might just have to give this a shot and see how it does!