WHY i can't beat 100MB/sec with bonded NICs?

[NGbnd0]

Hi

i'm trying to connect two PCs (each one has two Intel-Gbit-NICs) direct with two "bonded" cables ... but i can't find my mistake - please, can someone help me?

modprobe bonding mode=0 miimon=500 downdelay=500 updelay=500
ifconfig bond0 192.168.100.1 up
ifenslave bond0 eth2 eth3

# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)

Bonding Mode: load balancing (round-robin)
MII Status: up
MII Polling Interval (ms): 500
Up Delay (ms): 500
Down Delay (ms): 500

Slave Interface: eth2
MII Status: up
Link Failure Count: 1
Permanent HW addr: **:**:**:**:**:71

Slave Interface: eth3
MII Status: up
Link Failure Count: 0
Permanent HW addr: **:**:**:**:**:70

# ifconfig
bond0     Link encap:Ethernet  HWaddr **:**:**:**:**:71  
          inet addr:192.168.100.1

but iperf measured only a Bandwidth of 99.2 MBytes/sec :-(

helpless
Daniel

 

[drebo]

Bonded NICs don't load balance, they link balance. That means that each connection (based on source/destination MAC or IP, depending on configuration) will go across one link or another. That means that total aggregate bandwidth will be double, but that the speed of any one connection will never be more than the speed of a single link.

 

[Pheran]

Bonded NICs don't load balance, they link balance. That means that each connection (based on source/destination MAC or IP, depending on configuration) will go across one link or another. That means that total aggregate bandwidth will be double, but that the speed of any one connection will never be more than the speed of a single link.

For the Linux bonding driver in round robin mode (which is what the OP is showing), this is wrong. I don't know what his bottleneck is, but it isn't that.

 

[spidey07]

Bonded NICs don't load balance, they link balance. That means that each connection (based on source/destination MAC or IP, depending on configuration) will go across one link or another. That means that total aggregate bandwidth will be double, but that the speed of any one connection will never be more than the speed of a single link.

Well said. No single conversation can ever go above 100 Mbs.

 

[spidey07]

For the Linux bonding driver in round robin mode (which is what the OP is showing), this is wrong. I don't know what his bottleneck is, but it isn't that.

You can't get around the basic layer2 operation.

-edit-
Welcome to the forums NGbnd0

 

[Pheran]

You can't get around the basic layer2 operation.

-edit-
Welcome to the forums NGbnd0

Spidey, I don't know what you are talking about here. The Linux bonding driver, in round-robin mode, does exactly what it says - sends one frame out the first interface, and the next frame out the second interface. That mode of operation is not 802.3ad compliant as it can reorder frames in a single connection, but it does ensure essentially perfect load balancing.

 

[spidey07]

Spidey, I don't know what you are talking about here. The Linux bonding driver, in round-robin mode, does exactly what it says - sends one frame out the first interface, and the next frame out the second interface. That mode of operation is not 802.3ad compliant as it can reorder frames in a single connection, but it does ensure essentially perfect load balancing.

What is the source MAC of the frame? If it's different then ARP will be screwed, if it's the same then a switch won't like seeing the MAC on two different ports.

 

[imagoon]

What is the source MAC of the frame? If it's different then ARP will be screwed, if it's the same then a switch won't like seeing the MAC on two different ports.

Digging around in the FAQ, the bonding driver changes the MAC addresses of all NICs to be the same. They say you must have something like etherchannel configured to handle that on the switch.

Another note was this:

mode=0 (balance-rr)
Round-robin policy: Transmit packets in sequential order from the first available slave through the last. This mode provides load balancing and fault tolerance.

My question would them be... what switch are you using and does it support what is going on. Once the packets are on the switch, it is up to the switch to choose which nic get the packet so it is very possible that even though your are round robining them out of the nics the switch is using something like IP balancing and sending them down one pipe to the receiving computer.

Never attempted this before and this is what I gleaned from the web...

 

[Pheran]

Digging around in the FAQ, the bonding driver changes the MAC addresses of all NICs to be the same. They say you must have something like etherchannel configured to handle that on the switch.

Another note was this:

mode=0 (balance-rr)
Round-robin policy: Transmit packets in sequential order from the first available slave through the last. This mode provides load balancing and fault tolerance.

My question would them be... what switch are you using and does it support what is going on. Once the packets are on the switch, it is up to the switch to choose which nic get the packet so it is very possible that even though your are round robining them out of the nics the switch is using something like IP balancing and sending them down one pipe to the receiving computer.

Never attempted this before and this is what I gleaned from the web...

Exactly, the source MAC is the same. The switch must be configured for link aggregation (aka etherchannel), Linux bonding is not meant to be used with a dumb switch. Note, however, that the OP isn't using a switch at all if I understand his post correctly.

 

[Gooberlx2]

Aside from the questions about bonding mode, what kind of connection/hardware is the OP behind? I can't really tell if he's connecting directly. Isn't 100MBytes/s nearly saturating a 1000 Mbit/s link after overhead, anyway?

 

[imagoon]

Aside from the questions about bonding mode, what kind of connection/hardware is the OP behind? I can't really tell if he's connecting directly. Isn't 100MBytes/s nearly saturating a 1000 Mbit/s link after overhead, anyway?

Well in theory round robin would let you saturate both getting you close to 200MB/s assuming the cpu /disk can handle it.

 

[Gooberlx2]

Maybe OP needs to use balance-alb instead?

http://www.enterprisenetworkingplanet.com/nethub/article.php/3696561

 

[NGbnd0]

Hi

many thanks for your thoughts - now on each PC are three Gbit-NICs, but nothing has changed ... i'm still standing here like a duck in a thunderstorm ;-)

@Pheran: you're right, there is no switch involved
@Gooberlx2: in mode 6 (balance-alb) only one interface - says iptraf - has traffic ... then the bandwidth reaches 105Mb/sec (measured with iperf, and also with a nfs-share).

by the way: "Link failure count:0" and "errors:0 dropped:0 overruns:0"

In mode 0 the traffic is equally distributed over all interfaces (45/45 MB/sec respectively 30/30/30 MB/sec ... *#&$@! ... and if i disconnect one cable during the transfer, i see 30/55/0 or so)

still helpless
Daniel