Linux for simple layer 3 switching?

jaydee

Diamond Member
May 6, 2000
4,500
4
81
Not sure if I should post this in *nix Software, or Networking, I'll try here first. Bare with me if my terminology isn't 100%, I'm an EE and this isn't my area of expertise (obviously or I wouldn't be asking).

I am trying to create/manage 8 separate LANs, on separate Cisco 2960 switches (let's call them LAN 1-8). I have a three 2-port Cisco Routers (2x 1900 and 1x 2900) at my disposal, which are currently connecting LAN 1-2, LAN 3-4 and the last one is not configured.

I need to keep all 8 networks on separate LANs, but allow them to talk to each other. Nothing fancy necessary, just static IP, no firewall, no wireless, no dynamic routing, no internet, etc.

I have a server sitting here with 8 ethernet ports (which is otherwise incredibly overspec'd for this task, but we have no other use for at the moment), I am hopeful I can throw Linux on it and configure it to be an L3 switch.

First, does all this seem reasonable so far?

If so, should I pursue:
1. base linux networking (on Debian or CentOS for example) to make it a switch
2. try to find a 3rd party package, (if so, suggestions?)
3. a specific OS version geared towards this (if so, suggestions?)

I've done some googling and haven't come to a consensus on which is the best approach, most of what I've found is focused on more complex switching/routing features then what I need.

Edit: removed the term VLANs.
 
Last edited:

imagoon

Diamond Member
Feb 19, 2003
5,199
0
0
You could make them all talk via a single 2900. You would need to use a switch to consolidate the networks in to VLANs then use the 2900 as a router on a stick. You only would need to use a single port on the 2900 for this.

You should draw a network diagram. The design you have sounds needlessly complicated and trying to use Linux as a layer 3 switch (what you are describing is close to using Linux as a router) is going to just add more equipment/power/complication with little gain.

--edit--

Also as an EE.... don't call the lans "vlans" as what you described, you are not using vlans. That will further complicate the issue if you are using the terms incorrectly.
 
Last edited:

jaydee

Diamond Member
May 6, 2000
4,500
4
81
You could make them all talk via a single 2900. You would need to use a switch to consolidate the networks in to VLANs then use the 2900 as a router on a stick. You only would need to use a single port on the 2900 for this.

You should draw a network diagram. The design you have sounds needlessly complicated and trying to use Linux as a layer 3 switch (what you are describing is close to using Linux as a router) is going to just add more equipment/power/complication with little gain.

--edit--

Also as an EE.... don't call the lans "vlans" as what you described, you are not using vlans. That will further complicate the issue if you are using the terms incorrectly.

You're saying, with a spare 2960, I can bring all 8 LANs together, link them through one port on the router?

Can you describe (or point me to a link) at a high level a little bit about how the 2960 and 2900 would be configured for this?
 

imagoon

Diamond Member
Feb 19, 2003
5,199
0
0
You're saying, with a spare 2960, I can bring all 8 LANs together, link them through one port on the router?

Can you describe (or point me to a link) at a high level a little bit about how the 2960 and 2900 would be configured for this?

Very high level (I have no idea what you have to connect all this)

2960 Switch, assuming that all the other switches are not participating in the same vlan domain:

Switch ports 1-8 configured similar to this:

vlan database
vlan 101
vlan 102
--- etc

e0/1
switchport mode access
switchport access vlan 101

e0/2
switchport mode access
switchport access vlan 102

e0/3
switchport mode access
switchport access vlan 103

--- etc

Router port:

e0/24
Switchport mode trunk
switchport trunk allowed vlan add 101,102,103,104,105,106,107,108

router:

interface g0/0.101
ip address 192.168.101.1 255.255.255.0

interface g0/0.102
ip address 192.168.102.1 255.255.255.0

interface g0/0.103
ip address 192.168.103.1 255.255.255.0

--- etc

Basically you use one of the switches as a fancy "vlan converter" and then trunk that to the router where it has IP addresses in each LAN. You then create your routes and ACLs as required. If the LANs have a a different default gateway than the router doing the inter LAN traffic, you will need to put routes to the other LANs in that device as well.

Look up "router on a stick" for more information and lots of examples. Also depending on the model of 2900 series router, you can just get ethernet cards and pop them right in to WIC slots. Larger routers like the 2921 also support devices like SM-X-ES3-16-P which would just bolt the switch right in to the router.

--edit--

WIC card version (routable slots)

http://www.cisco.com/c/en/us/produc...cards/product_data_sheet0900aecd8016bf0b.html

To clarify, the switch module is a switch and acts like a switch. It makes the router "hang off" the network in router on a stick mode. Using WICs makes them routable ports, IE each port can have an a discrete IP attached and act as router ports. IT 2 4 port wics would turn your 2 port 2900 in to an 10 port 2900 router.
 
Last edited:

imagoon

Diamond Member
Feb 19, 2003
5,199
0
0
The 2901 is the 1U device so it doesn't have that slot. The smaller 4 port ones should work. Make sure to check you have the right IOS version. I have seen the 4 port ones around used for $40 or less at times.

And yes you would basically plug the various networks in to those ports.

"edit that"

Cisco lists the 9 port as working in the 2901, up to 2. Likely means there is a way to pop the divider out.
 
Last edited:

jaydee

Diamond Member
May 6, 2000
4,500
4
81
The 2901 is the 1U device so it doesn't have that slot. The smaller 4 port ones should work. Make sure to check you have the right IOS version. I have seen the 4 port ones around used for $40 or less at times.

And yes you would basically plug the various networks in to those ports.

"edit that"

Cisco lists the 9 port as working in the 2901, up to 2. Likely means there is a way to pop the divider out.

Thanks for the help. The 9 port switch is on order, I'll let you know how this works out. I was able to remove the divider, so it should work. What do you mean by the right IOS version? Which version should the 2901 have?
 

jaydee

Diamond Member
May 6, 2000
4,500
4
81
http://www.cisco.com/c/en/us/produc...cards/product_data_sheet0900aecd8016bf0b.html

• Minimum Cisco IOS Software Release 15.0(1)M for Cisco 1900 Series, Cisco 2900 Series and
Cisco 3900 Series routers: IP Base License of the Universal image.

Thanks!

So I will have 6 physical Cisco 2960 switches, broken up into 12 VLANs. Do all 6 switches need to be on version 15? I haven't checked all of them yet, but the first one has 12.2<55>SE5, and I suspect the rest do as well. Should they all be upgraded to v15, or are they ok?
 

drebo

Diamond Member
Feb 24, 2006
7,034
1
81
2960S and 2960X can do layer 3 switching up to 16 networks, if you change the SDM template.
 

imagoon

Diamond Member
Feb 19, 2003
5,199
0
0
Thanks!

So I will have 6 physical Cisco 2960 switches, broken up into 12 VLANs. Do all 6 switches need to be on version 15? I haven't checked all of them yet, but the first one has 12.2<55>SE5, and I suspect the rest do as well. Should they all be upgraded to v15, or are they ok?

The IOS version requirements only refer to the router you use. Drebo brings up a good point also, I didn't think to ask you if your 2960 switches had layer 3 support or not. What is the version of the switch?
 

jaydee

Diamond Member
May 6, 2000
4,500
4
81
6x "2960 series"
2x "2960-S series"
1x "2960-S series SI"

The latter 3 are already serving a purpose that needs the GB Ethernet ports though.

We already have the two 7-port expansion cards ordered, and on their way this week, if you are suggesting not using them, I'd have to come up with an extremely compelling reason...
 

drebo

Diamond Member
Feb 24, 2006
7,034
1
81
6x "2960 series"
2x "2960-S series"
1x "2960-S series SI"

The latter 3 are already serving a purpose that needs the GB Ethernet ports though.

We already have the two 7-port expansion cards ordered, and on their way this week, if you are suggesting not using them, I'd have to come up with an extremely compelling reason...

Biggest issue is throughput. Routers have a much lower forwarding rate than layer 3 switches.

The EtherSwitch WICs are great for L2, as it's switched on-module, but for L3, that has to hit the proc of the router which subjects it to the same forwarding limitations of their onboard ports (think router on a stick.)

If throughput isn't a huge concern, then a router with an EtherSwitch module is the way to go.
 

imagoon

Diamond Member
Feb 19, 2003
5,199
0
0
Biggest issue is throughput. Routers have a much lower forwarding rate than layer 3 switches.

The EtherSwitch WICs are great for L2, as it's switched on-module, but for L3, that has to hit the proc of the router which subjects it to the same forwarding limitations of their onboard ports (think router on a stick.)

If throughput isn't a huge concern, then a router with an EtherSwitch module is the way to go.

Interestingly there seems to be almost no information about the performance of the 2960S in LAN-Base mode. Looking at what the switch loses (MAC addresses, buffers etc) in that mode, it looks like it routes entirely in software (at least it suggest it, I am not looking that deep.) The 2921 should be able to push ~245mpbs, not sure how that compares.
 
Last edited:

jaydee

Diamond Member
May 6, 2000
4,500
4
81
Biggest issue is throughput. Routers have a much lower forwarding rate than layer 3 switches.

The EtherSwitch WICs are great for L2, as it's switched on-module, but for L3, that has to hit the proc of the router which subjects it to the same forwarding limitations of their onboard ports (think router on a stick.)

If throughput isn't a huge concern, then a router with an EtherSwitch module is the way to go.

Oh gosh no, throughput is no concern at all. We're talking a max of a dozen or two messages per second, less than 1 kb each and as long as they're delivered within seconds, we're fine.
 

drebo

Diamond Member
Feb 24, 2006
7,034
1
81
Interestingly there seems to be almost no information about the performance of the 2960S in LAN-Base mode. Looking at what the switch loses (MAC addresses, buffers etc) in that mode, it looks like it routes entirely in software (at least it suggest it, I am not looking that deep.) The 2921 should be able to push ~245mpbs, not sure how that compares.

It has to do with TCAM allocation. Switching and forwarding is all handled at the ASIC level.

When you enable the capability of the switch to handle layer 3 routes, the switch has to dedicate some TCAM to those entries, which means it loses TCAM that would have been allocated to the MAC table.
 

imagoon

Diamond Member
Feb 19, 2003
5,199
0
0
It has to do with TCAM allocation. Switching and forwarding is all handled at the ASIC level.

When you enable the capability of the switch to handle layer 3 routes, the switch has to dedicate some TCAM to those entries, which means it loses TCAM that would have been allocated to the MAC table.

Well what I meant is that Cisco added "routing" in the middle of the run. Basically they rolled an updated IOS and added it to the 2960S. So it makes me wonder what it can do since it wasn't purpose built like a 3560 / 3750 and other bigger switches. It seems they added it to hit the SBM which may mean it doesn't have a whole lot of power.
 

drebo

Diamond Member
Feb 24, 2006
7,034
1
81
Well what I meant is that Cisco added "routing" in the middle of the run. Basically they rolled an updated IOS and added it to the 2960S. So it makes me wonder what it can do since it wasn't purpose built like a 3560 / 3750 and other bigger switches. It seems they added it to hit the SBM which may mean it doesn't have a whole lot of power.

Well, that's kind of my point...when you do L3 switching in the ASIC the way Cisco does (basically, all "destinations" for packets are stored in TCAM), then there is no performance difference between L2 and L3 traffic.

TCAM evaluates all configurations and current known addresses and destinations and stores them with their egress port/next hop MAC already calculated. This is all evaluated in hardware in any system that uses TCAM, so L2 vs L3 traffic doesn't matter.
 

jaydee

Diamond Member
May 6, 2000
4,500
4
81
Ok, we've got the two 9-port switches installed into the Cisco 2901. We think we have it configured correctly, but for some reason, only one of the 9-port expansion card switches will respond at a time. When they are both plugged in, we can only ping the ports on one of the 9-port expansion card (the card that's plugged into the EHWIC 0/1 slot). If we take out that card, we can ping the ports on the card that is plugged into the EHWIC 2/3 slot.

Other than not getting both cards to work at the same time, we seem have it configured the way we want. All the VLANs represented on an expansion card are able to communicate with all the other VLANs on that expansion card.

Any special tips/tricks? I forget exactly which software version we have, but it did meet the minimum posted on the Cisco link posted earlier in this thread: 15.0(1)M.
 

imagoon

Diamond Member
Feb 19, 2003
5,199
0
0
Does the second card show up in the "show hardware" cli command? Cisco specs say that 2 should work. The other thing could be that when you have to the naming scheme changes in a way you didn't expect. IE interface e2/0/1 or some such. You should be able to do a show interface status and see all the ports.

Something else I have seen on some of those... the second card may need to be configured as a stacking partner and you use a crossover cable to connect them. Let me see if I can find those notes.

edit:

http://www.cisco.com/c/en/us/td/doc...tml#GUID-23AA4C03-61C6-49CF-B270-DDFF5C0A52A1

Not more than two EtherSwitch HWICs or network modules must be installed in a host device.

Multiple EtherSwitch HWICs or network modules installed in a host device will not act independently of each other. They must be stacked, as they will not work otherwise.

You need to stack the 2 boards.
 
Last edited:

jaydee

Diamond Member
May 6, 2000
4,500
4
81
Does the second card show up in the "show hardware" cli command? Cisco specs say that 2 should work. The other thing could be that when you have to the naming scheme changes in a way you didn't expect. IE interface e2/0/1 or some such. You should be able to do a show interface status and see all the ports.

Something else I have seen on some of those... the second card may need to be configured as a stacking partner and you use a crossover cable to connect them. Let me see if I can find those notes.

edit:

http://www.cisco.com/c/en/us/td/doc...tml#GUID-23AA4C03-61C6-49CF-B270-DDFF5C0A52A1

Not more than two EtherSwitch HWICs or network modules must be installed in a host device.

Multiple EtherSwitch HWICs or network modules installed in a host device will not act independently of each other. They must be stacked, as they will not work otherwise.

You need to stack the 2 boards.

Yes, that was it. We now have the entire network fully functional as we want it. I really appreciate all the contributions here (beginning of thread to now), I am not sure how much longer this would have taken without your kind help, but I'm pretty sure it would have been considerable.
 

RadiclDreamer

Diamond Member
Aug 8, 2004
8,622
40
91
What you are trying to accomplish is sometimes called router on a stick, you setup a bunch of sub interfaces on the router and then trunk it down to the switch, they will all talk just fine but all be on separate subnets