Route Cache [On|Off] on Core L3 Switch

[Santa]

My goal is to delay any need for additional WAN links or Bandwidth voodoo magic bonding at this point.

We currently have 2 cisco 4506 at the core and 2 cisco 3745 at the edge.

Each 3745 hosts two IMA groups that can each know how to get to a Datacenter but do not check each other for load.

Problem is once a destination host is locked onto one of the lines in cache it tends to stay there.

In other words my users tend to hit just one IP at the DC and they do alot of traffic to that one IP and I want to disperse the load across the two 3745 thus across two seperate physical links.

I think turning route cacheing should at least get me round robin but is the L3 core where I should do this or should I try to set something up on the 3745s?

I don't have them fully meshed with HSRP yet and at the other end I am not sure that they have things set up for HSRP either but at the very least I am under the assumption I will work in getting my end to send down both pipes as evenly as possible and I will work with their people to get their end configured.

Should I even be thinking CEF? I am wondering if processor utilization will become an issue for either turning off route cacheing or doing CEF.

 

[spidey07]

not really following what you're trying to do but turning off route caching is a pretty bad idea - it will basically turn off any fast routing and do it all through the processor which can bring a switch/router to its knees.

If you can describe the physical and logical layout a little better and what you're trying to accomplish maybe I can help.

 

[bgroff]

My suggestion is to look at Cisco's GLBP (Gateway Load Balacing protocol). It operates similarly to HSRP, except both routers are active. Perhaps this might be an answer to your problems...

 

[Santa]

right now our situation is that we have 2 IMA groups at our location each 3Mbit bonded NxT1.

At our other DC we have two T1 not bonded so all in all we have the following:


----3Mbit--------------------- 1.5Mbit---
----| |---------
----3Mbit--------------------- 1.5Mbit---

Best ASCII can can do but more or less the two 3mbit pipes have T1 they are pumping into so utilization isn't peaked on our line.

But it appears we can't seem to get above CIR per line on their end. Granted they may not have the routers configured to burst but thats a differnt issue. We sometimes have all the traffic going over just the one line as apposed to any true round robin load balancing.

I know there is some hit to the processing power of the routing equipment when it comes to looking at the routing table each time. so you are recommending that we do not do it at the core level and leave it on.

Can GLBP provide Load balancing even if not per packet but per conversation/session stream? I read a little on the GLBP but couldn't really get a grasp on whether it was more for redundancy in case a router in the cluster fails.

Appreciate the help.

 

[Santa]

*bump*

 

[bgroff]

GLBP works by changing the arp responce as to who is the active router. So router A will get the virtual mac pointed at it, followed by router b, router c, etc. That is how the traffic is load balanced. Since you have a layer 3 core in place, this may or may not be the best solution but it should at least help some. The problem is the switch will cache the arp response from the router. The arp will have to time out of the arp table of the switch to load balance. What I'm not sure of is whether GLBP will gratuitously arp to update the cache. If this is the case, then you are good to go.

 

[Santa]

So in the case where I need to get better balance between two routers and I am using an L3 switch as the user's default gateway should I make the two L3 switches default route for the destination network be a certain core router and then do some sort of dual group HSRP on the routers for redundancy in case one fails? Then run GLBP across the two L3 core routers?

I am thinking if Switch 1 answers the response and it's only route to the destination network is router 1 then it should send the packet to that router and that router only no matter what route cache. But then Switch 2 would send to router 2.

right now on both L3 switch there are two static routes pointing at both routers but the end user tends to stick to one switch and one router. I need them to load balance more on the WAN side than the LAN but will GLBP be less sticky(for lack of a better word) than equal cost static routes, round robin, and route caching combined?

Sorry for the semi disorganized ramble don't feel like going through and rewriting :/

 

[Pheran]

Santa hasn't really given us enough topology information to know at this point, but GLBP is not likely to help all that much in this scenario, since it sounds like the problem is further down the line than the default gateway. First, if you're not using CEF, you should be. It's Cisco's fastest packet switching mechanism. Every router you ever deploy should have CEF turned on, unless there's a memory constraint, in which case you should upgrade the memory.

You don't mention what routing protocol is in use. If it's OSPF, make sure you've set maximum-paths to two or more so you pick up dual routes in your tables. Anyway, as spidey says, unless you give us more info (like precisely describing your layer 3 topology), we're not likely to be much help.

 

[Santa]

EIGRP within the core, No OSPF.
Down to the other datacenter we have no sharing or dynamic routing just static routing.

Within the L3 core there currently is just two L3 cisco 4506 catalyst switches one sitting there just for failover at the moment but we are looking either at HSRP or this new GLBP. GLBP seems to be kind of newer feature set that we have to upgrade our IOS to get to. We are a few revisions old from when GLBP was first offered.

Currently the clients default into these core switches at Layer 3 in other words the switches host the default gateway.

Then the switches either hand off to the core router if it is a WAN related request or if it is internet related it sends it directly to the firewall segment of the network.

Very simple configuration but probably lacks flexibility in its current state and thats what this thread is about.

I am thinking of trying to better mesh the L3 switches and core routers to create better load balancing along with not sacraficing any redundancy. I will research CEF some more but we have never used it in the past because our routers could not handle the extra overhead.

Would CEF alone load balance the outgoing packets between two routers? Or only load balance the packets if one router hosted both lines?

Right now there are two IMA groups hosted by 2 routers each only having one route to the other datacenter via one path (its own hosted line)

Trying to provide as much information as possible. (just think simple design and you probably guessed my set up currently)

 

[bgroff]

So if I'm understanding you correctly, your clients are on something like 2950s in the closets that come back to the 4506 switches. In which case you set up GLBP on the 4506 switches, and the dual group hsrp on the WAN routers. The each 4506 would have a different default to one of HSRP groups. That way the clients get the advantage of GLBP (which will work better due to the arp-ing nature of the beast) and you still have dedundancy.

 

[Santa]

Well bgroff, that was a configuration I came up with after hearing about GLBP. I was hoping to get confirmation whether that is the best solution for what I am doing.

And you are correct about the access layer being 2950s coming into the 4506s.

So the consensus is to leave route caching on and try to load balance using GLBP/HSRP and perhaps CEF also?

I am curious whether CEF will even work on a router with only one link hosted. Or do I have to set another equal cost route to the other router and link the routers?

I will eventually need to do something facing the WAN for redundancy also (this is not complete yet)

I was going to do the HSRP group once again since most branch offices do not have redundant physical links but just 1 physical link with 2 PVC pointing back at each IMA group.

Hope this is clearing up for someone other than myself

 

[bgroff]

Yes, it should do the trick. Each 4506 should be an equal cost peer in the GLBP group. Each 4506 should have a static default to a different IP in the dual HSRP config. Every router should have IP CEF turned on (this improves IP forwarding speed, it does not change the way IP routing works). Leave all route caches on (this also improves IP forwarding speed).

 

[Santa]

Does CEF help on a L3 core switch? or maybe what I am most interested in is, will it help in my core switch scenerio?

One thing I am also tackling with right now is how exactly will the GLBP work if the server is only connected to or talking to just one of the switches at Layer 1 (Active NIC only on one Switch).

Will the GLBP work and the second switch still arp the MAC address to the server through the backdoor link so that the second switch will recieve the packet to foward on? Or will only the switch that the server is hooked into recieve the packet and forward it onto the router it was statically defaulted to?

Our server environment has dual NIC that are hooked one each into each core switch but only one NIC is active and the secondary one is just passive until the first one dies.

This causes the server to technically be only on one switch at a time.

So will the ARP trickery work in this situation? The two switches are linked together so I am guessing there will be some passing of ARP messages through these links and every other time the second router will answer an ARP for the default gateway L3 address.

Confusing still?

I really do need to upload a pic of all this hehe. Lemme see if I can get one together here and upload it to view.

Thanks everyone. I think my biggest problem is that we don't have a test bed to play in Gotta do this on the core and its nerve racking when you play with meshing and trying to do tricks with making the packets go multiple ways until there is a failure then pick the surviving route. oh how fun

 

[Santa]

One thing I thought of. If we had a core switch failure then we would effectivly kill 1/2 of our WAN bandwidth. Not good.

Any way to avoid this when a core switch dies? Thinking.. thinking..

 

[Santa]

After speaking with some of my engineers at cisco they can't think of doing anything to prevent the issue I will have if a core switch dies. I will lose 1/2 the WAN bandwidth no matter what in this setup.

routing protocols will keep the network online but I will need to get either another static route into the surviving switch or bring the dead switch back online in a hurry or one router will not be used as much.

He was mentioning that he has not had anyone use the GLBP and that I should keep in mind that there may be some additional troubleshooting issues when the client has multiple paths to the default gateway but if I can't come up with a differnt design then this will have to work for now.

 

[Santa]

Well ain't that something. In trying to obtain the IOS to get GLBP feature set I end up finding out GLBP is only availble currently on Routers not L3 Switches.

Shoot.. now back to the drawing board.

Any other ideas? that one would of worked with a small contingent plan but now that the L3 switches can't do GLBP I am at a loss for what next.