Migrating to a new core switch/router?

[cpals]

We are finally getting our 10+ year old 6509 replaced for our main switch. We are doing it in stages and trying to minimize downtime. So right now I have the 6509 up and our new switch connected to it with a 6-1Gb LACP between them tagging vlans. The 6509 still has all of the vlans with the ips (gateways).

The main thing connected to the new switch are our servers with Iscsi drives and our NetApp San. Correct me if I'm wrong but if a server on our vlan 4 is going to talk to a different server on vlan 5 or storage on vlan 6 the traffic would start on the new switch, go out the 6 port lag and then back up the lag to its destination on the new switch?

I want to fix that as my next step so I think I need to move the gateways for the vlans and put those ips on the vlans on the new switch. My question is how would I do this? The 6509 is still the main hub and I'm trying to figure out how user traffic would know about the gateway move. Is it easy as moving the ip on the vlan? Would I leave the vlan on the 6509 or just get rid of it if I have no more servers connected on the 6509 and remove that from being tagged on the lag between switches?

Sorry if I'm asking a stupid question... Running on 4 hours of sleep after a large project last night! Thought I'd ping the community to make sure I'm not crazy.

[imagoon]

While it always depends on your environment... I personally would have grabbed the 6509's config, completely configured the replacement and done a drop in replacement.

If the 6509 is still doing routing, then your comment is correct, traffic is going out as tagged frames on the LAG, being routed and then sent back out tagged over that lag, assuming the traffic is generated on the replacement and being sent back out a port on the replacement. Are these switches trading any routing information? Simply moving the IP addresses may open far more of a can of worms than migrating the entire config in one shot. It depends on the routing protocols and other devices, and how their routing security is done. It could also "just work."

[drebo]

I would have done it in one shot during a maintenance window. As it is, you'll need one to migrate the L3 interfaces from the old switch to the new switch anyway.

You'll need to shut down the L3 interfaces (the VLAN interfaces) on the 6509 and then bring up identical ones on the new switch.

The L2 topology won't change, so you don't need to touch the clients. Some may have issues with arp timeouts and stuff, but that's easily repairable by clearing the ARP cache or restarting all of your access layer switches (link down should tell most operating systems to clear ARP.)

[cpals]

I know... I would've done the same thing, unfortunately some portions of the 6509 are not being replaced by the new switch. We have a MetroE connection that is currently being encrypted by an ipsec module in the 6509. These encrypted connections are now going to be moved to two Juniper SRX1400 boxes, but we've been having issues with our installer and Juniper taking forever to get them configured and installed so we thought we would forge on and at least move the L2/L3 connections over to the new switch.

[cpals]

imagoon: No, the new switch has no routing information on it currently... The 6509 has static routes and we are utilizing EIGRP for a small ring we have for our main sites. it's strictly sharing the vlans. To add a little more detail to the scenario, we have 5 server racks and every single connection was plugged into the 6509. Our new switch solution we're doing has chassis for the core and then ToR switches in each rack with two 10Gb twinax connections going from each rack to the core switch.

In the short term, I'm just trying to make sure server-type traffic stays internal to the new switch infrastructure and doesn't get bogged down across the 6-1Gb lag.

So I don't want to remove the vlans on the 6509, just the ip on them correct? I'm essentially swapping the setup. The only way to permanently remove the vlan on the 6509 would be to route to the new switch, correct? I'm just curious how much broadcast traffic, etc is going to be passing between the two switches.

[spidey07]

There shouldn't be much traffic, but keep an eye on that LAG. Really all depends on what kind of load is on the 6500 and where the conversations are.

All you really need to do is move the SVI/routing interfaces to the new switch. That would move routing for those VLANs to the new switch and keep it all in there. It would only leave that switch if it needed to talk L2 to something on 6500 and L3 to a route it needs to keep.

Does the new switch support EIGRP? Or are you content with statics (eww).

[cpals]

Awesome, that sounds just like what I was thinking.

We've decided to move away from Cisco (for different reasons, partially due to our reseller) and are moving to a complete Enterasys solution. Our small sites are all static /30 connections between the 6509 and 2821 routers. This is being replaced with the Junipers and our main sites will stay on a L2 ring doing OSP most likely instead of EIGRP.

Edit: Whoops, forgot to ask. If I change the routing interface from the 6509 to the new switch at the same time, what kind of time are we talking about for the network to learn about the new destination? Almost instantly or would some of the users see a blip? All user traffic ends up at the 6509 currently so if I did a quick clearing of the arp cache would that fix it?

[drebo]

You'd have to clear the arp cache on the end user workstations.

You can't do this outside of a maintenance window. Downtime, at the very least, will be a couple of minutes. At the most, it could be a couple hours as IT goes around to everyone's desktop and manually clears ARP. Or you could reset the access switches to cause link down to the clients, which should clear ARP on most OSes.

This is one of the reasons why I don't like L2 in the core.

[cpals]

Correct me if I'm wrong, but all of our user traffic is on different subnets...

Example Scenario:
User - 192.168.1.55 (vlan 10 on 6509)
Server - 192.168.200.30 (vlan 200 on 6509)

If a user tries to access the server won't it's arp request and the mac that comes back be the router's mac? So going outside it's own subnet the mac destination will always be the same? The only place that would need to know the new location of the gateway for the server network would be the 6509 so clearing the cache on that device would fix things?

Am I thinking about it wrong?

[drebo]

Only if they haven't arped their gateway in the last 4 hours (default arp timeout.)

Clients won't issue an ARP request for an IP that they know is not on their own subnet...they'll ARP for the gateway instead.

[imagoon]

Windows ARP timeouts are not 4 hours (linux / other hardware might be however)
2k / xp / 2003: 2 minutes
Vista / 7 / 8 / 2008 [r2]: Random value 15 - 45 seconds

The users would still see some oddness during that time. Link bounce clears the cache immediately.

This of course can be way different if you are doing arp caching else where on the network which is more common on L2 over WAN. If a local router is being used as an L2 gateway of a sort and is caching the gateway address, no matter of clearing clients will get them online until those caches are flushed / reset.

"Know your network" is the key here though.

[cpals]

Originally Posted by drebo

Only if they haven't arped their gateway in the last 4 hours (default arp timeout.)

Clients won't issue an ARP request for an IP that they know is not on their own subnet...they'll ARP for the gateway instead.

Ah right, but along those same lines... the user will arp for 192.168.1.1 which has not changed (yet) in order to get to the server. The packet reaches 1.1 (6509) which does know about the new path to the server vlan gateway. So the only place that would need an arp cleaning would be the 6509?

Now when I move the 1.1 vlan to the new switch I could see that causing some issues.

Am I on the right path?

[spidey07]

You're still going to need routing between the 6500 and new switch when you move the SVIs (vlan/routing interfaces on the 6500). Seems you're going to need to use statics and likely a default on the new switch. It all depends on how your routing is being done and the paths.

[cpals]

Looks like I missed something here... tried it tonight on a non-essential vlan and once I put the IP on the new switch and shut down the 6509's interface I began to get TTL expired messages in my pings. I even cleared cache, etc.

Once it failed I tried to play around with different routing commands and tried like 'ip route 192.168.2.0 255.255.255.0 vlan 5' to see if that would actually work, but it did not. Once I did a shut on the new switch and no shut on the 6509 everything instantly started pinging.

Time to investigate more I guess.

I was considering setting the new switch up like our remote sites??? Create a /30 subnet between them and then put the routes into the 6509 and I think I would just need a default route on the new switch going back to the 6509 so it catches everything else and can go out to the internet?

[spidey07]

TTL expired means you have a routing loop. This is something that really needs to be planned out by somebody that knows what they are doing. Sorry.

[Cooky]

Besides routing, you may also want to look into setting up VRRP, or whatever's the standard FHRP nowadays.
When you have FHRP, your future gateway SVI migration to a different device can be done in a much easier fashion, w/ no downtime, if it's carried out correctly.
This applies even if you're running a single core today.

[cpals]

No offense taken Spidey. I think my problem is I'm running off of around 6 hours sleep for the last two days so may not be thinking clearly. I just need to sit down and plan/think it through.

We unfortunately are on a time crunch and our boss doesn't think we need help so we pretty much have to figure it out.

Ill get with our new switch vendor to see what they think about the vrrp Cooky. Thanks for the tip.

[kevnich2]

Originally Posted by cpals

No offense taken Spidey. I think my problem is I'm running off of around 6 hours sleep for the last two days so may not be thinking clearly. I just need to sit down and plan/think it through.

We unfortunately are on a time crunch and our boss doesn't think we need help so we pretty much have to figure it out.

Ill get with our new switch vendor to see what they think about the vrrp Cooky. Thanks for the tip.

Not that I'm telling you anything you don't already know judging by your post but stuff like this REALLY needs planned out on paper with the entire physical and logical network laid out with your current setup and one with the new setup and just go through everything and make sure all L2 and L3 work, then go into the switch and start programming up the new one. Even better if you can make a mini lab to install it and test it with. Then plan a maintenance window and install the new switch.

[cpals]

Originally Posted by kevnich2

Not that I'm telling you anything you don't already know judging by your post but stuff like this REALLY needs planned out on paper with the entire physical and logical network laid out with your current setup and one with the new setup and just go through everything and make sure all L2 and L3 work, then go into the switch and start programming up the new one. Even better if you can make a mini lab to install it and test it with. Then plan a maintenance window and install the new switch.

Believe me... I know. It's typical for where I work... take 6 months to buy the products and then expect it installed in two weeks without any knowledge or training on it.

Also, we have no planned downtime outages due to the nature of our work. It's 24/7 and someone will always be mad.

Thanks for all the inputs!