Changing website IP - there has to be a good way

[Ares2600]

Cliffs notes: ip change caused support issues due to cached DNS entries on the client systems, ignoring TTL. looking for the best way to avoid the headaches

Long story:

I work for a fairly large and growing financial website... we upgraded our internet circuits over the weekend and were forced to change the IP address. Despite a reasonably low TTL in our DNS records, our support queues were slammed this morning with connectivity issues due to cached ips at what seems to be browser, system and home router layers. It wasn't the best customer experience.

We upgraded by failing over to our second data center for the weekend while the upgrade was completed in our primary data center. Failover is accomplished by a DNS change to the ips for our backup data center. Maintenance was completed and we failed back, but to a new set of ips from our new ISP.

It seems to me that the best way to actually do this is avoid a hard cut over.. leave the old ip addresses active for some overlap to allow the caches to time out. It's frustrating that TTL isn't honored on a larger scale on the client side. Having an active-active setup, with multiple ips in the DNS record would mitigate this too, I'd imagine. In the scenario where we are forced to change, only some of the active ips would be forced to change and those clients who haven't refreshed aren't left out in the cold.

Is there a best practice here I don't know about? How is this accomplished elsewhere?

 

[imagoon]

Typically it is what you mentioned. Set the TTL low, bring the new ranges up, leave the old ones there with redirection to the proper IPs via a firewall or otherwise. Take the old ones down.

The next best method is to buy a range from ARIN, and get a provider that can support having you has a fully BGP routable group. Bonus points because now you can easily support multiple paths and providers.

 

[ViviTheMage]

Typically it is what you mentioned. Set the TTL low, bring the new ranges up, leave the old ones there with redirection to the proper IPs via a firewall or otherwise. Take the old ones down.

The next best method is to buy a range from ARIN, and get a provider that can support having you has a fully BGP routable group. Bonus points because now you can easily support multiple paths and providers.

Really, this is the best option, if you can do it.

Or, is it not possible to have 1 IP front facing, and load balance? Of course you would NEVER change/lose that 1 IP.

 

[Cooky]

We're fortunate to have our own IP block from ARIN, so this is a non-issue for us.
For those who have ISP assigned IP's, your best action is what the OP described, as far as I know.

We looked into load balancing Internet facing apps between the two datacenters, but decided not to because some systems wouldn't honor the TTL.

 

[Ares2600]

Thanks for the input everyone. The BGP enabled connections are the way to go. I'm pushing towards a setup where we never really touch our public DNS record again. I took over all of this and I'm more of a software nerd than a network nerd, so it's ammunition like this that I need when dealing with my infrastructure group.

 

[Emulex]

ares2600 - tell us about the cost of getting portable IP's and two BGP4 links to your office.

Metro-E (need two at least) is not cheap - routers that can handle two full bgp4 tables are not cheap (well software ones are).

Portable ip's - not cheap at all these days even if you can get them.

I'm curious.

Say i want a Metro-E 10meg/10meg and something else (DS3 fractional or T-1) with one AS, router to handle that, and the smallest size portable IP set (what is it these days? like 4-8 c classes?) - real costs and time to install/get all the goods?

 

[Cooky]

I'll let OP share his thoughts too, but for me, all the items you mentioned are bare minimum if you want to run mission critical sites & apps.