• We’re currently investigating an issue related to the forum theme and styling that is impacting page layout and visual formatting. The problem has been identified, and we are actively working on a resolution. There is no impact to user data or functionality, this is strictly a front-end display issue. We’ll post an update once the fix has been deployed. Thanks for your patience while we get this sorted.

2 (web)Server's connected to two different ISPs at the same time for redundancy - ok spidey, bad idea!

W

wallsfd949

Golden Member
I may be over complicating this but I want to make sure I'm not "dreaming up" something that can't actually happen. We've been plauged by network connectivity troubles over the past year off and on with our current host due to other servers on the same network under DOS attacks. We fully intend to stay with them as they are very reputable and customer service is top notch.

I am told that we can get hooked up to two separate networks and two separate ISPs in the DC that we are in without much trouble. Our two servers have two NICs in each, but I'd like to add a 3rd. We have three domain names that will point to whatever server/network is live.

Here is my plan:

Each server will have connectivity to each ISP and will be hooked to each network. Both servers will respond on each interface to http/ftp/SSH requests running RedHat Enterprise Linux, Apache 2, and MySQL 4

The backup server will be directly connected to the primary server via a Gigabit NIC and X-over cable for fast transfers between the boxes and a 3rd way to gain entrance to the servers (since we don't have/need Vantage type consoles)

We will let DNSMadeEasy handle the monitoring and (cheap) failover of the servers through short TTLs and DNS records - no need for Veritas type failovers. It will check both sides to the primary box before dropping back to the backup server.

The database (MySQL) will run on the primary server with the backup server in replication/slave mode.

The 2nd power supply of the primary server will be connected to a different PDU in the DC so if one PDU fails the primary server will still be up.

I've done a lot to eliminate single points of failure with the obvious exception being geographic diversity.

My question is, is there a major problem with each server being connected to two ISPs at the same time?


I'll attach a simple diagram of what I think needs to happen.

:camera:




Update 3/29/06
see most recent post.
 
That's going to be very difficult to do.

A host can't really have two default gateways. This would be needed to send traffic back out.

Now if you had 2 routers you could then use those to send/receive traffic from both ISPs. You'd use two for redundancy.
 
Spidey is right, you can only *technically* have one default gateway for the box. Some of your options would be a multi-input router that you could have multiple input network sources heading into. Not sure if you're in a small office or a larger one, but this type of config even starts with the Cisco low-end product of Linksys RV082 series. Another option you could look at would be a type of load balancer hardware. I've worked with a few different types over the years, and have found a lot of companies using F5 in my particular industry, but there are many sources. The F5 load balancers effectively act as a router, switch and load balancer all in a neat little package. Do some searching for vendors in your area and pose them with your "hypothetical" scenario, you'd be surprised at the solutions they'll come up with if you could be a potential customer.
 
your cheapest best would be BGP routing, which you can pay a network engineer pretty cheap to program for you nowadays.

or

for more enterprise implementation...use fatpipe! it's the easiest thing I've ever used...seriously!

hope this helps!
 
I got the idea from several places, but wouldn't the IP tools (not ifconfig) that come with most modern distros work for interface routing? Here is one example I was thinking of:

lartc.org


Thanks for the input, I'd like to see where this discussion goes.
 
that links shows you doing this at layer3 (where eveyone has told you to do this). DOn't do this with your webserver itself, do it with your hardware dedicated to L3, as it's *drumroll please* designed to route and find the best path already!
 
Yeah BGP is really the only way to do this successfully. Even if you have short TTL's set with DNS caching and proxy servers and things like that some hosts may not check DNS for a new IP for 24 hours or so.
 
Originally posted by: nweaver
that links shows you doing this at layer3 (where eveyone has told you to do this). DOn't do this with your webserver itself, do it with your hardware dedicated to L3, as it's *drumroll please* designed to route and find the best path already!
Click to expand...


What would be the difference if I stick a Linux box infront of the webserver as the router than having the webserver do it? Is that not just adding one more Linux box that could be prone to failure?

The primary server is already redundant and that would just mean one more server to stick in the datacenter that could have problems.

If you were referring to not using Linux and using a piece of hardware, what hardware are you suggesting. I've heard of Foundry ServerIron but I'm not too familiar with it. I've been building smoothwall boxes (both .net and .org) that have the same ability to route between two ISPs so I believe Linux is very capable of doing this?.?.


 
Originally posted by: wallsfd949
Originally posted by: nweaver
that links shows you doing this at layer3 (where eveyone has told you to do this). DOn't do this with your webserver itself, do it with your hardware dedicated to L3, as it's *drumroll please* designed to route and find the best path already!
Click to expand...


What would be the difference if I stick a Linux box infront of the webserver as the router than having the webserver do it? Is that not just adding one more Linux box that could be prone to failure?

The primary server is already redundant and that would just mean one more server to stick in the datacenter that could have problems.

If you were referring to not using Linux and using a piece of hardware, what hardware are you suggesting. I've heard of Foundry ServerIron but I'm not too familiar with it. I've been building smoothwall boxes (both .net and .org) that have the same ability to route between two ISPs so I believe Linux is very capable of doing this?.?.
Click to expand...

I wouldn't do this on my server, as It's adding complexity to that single machine. Putting it on a dedicated machine would allow you to expand in the future with your needs. If your apache config is working, I wouldn't risk the crap trying to dual home that machine, I would do it on a dedicated linux machine. In fact, I would probably choose *bsd, as I think ipfw might do this better then iptables.
 
I'm curious what the host says about this problem. Did they suggest you set up the second ISP? Do they pay for the second ISP?
 
Originally posted by: RebateMonger
I'm curious what the host says about this problem. Did they suggest you set up the second ISP? Do they pay for the second ISP?
Click to expand...

That is part of the deal.

My original throughs were to setup the primary server with ISP1 and the backup server with ISP2. Then if there were to be a network outage with ISP1 it would failover to the backup server on ISP2. This will work fine, but the backup server is a Dual PIII 1.0 where the primary box is a Dual Xeon 3.4. The application will run slower but won't be down for 10-12 hrs like it was earlier this month. We can deal with a few customer calls about a slow app, but constant customer calls about a down site/app is much more stressful and not good for business.

My second thought was to connect both boxes to each ISP that way it would take a hardware failure to have to failover the app to the backup server. The 1st failover would be from ISP1 to ISP2 on the Primary server and then, if it was a hardware problem, it would be on to the Backup server. This would be ideal as a network outage/issue would not cause a performance decrease by failing to the slower server.

I don't think it would be in the budget to add additional hardware as the 1st plan will accomplish the goal, I was just hoping to have a hardware failure before we saw a performance decrease.

 
Unsolicited Comment:
It'd seem like the two prime reasons for selecting a web host would be:
1) Reliability
2) Cost
It seems like your host is giving you neither at this point.
And, you are getting irritated customers, which is even worse!
 
Originally posted by: RebateMonger
It'd seem like the two prime reasons for selecting a web host would be:
1) Reliability
2) Cost
It seems like your host is giving you neither at this point.
Click to expand...

You are correct in reasons, and if I gave the impression that our web host is not offering either of the two, I appologize.

The reliability issue was due to Denial of Service attacks on other machines on our segment of the network. Our host offered to resolve this by giving us a 2nd line of connectivity through a separate ISP and not charge us for it (only usage/bandwidth overage). This planned upgrade will give us two servers each with 900GB of available space in RAID5 w/hot spare and we did evaluate other hosts to compare pricing. Our current host was as competative (if not better) than other hosts and we've had tremendous customer service from our host over the past 4 years so we didn't see any advantage to switching.

The power supply failure that caused our 12 hour outage was not the fault of our host, I blame the folks who managed the servers before us for not putting that in the server spec. Our host built it to the exact specifications that the previous SA's specified which did not include a redundant power supply.


So back to the title of the thread...


I'd like to know if this can work. Not really how good will it work, but can it work. I'm not looking to load balance traffic, I just want requests from NIC1 to be answered on NIC1 and requests from NIC2 to be answered on NIC2.
 
won't work and a bad idea.

you need a way to route the return traffic from the websever.

As an aside your hosting company is incompetent if what you are describing is happening.

Look at it this way - you are pounding a square peg into a round hole.
 
Thanks for the advice. I'd like to ask for some recomendations if possible. This is what I do know:

- We have access to 2 networks/ISPs
- We will have 2 identical servers
- I don't really want to add a piece of hardware two the mix or NAT the servers

We can setup 1 server to 1 ISP and use DNS to failover between the two but that defeats DNS caching, etc..
I've heard that we can have each ISP able to use the same IP, but is that what has been talked about for BGP?

If we use BGP, does each server need 1 NIC per ISP?


Please understand, I'm not asking for someone to do my job for me 😛, but if you have some good links to doccumentation on what I'm trying to accomplish in a web/application server environment, that would be more than appreciated.

I see a lot of stuff about BGP choosing the 'best' route but not in a failover situation.
 
It is normally accompained like this.

ISP 1 goes into router 1 and BGP peers with ISP 1 router
ISP 2 goes into router 2 and BGP peers with ISP 2 router
router 1 and router 2 have a iBGP session between them
they also run VRRP or HSRP to provide a virtual IP address as a default gateway

There - you have redundancy and failover at layer3, you could even load balance.

Then you have a pair of switches and a pair of load balancers that send the requests to the appropriate web servers.

Your main problem is going to be DNS - it's not meant to be used as a failover mechanism because servers all over the world will still have the "bad" IP address. If you want to do it proper you use a single or pool of IP addresses for the site that are always reachible (and make sure they are always reachible via BGP) and then load balance or failover these IPs (via a load balancer) to a server farm or cluster.

It simply cannot work using DNS because clients will be making requests to an IP address that is not reachible.

And I should say it again - find another host, that is unacceptible these days.
 

I believe our host does have the capability of doing BGP4. He has mentioned to me something else called Anycast that he is testing in a non-production environment.

If I understand correctly, with BGP everything is setup at the routers and not at our server. This would make the two ISPs transparent to our servers.?.?


Thanks for all the input guys & gals -> have a great Friday/weekend!
 
updated with new information (see OP).


Update 3/29/06

After much lab testing and discussion with our host I have found some new information. It appears that the DOS attacks causing our service interruption were due to another box on our 'vlan' thus affecting our servers also.

I did find out that our host has several providers -> level 3, internap, cogent, and several others. Our IPs are already setup with BGP but it wouldn't have made a difference in this case because of where the DOS attacks were aimed.

Our new setup will have either server on a different Vlan with a shared pool of IPs using the linux-ha project which I've been heavily testing in our lab over the past two weeks.

From my understanding, I can bring up the shared IPs on either vlan just not at the same time. This should make a very reliable highly available webserver. Correct me if I'm wrong.

Thanks - Fireman

updated pic :camera:
 
We currently are doing this very same thing.
We have roughly 140 servers or so, 80 or so provide email.

Each server has two nics

Nic one is configured to connect to Primary ISP
Nic two is set to secondary ISP

We use SureSync on our Merek boxes

and have auto failover on the networks, one goes down the other picks up all network traffic.

The only downside is the server IP changes and some of our customers TTL is several hours, So they dont always route to the IP thats operational
 
You must log in or register to reply here.

TRENDING THREADS

Back
Top