For my intended application... Why Cisco?

[Acanthus]

I am toying with the idea of starting a small datacenter featuring 4 x 10gb bonded lease lines and i'm eyeing the price tags on some of the Cisco routers that could handle this kind of switching load...

The price is prohibitive.

Is there a reason a home built 12-core windows server with 4 dual 10gbit cards couldn't do everything that a Cisco router would be able to do? I can't imagine that performance would be an issue with that much processing power on hand, is this underpowered or overpowered for this application? Would a properly configured Windows / CentOS / FreeBSD gateway perform as well as a high end Cisco router?

 

[theevilsharpie]

You want to route 40Gb/s through a general-purpose server?

Is this a troll thread?

 

[Acanthus]

You want to route 40Gb/s through a general-purpose server?

Is this a troll thread?

Not at all. I do not have enough general knowledge about the performance of a PC vs a specialized router for this purpose.

I don't see a theoretical reason why it wouldn't work? The PCI-E bandwidth would be more than sufficient. Where would the issues lie? Would there be latency issues? Would the windows scheduler have any issues with so much data? Would dual 6 core Xeons be enough to handle the data load?

 

[RadiclDreamer]

Cisco is expensive not only for the hardware, but also for their software

 

[debian0001]

Look into Force10. Cheaper and just as good. We built our new datacenter using Force10.

 

[xSauronx]

Not at all. I do not have enough general knowledge about the performance of a PC vs a specialized router for this purpose.

I don't see a theoretical reason why it wouldn't work? The PCI-E bandwidth would be more than sufficient. Where would the issues lie?

support.

you wanna call microsoft about a routing or switching issue when that thing loses its mind one day?

or redhat? or have to hit up some user forums?

even if you dont go cisco, support alone is going to be worth going with a trusted vendor with specific hardware made for something like this.

 

[OlafSicky]

No problem all you need is just an old PIII box intel cards and pfsense all could be had free

 

[Nothinman]

I am toying with the idea of starting a small datacenter featuring 4 x 10gb bonded lease lines and i'm eyeing the price tags on some of the Cisco routers that could handle this kind of switching load...

The price is prohibitive.

Is there a reason a home built 12-core windows server with 4 dual 10gbit cards couldn't do everything that a Cisco router would be able to do? I can't imagine that performance would be an issue with that much processing power on hand, is this underpowered or overpowered for this application? Would a properly configured Windows / CentOS / FreeBSD gateway perform as well as a high end Cisco router?

Do you want routing or switching? They're different functions.

A home built router could possibly work, but it's hard to say for sure and if you use Windows, even Server Core, you open yourself up to a whole other world of issues. One major one being that you'll have a Windows machine directly on the Internet and practically every update still requires a reboot despite MS saying they're trying to fix that.

Given my recent experiences with TAC I would give RedHat or MS equal chanced of fixing any issues you run into, probably better chances than Cisco at this point. But if you're really leaning towards a home brew router I would definitely lean towards FreeBSD, Linux or even OpenBSD. Windows handles being multihomed pretty poorly and making routers out of free unix systems is really well documented.

 

[Acanthus]

Do you want routing or switching? They're different functions.

A home built router could possibly work, but it's hard to say for sure and if you use Windows, even Server Core, you open yourself up to a whole other world of issues. One major one being that you'll have a Windows machine directly on the Internet and practically every update still requires a reboot despite MS saying they're trying to fix that.

Given my recent experiences with TAC I would give RedHat or MS equal chanced of fixing any issues you run into, probably better chances than Cisco at this point. But if you're really leaning towards a home brew router I would definitely lean towards FreeBSD, Linux or even OpenBSD. Windows handles being multihomed pretty poorly and making routers out of free unix systems is really well documented.

I was strongly leaning toward OpenBSD for security reasons alone. Yes, I am thinking of routing and not switching.

What kind of CPU power would you need for heavy utilization of multiple 10gbit lines? I would just need a ballpark as in number of cores. I would certainly use jumbo frames to reduce the CPU load.

 

[Acanthus]

support.

you wanna call microsoft about a routing or switching issue when that thing loses its mind one day?

or redhat? or have to hit up some user forums?

even if you dont go cisco, support alone is going to be worth going with a trusted vendor with specific hardware made for something like this.

Understandable. I also considered a topology with two internet gateways utilizing 2x10gbit each. This would create a redundant environment where one gateway would fail and still have half of the total bandwidth available during downtime.

I am very very good with hardware but have very little real world enterprise level networking experience. Only classes and enough to get my CCENT.

 

[alkemyst]

Cisco really owns the route / switch market for large needs. At lower needs you have a lot of options. However; 40GB is a large need IMHO. What's your real metrics?

 

[Nothinman]

Have you looked at Juniper or Brocade? They tend to kill Cisco on price and in some cases provide better hardware.

I actually have no idea what kind of processing power is needed to route packets at those rates and other things come into play as well. Number of routes, firewall rules, DPI, etc. And you won't get any useful numbers from Cisco because they use ASICs for much of that which break comparisons.

 

[theevilsharpie]

I don't see a theoretical reason why it wouldn't work? The PCI-E bandwidth would be more than sufficient. Where would the issues lie? Would there be latency issues? Would the windows scheduler have any issues with so much data? Would dual 6 core Xeons be enough to handle the data load?

The bottleneck in this case would be the operating system's TCP/IP stack. I don't know of any general-purpose OS that is capable of routing at 40Gb/s in software. It would probably work, but you wouldn't get the performance you expect.

 

[Nothinman]

http://www.myricom.com/scs/performance/Myri10GE/

Assuming that scales well and all, it implies that you could get 40Gb out of any OS without much CPU utilization. Of course a home brew build is going to require more knowledge, testing, etc up front and possibly more support costs. But on the other side, if you build it yourself you'll better understand how it works and require less support calls because you'll be able to fix issues yourself.

So a Cisco, Juniper or Brocade device would most likely cost less up front but support and contacts may cost more in the long run.

But all of that is outsider speculation as we have no idea about the knowledge of your engineers and their cost to your company. Chances are the cheaper solution would be the branded appliance one, but if this is going to be a core service it may be worth it to put in the engineering time to build your own infrastructure at least for some portions.

 

[cmetz]

Is this a troll thread?

+1.

If you're seriously buying 4x10G data center handoff circuits, you can spend the money for a real router, and for people who know how to build real networks. Yes, it doesn't have to be Cisco (and I'd look at other vendors personally for this particular need), but you fundamentally need real gear and a real network engineer.

OP, if you meant the question seriously, please take away that it's not just a bad idea, but such a bad idea that it is absurd enough to sound like a troll. There are technical reasons why this is a bad idea, but the real issues are at layers 8 & 9.

 

[Acanthus]

+1.

If you're seriously buying 4x10G data center handoff circuits, you can spend the money for a real router, and for people who know how to build real networks. Yes, it doesn't have to be Cisco (and I'd look at other vendors personally for this particular need), but you fundamentally need real gear and a real network engineer.

OP, if you meant the question seriously, please take away that it's not just a bad idea, but such a bad idea that it is absurd enough to sound like a troll. There are technical reasons why this is a bad idea, but the real issues are at layers 8 & 9.

This explained nothing. Just that there are "technical issues", and then went on to insult my knowledge level.

I'm trying to discern the difference between a highly customized server for this purpose, and a router that is sold at what appears to be a 10000% markup.

 

[drebo]

40gb/s out of a general purpose CPU? Probably not.

I mean, you could certainly set it up and use BIRD or quagga to establish your routing protocols, but those still rely on the OS's underlying network stack for the actual routing of packets (read: iptables) which means it's going to perform like shit.

Cisco's ISR G2s are x86 CPUs and route on the hundreds of megabits level with purpose-built software. Juniper's J-series are also x86 based and have specialized routing software. J6350s top out at about 2gbps of IMIX traffic when they have two full tables worth of routes.

4x 10gb feeds with 4x full tables is a lot to ask out of two routers. That said, you might be able to pick up a couple 6503s with Sup32-10Gs pretty cheap. They won't support full tables, though. But that'd be your cheapest Cisco 10gb connections.

On Juniper's side, the MX80-48T would be your best bet. Two of those would be fine, and they'll support a couple of 10gb connections each.

Seriously, though, a "small DC" is not 4x 10gb. A "small DC" is one or two 1gb circuits. If you really need 40gb/s, that's a pretty fucking big DC and is going to need a LOT more than any homebrew router is going to get you.

Also, consider your customers. I sure as shit wouldn't use a DC that had homebrew linux-based routers at its edge or its core.

 

[Acanthus]

http://www.myricom.com/scs/performance/Myri10GE/

Assuming that scales well and all, it implies that you could get 40Gb out of any OS without much CPU utilization. Of course a home brew build is going to require more knowledge, testing, etc up front and possibly more support costs. But on the other side, if you build it yourself you'll better understand how it works and require less support calls because you'll be able to fix issues yourself.

So a Cisco, Juniper or Brocade device would most likely cost less up front but support and contacts may cost more in the long run.

But all of that is outsider speculation as we have no idea about the knowledge of your engineers and their cost to your company. Chances are the cheaper solution would be the branded appliance one, but if this is going to be a core service it may be worth it to put in the engineering time to build your own infrastructure at least for some portions.

This is the kind of information i was looking for. I have ample time to beta test and configure these servers myself should i put them together... I just need to know if I am wasting my time. If i can get the throughput and maintain a secure environment for literally a fraction of the cost, it is worth it to me to pursue this idea.

 

[VirtualLarry]

I'm trying to discern the difference between a highly customized server for this purpose, and a router that is sold at what appears to be a 10000% markup.

Well, one difference is that the router appliance should "just work". Whereas a home-brewed solution may have unforseen issues (*).


(*) Not that routing appliances don't have bugs too. But vendors have trained engineers to solve those problems, and their R&D budget is funded by economies of scale of sales of those products. Do you have an equivalent R&D budget for a one-off project?

 

[Acanthus]

40gb/s out of a general purpose CPU? Probably not.

I mean, you could certainly set it up and use BIRD or quagga to establish your routing protocols, but those still rely on the OS's underlying network stack for the actual routing of packets (read: iptables) which means it's going to perform like shit.

Cisco's ISR G2s are x86 CPUs and route on the hundreds of megabits level with purpose-built software. Juniper's J-series are also x86 based and have specialized routing software. J6350s top out at about 2gbps of IMIX traffic when they have two full tables worth of routes.

4x 10gb feeds with 4x full tables is a lot to ask out of two routers. That said, you might be able to pick up a couple 6503s with Sup32-10Gs pretty cheap. They won't support full tables, though. But that'd be your cheapest Cisco 10gb connections.

On Juniper's side, the MX80-48T would be your best bet. Two of those would be fine, and they'll support a couple of 10gb connections each.

Seriously, though, a "small DC" is not 4x 10gb. A "small DC" is one or two 1gb circuits. If you really need 40gb/s, that's a pretty fucking big DC and is going to need a LOT more than any homebrew router is going to get you.

Also, consider your customers. I sure as shit wouldn't use a DC that had homebrew linux-based routers at its edge or its core.

This "small" DC would be a physically small facility that is already lit and would only serve very specific customers with very high bandwidth demands.

 

[Nothinman]

Also, consider your customers. I sure as shit wouldn't use a DC that had homebrew linux-based routers at its edge or its core.

So you go on a physical walkthrough of every DC for every "cloud" service and make sure they're using "approved" brand name hardware?

Well, one difference is that the router appliance should "just work". Whereas a home-brewed solution may have unforseen issues (*).

At the pure routing level any OS (except probably Windows) works with a few settings. For example, setting up a few interfaces and iptables to allow the routing is like ~5min per-interface and will be just as trouble-free as IOS and allow for a lot more flexibility with infinitely less licensing costs.

(*) Not that routing appliances don't have bugs too. But vendors have trained engineers to solve those problems, and their R&D budget is funded by economies of scale of sales of those products. Do you have an equivalent R&D budget for a one-off project?

That's what they tell you, but from my experiences with Cisco TAC recently it's blatantly obvious that's not true. Maybe 1 out of 5 engineers that you get truly understands what's going on, but the rest of the time you get a random minimum wage engineer that barely understands what's in their script.

 

[theevilsharpie]

I'm trying to discern the difference between a highly customized server for this purpose, and a router that is sold at what appears to be a 10000% markup.

The router is going to be switching packets in hardware.

 

[Nothinman]

The router is going to be switching packets in hardware.

And? I really doubt a few of those ASICs plus SmartNet are worth what Cisco charges.

I'm not saying a DIY solution is obviously the best, but if people like Bill Hewlett, Steve Jobs, Bill Gates, Linus Torvalds, etc all just took the de facto answers instead of building what they thought made the most sense from scratch where would we be today?

 

[alkemyst]

And? I really doubt a few of those ASICs plus SmartNet are worth what Cisco charges.

I'm not saying a DIY solution is obviously the best, but if people like Bill Hewlett, Steve Jobs, Bill Gates, Linus Torvalds, etc all just took the de facto answers instead of building what they thought made the most sense from scratch where would we be today?

Lol

 

[Nothinman]

Lol

Thanks for the ever-so-meaningful input on the subject.

If you don't want to be helpful, just go away.