Go Back   AnandTech Forums > Hardware and Technology > Networking

Forums
· Hardware and Technology
· CPUs and Overclocking
· Motherboards
· Video Cards and Graphics
· Memory and Storage
· Power Supplies
· Cases & Cooling
· SFF, Notebooks, Pre-Built/Barebones PCs
· Networking
· Peripherals
· General Hardware
· Highly Technical
· Computer Help
· Home Theater PCs
· Consumer Electronics
· Digital and Video Cameras
· Mobile Devices & Gadgets
· Audio/Video & Home Theater
· Software
· Software for Windows
· All Things Apple
· *nix Software
· Operating Systems
· Programming
· PC Gaming
· Console Gaming
· Distributed Computing
· Security
· Social
· Off Topic
· Politics and News
· Discussion Club
· Love and Relationships
· The Garage
· Health and Fitness
· Merchandise and Shopping
· For Sale/Trade
· Hot Deals with Free Stuff/Contests
· Black Friday 2014
· Forum Issues
· Technical Forum Issues
· Personal Forum Issues
· Suggestion Box
· Moderator Resources
· Moderator Discussions
   

Reply
 
Thread Tools
Old 12-24-2012, 05:20 AM   #1
Acanthus
Lifer
 
Acanthus's Avatar
 
Join Date: Aug 2001
Posts: 19,916
Default For my intended application... Why Cisco?

I am toying with the idea of starting a small datacenter featuring 4 x 10gb bonded lease lines and i'm eyeing the price tags on some of the Cisco routers that could handle this kind of switching load...

The price is prohibitive.

Is there a reason a home built 12-core windows server with 4 dual 10gbit cards couldn't do everything that a Cisco router would be able to do? I can't imagine that performance would be an issue with that much processing power on hand, is this underpowered or overpowered for this application? Would a properly configured Windows / CentOS / FreeBSD gateway perform as well as a high end Cisco router?
__________________
::Intel Core i7 2600K @ 5.0ghz (100x50) 1.400v ~ 47% performance increase
::2x4GB Mushkin DDR3-1333
::Gigabyte GA-P67A-UD3-B3
::XFX Radeon 6850 1GB
::OCZ Vertex 4 128GB
::Water Cooling - Swiftech Apogee GTX, Triple 120mm rad, 120GPH pump
Acanthus is offline   Reply With Quote
Old 12-24-2012, 05:55 AM   #2
theevilsharpie
Platinum Member
 
Join Date: Nov 2009
Location: Southern California
Posts: 2,314
Default

You want to route 40Gb/s through a general-purpose server?

Is this a troll thread?
theevilsharpie is offline   Reply With Quote
Old 12-24-2012, 09:41 AM   #3
Acanthus
Lifer
 
Acanthus's Avatar
 
Join Date: Aug 2001
Posts: 19,916
Default

Quote:
Originally Posted by theevilsharpie View Post
You want to route 40Gb/s through a general-purpose server?

Is this a troll thread?
Not at all. I do not have enough general knowledge about the performance of a PC vs a specialized router for this purpose.

I don't see a theoretical reason why it wouldn't work? The PCI-E bandwidth would be more than sufficient. Where would the issues lie? Would there be latency issues? Would the windows scheduler have any issues with so much data? Would dual 6 core Xeons be enough to handle the data load?
__________________
::Intel Core i7 2600K @ 5.0ghz (100x50) 1.400v ~ 47% performance increase
::2x4GB Mushkin DDR3-1333
::Gigabyte GA-P67A-UD3-B3
::XFX Radeon 6850 1GB
::OCZ Vertex 4 128GB
::Water Cooling - Swiftech Apogee GTX, Triple 120mm rad, 120GPH pump
Acanthus is offline   Reply With Quote
Old 12-24-2012, 11:16 AM   #4
RadiclDreamer
Diamond Member
 
RadiclDreamer's Avatar
 
Join Date: Aug 2004
Posts: 8,044
Default

Cisco is expensive not only for the hardware, but also for their software
__________________
CCENT, CCNA, A+, Net+
RadiclDreamer is offline   Reply With Quote
Old 12-24-2012, 11:18 AM   #5
debian0001
Senior Member
 
Join Date: Jun 2012
Posts: 344
Default

Look into Force10. Cheaper and just as good. We built our new datacenter using Force10.
debian0001 is offline   Reply With Quote
Old 12-24-2012, 01:43 PM   #6
xSauronx
Lifer
 
xSauronx's Avatar
 
Join Date: Jul 2000
Location: Eastern NC
Posts: 19,315
Default

Quote:
Originally Posted by Acanthus View Post
Not at all. I do not have enough general knowledge about the performance of a PC vs a specialized router for this purpose.

I don't see a theoretical reason why it wouldn't work? The PCI-E bandwidth would be more than sufficient. Where would the issues lie?
support.

you wanna call microsoft about a routing or switching issue when that thing loses its mind one day?

or redhat? or have to hit up some user forums?

even if you dont go cisco, support alone is going to be worth going with a trusted vendor with specific hardware made for something like this.
__________________
LET'S GO 'CANES [they arent going anywhere. ever.]
Free speech doesn't protect speech you like, it protects speech you hate.

Heatware |
xSauronx is offline   Reply With Quote
Old 12-24-2012, 01:57 PM   #7
OlafSicky
Platinum Member
 
OlafSicky's Avatar
 
Join Date: Feb 2011
Posts: 2,368
Default

No problem all you need is just an old PIII box intel cards and pfsense all could be had free
OlafSicky is offline   Reply With Quote
Old 12-24-2012, 02:40 PM   #8
Nothinman
Elite Member
 
Nothinman's Avatar
 
Join Date: Sep 2001
Posts: 30,672
Default

Quote:
Originally Posted by Acanthus View Post
I am toying with the idea of starting a small datacenter featuring 4 x 10gb bonded lease lines and i'm eyeing the price tags on some of the Cisco routers that could handle this kind of switching load...

The price is prohibitive.

Is there a reason a home built 12-core windows server with 4 dual 10gbit cards couldn't do everything that a Cisco router would be able to do? I can't imagine that performance would be an issue with that much processing power on hand, is this underpowered or overpowered for this application? Would a properly configured Windows / CentOS / FreeBSD gateway perform as well as a high end Cisco router?
Do you want routing or switching? They're different functions.

A home built router could possibly work, but it's hard to say for sure and if you use Windows, even Server Core, you open yourself up to a whole other world of issues. One major one being that you'll have a Windows machine directly on the Internet and practically every update still requires a reboot despite MS saying they're trying to fix that.

Given my recent experiences with TAC I would give RedHat or MS equal chanced of fixing any issues you run into, probably better chances than Cisco at this point. But if you're really leaning towards a home brew router I would definitely lean towards FreeBSD, Linux or even OpenBSD. Windows handles being multihomed pretty poorly and making routers out of free unix systems is really well documented.
__________________
http://www.debian.org
Nothinman is offline   Reply With Quote
Old 12-24-2012, 02:48 PM   #9
Acanthus
Lifer
 
Acanthus's Avatar
 
Join Date: Aug 2001
Posts: 19,916
Default

Quote:
Originally Posted by Nothinman View Post
Do you want routing or switching? They're different functions.

A home built router could possibly work, but it's hard to say for sure and if you use Windows, even Server Core, you open yourself up to a whole other world of issues. One major one being that you'll have a Windows machine directly on the Internet and practically every update still requires a reboot despite MS saying they're trying to fix that.

Given my recent experiences with TAC I would give RedHat or MS equal chanced of fixing any issues you run into, probably better chances than Cisco at this point. But if you're really leaning towards a home brew router I would definitely lean towards FreeBSD, Linux or even OpenBSD. Windows handles being multihomed pretty poorly and making routers out of free unix systems is really well documented.
I was strongly leaning toward OpenBSD for security reasons alone. Yes, I am thinking of routing and not switching.

What kind of CPU power would you need for heavy utilization of multiple 10gbit lines? I would just need a ballpark as in number of cores. I would certainly use jumbo frames to reduce the CPU load.
__________________
::Intel Core i7 2600K @ 5.0ghz (100x50) 1.400v ~ 47% performance increase
::2x4GB Mushkin DDR3-1333
::Gigabyte GA-P67A-UD3-B3
::XFX Radeon 6850 1GB
::OCZ Vertex 4 128GB
::Water Cooling - Swiftech Apogee GTX, Triple 120mm rad, 120GPH pump
Acanthus is offline   Reply With Quote
Old 12-24-2012, 02:53 PM   #10
Acanthus
Lifer
 
Acanthus's Avatar
 
Join Date: Aug 2001
Posts: 19,916
Default

Quote:
Originally Posted by xSauronx View Post
support.

you wanna call microsoft about a routing or switching issue when that thing loses its mind one day?

or redhat? or have to hit up some user forums?

even if you dont go cisco, support alone is going to be worth going with a trusted vendor with specific hardware made for something like this.
Understandable. I also considered a topology with two internet gateways utilizing 2x10gbit each. This would create a redundant environment where one gateway would fail and still have half of the total bandwidth available during downtime.

I am very very good with hardware but have very little real world enterprise level networking experience. Only classes and enough to get my CCENT.
__________________
::Intel Core i7 2600K @ 5.0ghz (100x50) 1.400v ~ 47% performance increase
::2x4GB Mushkin DDR3-1333
::Gigabyte GA-P67A-UD3-B3
::XFX Radeon 6850 1GB
::OCZ Vertex 4 128GB
::Water Cooling - Swiftech Apogee GTX, Triple 120mm rad, 120GPH pump
Acanthus is offline   Reply With Quote
Old 12-24-2012, 02:54 PM   #11
alkemyst
No Lifer
 
alkemyst's Avatar
 
Join Date: Feb 2001
Location: Corner of EPIC and ┼WESOME ST.
Posts: 80,843
Default

Cisco really owns the route / switch market for large needs. At lower needs you have a lot of options. However; 40GB is a large need IMHO. What's your real metrics?
__________________
The Masamune blade would repel the leaves and let them flow safely down the stream, while the Murasame blade would attract and cut them up.
alkemyst is offline   Reply With Quote
Old 12-24-2012, 03:10 PM   #12
Nothinman
Elite Member
 
Nothinman's Avatar
 
Join Date: Sep 2001
Posts: 30,672
Default

Have you looked at Juniper or Brocade? They tend to kill Cisco on price and in some cases provide better hardware.

I actually have no idea what kind of processing power is needed to route packets at those rates and other things come into play as well. Number of routes, firewall rules, DPI, etc. And you won't get any useful numbers from Cisco because they use ASICs for much of that which break comparisons.
__________________
http://www.debian.org
Nothinman is offline   Reply With Quote
Old 12-24-2012, 04:43 PM   #13
theevilsharpie
Platinum Member
 
Join Date: Nov 2009
Location: Southern California
Posts: 2,314
Default

Quote:
Originally Posted by Acanthus View Post
I don't see a theoretical reason why it wouldn't work? The PCI-E bandwidth would be more than sufficient. Where would the issues lie? Would there be latency issues? Would the windows scheduler have any issues with so much data? Would dual 6 core Xeons be enough to handle the data load?
The bottleneck in this case would be the operating system's TCP/IP stack. I don't know of any general-purpose OS that is capable of routing at 40Gb/s in software. It would probably work, but you wouldn't get the performance you expect.
theevilsharpie is offline   Reply With Quote
Old 12-24-2012, 05:48 PM   #14
Nothinman
Elite Member
 
Nothinman's Avatar
 
Join Date: Sep 2001
Posts: 30,672
Default

http://www.myricom.com/scs/performance/Myri10GE/

Assuming that scales well and all, it implies that you could get 40Gb out of any OS without much CPU utilization. Of course a home brew build is going to require more knowledge, testing, etc up front and possibly more support costs. But on the other side, if you build it yourself you'll better understand how it works and require less support calls because you'll be able to fix issues yourself.

So a Cisco, Juniper or Brocade device would most likely cost less up front but support and contacts may cost more in the long run.

But all of that is outsider speculation as we have no idea about the knowledge of your engineers and their cost to your company. Chances are the cheaper solution would be the branded appliance one, but if this is going to be a core service it may be worth it to put in the engineering time to build your own infrastructure at least for some portions.
__________________
http://www.debian.org
Nothinman is offline   Reply With Quote
Old 12-24-2012, 05:51 PM   #15
cmetz
Platinum Member
 
Join Date: Nov 2001
Posts: 2,296
Default

Quote:
Originally Posted by theevilsharpie View Post
Is this a troll thread?
+1.

If you're seriously buying 4x10G data center handoff circuits, you can spend the money for a real router, and for people who know how to build real networks. Yes, it doesn't have to be Cisco (and I'd look at other vendors personally for this particular need), but you fundamentally need real gear and a real network engineer.

OP, if you meant the question seriously, please take away that it's not just a bad idea, but such a bad idea that it is absurd enough to sound like a troll. There are technical reasons why this is a bad idea, but the real issues are at layers 8 & 9.

Last edited by cmetz; 12-24-2012 at 06:02 PM.
cmetz is offline   Reply With Quote
Old 12-24-2012, 06:19 PM   #16
Acanthus
Lifer
 
Acanthus's Avatar
 
Join Date: Aug 2001
Posts: 19,916
Default

Quote:
Originally Posted by cmetz View Post
+1.

If you're seriously buying 4x10G data center handoff circuits, you can spend the money for a real router, and for people who know how to build real networks. Yes, it doesn't have to be Cisco (and I'd look at other vendors personally for this particular need), but you fundamentally need real gear and a real network engineer.

OP, if you meant the question seriously, please take away that it's not just a bad idea, but such a bad idea that it is absurd enough to sound like a troll. There are technical reasons why this is a bad idea, but the real issues are at layers 8 & 9.
This explained nothing. Just that there are "technical issues", and then went on to insult my knowledge level.

I'm trying to discern the difference between a highly customized server for this purpose, and a router that is sold at what appears to be a 10000% markup.
__________________
::Intel Core i7 2600K @ 5.0ghz (100x50) 1.400v ~ 47% performance increase
::2x4GB Mushkin DDR3-1333
::Gigabyte GA-P67A-UD3-B3
::XFX Radeon 6850 1GB
::OCZ Vertex 4 128GB
::Water Cooling - Swiftech Apogee GTX, Triple 120mm rad, 120GPH pump
Acanthus is offline   Reply With Quote
Old 12-24-2012, 06:25 PM   #17
drebo
Diamond Member
 
Join Date: Feb 2006
Posts: 6,552
Default

40gb/s out of a general purpose CPU? Probably not.

I mean, you could certainly set it up and use BIRD or quagga to establish your routing protocols, but those still rely on the OS's underlying network stack for the actual routing of packets (read: iptables) which means it's going to perform like shit.

Cisco's ISR G2s are x86 CPUs and route on the hundreds of megabits level with purpose-built software. Juniper's J-series are also x86 based and have specialized routing software. J6350s top out at about 2gbps of IMIX traffic when they have two full tables worth of routes.

4x 10gb feeds with 4x full tables is a lot to ask out of two routers. That said, you might be able to pick up a couple 6503s with Sup32-10Gs pretty cheap. They won't support full tables, though. But that'd be your cheapest Cisco 10gb connections.

On Juniper's side, the MX80-48T would be your best bet. Two of those would be fine, and they'll support a couple of 10gb connections each.

Seriously, though, a "small DC" is not 4x 10gb. A "small DC" is one or two 1gb circuits. If you really need 40gb/s, that's a pretty fucking big DC and is going to need a LOT more than any homebrew router is going to get you.

Also, consider your customers. I sure as shit wouldn't use a DC that had homebrew linux-based routers at its edge or its core.
__________________
"All men are not created equal, and if you believe they are, there's something seriously wrong with you. Some men are destined for greatness. Most aren't. End of story." - Jose Canseco
drebo is offline   Reply With Quote
Old 12-24-2012, 06:26 PM   #18
Acanthus
Lifer
 
Acanthus's Avatar
 
Join Date: Aug 2001
Posts: 19,916
Default

Quote:
Originally Posted by Nothinman View Post
http://www.myricom.com/scs/performance/Myri10GE/

Assuming that scales well and all, it implies that you could get 40Gb out of any OS without much CPU utilization. Of course a home brew build is going to require more knowledge, testing, etc up front and possibly more support costs. But on the other side, if you build it yourself you'll better understand how it works and require less support calls because you'll be able to fix issues yourself.

So a Cisco, Juniper or Brocade device would most likely cost less up front but support and contacts may cost more in the long run.

But all of that is outsider speculation as we have no idea about the knowledge of your engineers and their cost to your company. Chances are the cheaper solution would be the branded appliance one, but if this is going to be a core service it may be worth it to put in the engineering time to build your own infrastructure at least for some portions.
This is the kind of information i was looking for. I have ample time to beta test and configure these servers myself should i put them together... I just need to know if I am wasting my time. If i can get the throughput and maintain a secure environment for literally a fraction of the cost, it is worth it to me to pursue this idea.
__________________
::Intel Core i7 2600K @ 5.0ghz (100x50) 1.400v ~ 47% performance increase
::2x4GB Mushkin DDR3-1333
::Gigabyte GA-P67A-UD3-B3
::XFX Radeon 6850 1GB
::OCZ Vertex 4 128GB
::Water Cooling - Swiftech Apogee GTX, Triple 120mm rad, 120GPH pump
Acanthus is offline   Reply With Quote
Old 12-24-2012, 06:27 PM   #19
VirtualLarry
Lifer
 
VirtualLarry's Avatar
 
Join Date: Aug 2001
Posts: 25,663
Default

Quote:
Originally Posted by Acanthus View Post
I'm trying to discern the difference between a highly customized server for this purpose, and a router that is sold at what appears to be a 10000% markup.
Well, one difference is that the router appliance should "just work". Whereas a home-brewed solution may have unforseen issues (*).


(*) Not that routing appliances don't have bugs too. But vendors have trained engineers to solve those problems, and their R&D budget is funded by economies of scale of sales of those products. Do you have an equivalent R&D budget for a one-off project?
__________________
Rig(s) not listed, because I change computers, like some people change their socks.
ATX is for poor people. And 'gamers.' - phucheneh
haswell is bulldozer... - aigomorla
"DON'T BUY INTEL, they will send secret signals down the internet, which
will considerably slow down your computer". - SOFTengCOMPelec
VirtualLarry is offline   Reply With Quote
Old 12-24-2012, 06:33 PM   #20
Acanthus
Lifer
 
Acanthus's Avatar
 
Join Date: Aug 2001
Posts: 19,916
Default

Quote:
Originally Posted by drebo View Post
40gb/s out of a general purpose CPU? Probably not.

I mean, you could certainly set it up and use BIRD or quagga to establish your routing protocols, but those still rely on the OS's underlying network stack for the actual routing of packets (read: iptables) which means it's going to perform like shit.

Cisco's ISR G2s are x86 CPUs and route on the hundreds of megabits level with purpose-built software. Juniper's J-series are also x86 based and have specialized routing software. J6350s top out at about 2gbps of IMIX traffic when they have two full tables worth of routes.

4x 10gb feeds with 4x full tables is a lot to ask out of two routers. That said, you might be able to pick up a couple 6503s with Sup32-10Gs pretty cheap. They won't support full tables, though. But that'd be your cheapest Cisco 10gb connections.

On Juniper's side, the MX80-48T would be your best bet. Two of those would be fine, and they'll support a couple of 10gb connections each.

Seriously, though, a "small DC" is not 4x 10gb. A "small DC" is one or two 1gb circuits. If you really need 40gb/s, that's a pretty fucking big DC and is going to need a LOT more than any homebrew router is going to get you.

Also, consider your customers. I sure as shit wouldn't use a DC that had homebrew linux-based routers at its edge or its core.
This "small" DC would be a physically small facility that is already lit and would only serve very specific customers with very high bandwidth demands.
__________________
::Intel Core i7 2600K @ 5.0ghz (100x50) 1.400v ~ 47% performance increase
::2x4GB Mushkin DDR3-1333
::Gigabyte GA-P67A-UD3-B3
::XFX Radeon 6850 1GB
::OCZ Vertex 4 128GB
::Water Cooling - Swiftech Apogee GTX, Triple 120mm rad, 120GPH pump
Acanthus is offline   Reply With Quote
Old 12-24-2012, 06:47 PM   #21
Nothinman
Elite Member
 
Nothinman's Avatar
 
Join Date: Sep 2001
Posts: 30,672
Default

Quote:
Originally Posted by drebo View Post
Also, consider your customers. I sure as shit wouldn't use a DC that had homebrew linux-based routers at its edge or its core.
So you go on a physical walkthrough of every DC for every "cloud" service and make sure they're using "approved" brand name hardware?

Quote:
Originally Posted by VirtualLarry
Well, one difference is that the router appliance should "just work". Whereas a home-brewed solution may have unforseen issues (*).
At the pure routing level any OS (except probably Windows) works with a few settings. For example, setting up a few interfaces and iptables to allow the routing is like ~5min per-interface and will be just as trouble-free as IOS and allow for a lot more flexibility with infinitely less licensing costs.

Quote:
Originally Posted by VirtualLarry
(*) Not that routing appliances don't have bugs too. But vendors have trained engineers to solve those problems, and their R&D budget is funded by economies of scale of sales of those products. Do you have an equivalent R&D budget for a one-off project?
That's what they tell you, but from my experiences with Cisco TAC recently it's blatantly obvious that's not true. Maybe 1 out of 5 engineers that you get truly understands what's going on, but the rest of the time you get a random minimum wage engineer that barely understands what's in their script.
__________________
http://www.debian.org
Nothinman is offline   Reply With Quote
Old 12-24-2012, 08:06 PM   #22
theevilsharpie
Platinum Member
 
Join Date: Nov 2009
Location: Southern California
Posts: 2,314
Default

Quote:
Originally Posted by Acanthus View Post
I'm trying to discern the difference between a highly customized server for this purpose, and a router that is sold at what appears to be a 10000% markup.
The router is going to be switching packets in hardware.
theevilsharpie is offline   Reply With Quote
Old 12-24-2012, 08:23 PM   #23
Nothinman
Elite Member
 
Nothinman's Avatar
 
Join Date: Sep 2001
Posts: 30,672
Default

Quote:
Originally Posted by theevilsharpie View Post
The router is going to be switching packets in hardware.
And? I really doubt a few of those ASICs plus SmartNet are worth what Cisco charges.

I'm not saying a DIY solution is obviously the best, but if people like Bill Hewlett, Steve Jobs, Bill Gates, Linus Torvalds, etc all just took the de facto answers instead of building what they thought made the most sense from scratch where would we be today?
__________________
http://www.debian.org
Nothinman is offline   Reply With Quote
Old 12-24-2012, 08:26 PM   #24
alkemyst
No Lifer
 
alkemyst's Avatar
 
Join Date: Feb 2001
Location: Corner of EPIC and ┼WESOME ST.
Posts: 80,843
Default

Quote:
Originally Posted by Nothinman View Post
And? I really doubt a few of those ASICs plus SmartNet are worth what Cisco charges.

I'm not saying a DIY solution is obviously the best, but if people like Bill Hewlett, Steve Jobs, Bill Gates, Linus Torvalds, etc all just took the de facto answers instead of building what they thought made the most sense from scratch where would we be today?
Lol
__________________
The Masamune blade would repel the leaves and let them flow safely down the stream, while the Murasame blade would attract and cut them up.
alkemyst is offline   Reply With Quote
Old 12-24-2012, 09:40 PM   #25
Nothinman
Elite Member
 
Nothinman's Avatar
 
Join Date: Sep 2001
Posts: 30,672
Default

Quote:
Originally Posted by alkemyst View Post
Lol
Thanks for the ever-so-meaningful input on the subject.

If you don't want to be helpful, just go away.
__________________
http://www.debian.org
Nothinman is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -5. The time now is 05:23 AM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.