Go Back   AnandTech Forums > Hardware and Technology > Networking

Forums
· Hardware and Technology
· CPUs and Overclocking
· Motherboards
· Video Cards and Graphics
· Memory and Storage
· Power Supplies
· Cases & Cooling
· SFF, Notebooks, Pre-Built/Barebones PCs
· Networking
· Peripherals
· General Hardware
· Highly Technical
· Computer Help
· Home Theater PCs
· Consumer Electronics
· Digital and Video Cameras
· Mobile Devices & Gadgets
· Audio/Video & Home Theater
· Software
· Software for Windows
· All Things Apple
· *nix Software
· Operating Systems
· Programming
· PC Gaming
· Console Gaming
· Distributed Computing
· Security
· Social
· Off Topic
· Politics and News
· Discussion Club
· Love and Relationships
· The Garage
· Health and Fitness
· Merchandise and Shopping
· For Sale/Trade
· Hot Deals with Free Stuff/Contests
· Black Friday 2014
· Forum Issues
· Technical Forum Issues
· Personal Forum Issues
· Suggestion Box
· Moderator Resources
· Moderator Discussions
   

Reply
 
Thread Tools
Old 12-25-2012, 01:18 AM   #26
drebo
Diamond Member
 
Join Date: Feb 2006
Posts: 6,565
Default

Quote:
Originally Posted by Nothinman View Post
So you go on a physical walkthrough of every DC for every "cloud" service and make sure they're using "approved" brand name hardware?
If I'm colocating there, yes.

If I'm not colocating, but rather using a hosted service, then I just make sure my contract has an SLA. If it does and the terms of the SLA are appropriate, then I don't care what kind of hardware they run.
__________________
"All men are not created equal, and if you believe they are, there's something seriously wrong with you. Some men are destined for greatness. Most aren't. End of story." - Jose Canseco
drebo is offline   Reply With Quote
Old 12-25-2012, 01:29 AM   #27
Acanthus
Lifer
 
Acanthus's Avatar
 
Join Date: Aug 2001
Posts: 19,916
Default

Quote:
Originally Posted by theevilsharpie View Post
The router is going to be switching packets in hardware.
Shorter hops? More throughput?

That is not a distinct advantage unless it gets the job done better.

According to other sources in this thread, a single 6-core xeon can handle the forwarding for two 10gbe connections with less than 50% utilization in OpenBSD.

I'm just trying to find the hole between theory and practice.
__________________
::Intel Core i7 2600K @ 5.0ghz (100x50) 1.400v ~ 47% performance increase
::2x4GB Mushkin DDR3-1333
::Gigabyte GA-P67A-UD3-B3
::XFX Radeon 6850 1GB
::OCZ Vertex 4 128GB
::Water Cooling - Swiftech Apogee GTX, Triple 120mm rad, 120GPH pump
Acanthus is offline   Reply With Quote
Old 12-25-2012, 08:12 AM   #28
theevilsharpie
Platinum Member
 
Join Date: Nov 2009
Location: Southern California
Posts: 2,316
Default

Quote:
Originally Posted by Acanthus View Post
Shorter hops? More throughput?

That is not a distinct advantage unless it gets the job done better.

According to other sources in this thread, a single 6-core xeon can handle the forwarding for two 10gbe connections with less than 50% utilization in OpenBSD.

I'm just trying to find the hole between theory and practice.
The compute resources required to route a packet depends on its size, and this can cause the throughput to vary quite a bit. This is why routing performance is measured in packets per second, not bandwidth.

A modern high-end x86 processor has enough power and memory bandwidth to handle several million packets per second. This is plenty for <10Gb routing, but at 40Gb/s, you're going to have a very difficult time keeping up with the traffic, particularly if you have to handle multiple packet sizes. Best case, smaller packets degrade your speed, worst case (i.e., DDoS) your router can completely lock up. And that's just with extremely simple routing activity; you can forget about things like ACLs, traffic shaping, VRRP, etc.

Before you say "throw more processors at it!," Linux's (and I assume BSD's) TCP/IP stack can scale with additional processors, but only to a point. NICs have a limited number of queues, and that will necessarily limit how many cores can be assigned to a particular NIC. Also, having a single fast multi-core CPU is the ideal case for software routing, as multiple processors add NUMA-related headaches that can easily decrease performance if things aren't tuned just right.

A few years ago, a Linux kernel developer gave a presentation about using Linux as a bi-directional 10GbE router, and while it worked in that role for larger packet sizes, performance didn't scale when adding 10GbE links, and performance tanked with smaller packet sizes. Granted, server hardware has improved since then, but not enough to assure line-rate routing performance at >10GbE.
theevilsharpie is offline   Reply With Quote
Old 12-25-2012, 10:20 AM   #29
alkemyst
No Lifer
 
alkemyst's Avatar
 
Join Date: Feb 2001
Location: Corner of EPIC and ┼WESOME ST.
Posts: 81,174
Default

For routing usually a PC can handle many needs (hence the real reason why Cisco doesn't like Dynamips/GNS3).

For switching needs ASICS > CPU.
__________________
The Masamune blade would repel the leaves and let them flow safely down the stream, while the Murasame blade would attract and cut them up.
alkemyst is online now   Reply With Quote
Old 12-25-2012, 10:59 PM   #30
drebo
Diamond Member
 
Join Date: Feb 2006
Posts: 6,565
Default

Quote:
Originally Posted by alkemyst View Post
For routing usually a PC can handle many needs (hence the real reason why Cisco doesn't like Dynamips/GNS3).

For switching needs ASICS > CPU.
Dynamips tops out at about 1k PPS, even on high end PCs. It'll never be a threat to Cisco equipment.
__________________
"All men are not created equal, and if you believe they are, there's something seriously wrong with you. Some men are destined for greatness. Most aren't. End of story." - Jose Canseco
drebo is offline   Reply With Quote
Old 12-25-2012, 11:05 PM   #31
imagoon
Diamond Member
 
imagoon's Avatar
 
Join Date: Feb 2003
Location: Chicagoland, IL
Posts: 4,713
Default

Dynamips / GNS3 can't give you the route performance that a native cisco router it emulates can. Hell it chugs just doing router# ?

The main issue is going to be CPU limitations. From what I have seen, something like PFSense (BSD based) starts to have difficulty getting much above 1Gb/s route doing open /24 to /24. Once you add ACL's and other rules the performance starts to floor out. Given that is still a good little app for lower needs.

Also "Cisco doesn't like Dynamips/GNS3" is pretty funny because they often use it for their classes and most of the CCIE's I have dealt with are happy to use it to do testing and will even slap their CCIE # on it. Dynamips at the moment is a dead project looking for a keeper. I am pretty sure that Cisco will be happy to ignore it since it can't touch the performance of a 1721 yet.

--edit--

On topic, I would go Cisco / Juniper / Any other dedicated vendor before I would do a server based router. ASIC based >>>>> generic CPU once you start getting above 1 gb/s routing (or lower if you like ACLs etc.)

Last edited by imagoon; 12-25-2012 at 11:09 PM.
imagoon is offline   Reply With Quote
Old 12-26-2012, 04:54 AM   #32
freegeeks
Diamond Member
 
freegeeks's Avatar
 
Join Date: May 2001
Posts: 5,453
Default

Routerboard Mikrotik CCR1036-12G-4S
16 Gbit / 24 million pps routing for $1000 dollar
don't bother with building your own box

http://routerboard.com/CCR1036-12G-4S

Last edited by freegeeks; 12-26-2012 at 04:59 AM.
freegeeks is offline   Reply With Quote
Old 12-26-2012, 06:07 AM   #33
theevilsharpie
Platinum Member
 
Join Date: Nov 2009
Location: Southern California
Posts: 2,316
Default

Quote:
Originally Posted by freegeeks View Post
Routerboard Mikrotik CCR1036-12G-4S
16 Gbit / 24 million pps routing for $1000 dollar
don't bother with building your own box

http://routerboard.com/CCR1036-12G-4S
Those are GbE interfaces, not 10GbE.
theevilsharpie is offline   Reply With Quote
Old 12-26-2012, 06:53 AM   #34
freegeeks
Diamond Member
 
freegeeks's Avatar
 
Join Date: May 2001
Posts: 5,453
Default

Quote:
Originally Posted by theevilsharpie View Post
Those are GbE interfaces, not 10GbE.
It has 4 sfp, plug whatever you want
freegeeks is offline   Reply With Quote
Old 12-26-2012, 06:58 AM   #35
theevilsharpie
Platinum Member
 
Join Date: Nov 2009
Location: Southern California
Posts: 2,316
Default

Quote:
Originally Posted by freegeeks View Post
It has 4 sfp, plug whatever you want
lulz
theevilsharpie is offline   Reply With Quote
Old 12-26-2012, 07:56 AM   #36
imagoon
Diamond Member
 
imagoon's Avatar
 
Join Date: Feb 2003
Location: Chicagoland, IL
Posts: 4,713
Default

Quote:
Originally Posted by freegeeks View Post
It has 4 sfp, plug whatever you want
While that is a neat looking device, it doesn't work that way.
imagoon is offline   Reply With Quote
Old 12-26-2012, 09:32 AM   #37
drebo
Diamond Member
 
Join Date: Feb 2006
Posts: 6,565
Default

Mikrotik is such garbage.

I helped a guy set up a WISP using Mikrotik at his towers and I wanted to shoot myself in the face.

Also, in their own performance metrics, note how performance tanks when you start adding in any kind of QoS or ACLs. That's going to be the case with any non-ASIC-based routing platform.
__________________
"All men are not created equal, and if you believe they are, there's something seriously wrong with you. Some men are destined for greatness. Most aren't. End of story." - Jose Canseco
drebo is offline   Reply With Quote
Old 12-26-2012, 09:38 AM   #38
Acanthus
Lifer
 
Acanthus's Avatar
 
Join Date: Aug 2001
Posts: 19,916
Default

Quote:
Originally Posted by theevilsharpie View Post
The compute resources required to route a packet depends on its size, and this can cause the throughput to vary quite a bit. This is why routing performance is measured in packets per second, not bandwidth.

A modern high-end x86 processor has enough power and memory bandwidth to handle several million packets per second. This is plenty for <10Gb routing, but at 40Gb/s, you're going to have a very difficult time keeping up with the traffic, particularly if you have to handle multiple packet sizes. Best case, smaller packets degrade your speed, worst case (i.e., DDoS) your router can completely lock up. And that's just with extremely simple routing activity; you can forget about things like ACLs, traffic shaping, VRRP, etc.

Before you say "throw more processors at it!," Linux's (and I assume BSD's) TCP/IP stack can scale with additional processors, but only to a point. NICs have a limited number of queues, and that will necessarily limit how many cores can be assigned to a particular NIC. Also, having a single fast multi-core CPU is the ideal case for software routing, as multiple processors add NUMA-related headaches that can easily decrease performance if things aren't tuned just right.

A few years ago, a Linux kernel developer gave a presentation about using Linux as a bi-directional 10GbE router, and while it worked in that role for larger packet sizes, performance didn't scale when adding 10GbE links, and performance tanked with smaller packet sizes. Granted, server hardware has improved since then, but not enough to assure line-rate routing performance at >10GbE.
I could use jumbo packets, and the edge topology is very flexible. There would be nothing stopping me from implementing one PC per 10gbe as a "router".
__________________
::Intel Core i7 2600K @ 5.0ghz (100x50) 1.400v ~ 47% performance increase
::2x4GB Mushkin DDR3-1333
::Gigabyte GA-P67A-UD3-B3
::XFX Radeon 6850 1GB
::OCZ Vertex 4 128GB
::Water Cooling - Swiftech Apogee GTX, Triple 120mm rad, 120GPH pump
Acanthus is offline   Reply With Quote
Old 12-26-2012, 10:39 AM   #39
theevilsharpie
Platinum Member
 
Join Date: Nov 2009
Location: Southern California
Posts: 2,316
Default

Quote:
Originally Posted by Acanthus View Post
I could use jumbo packets, and the edge topology is very flexible. There would be nothing stopping me from implementing one PC per 10gbe as a "router".
In order for jumbo frames to work, jumbo frames have to be fully supported from one end of the network to the other. If the purpose of your "router" is to connect to the Internet, you're not going to be able to use jumbo frames.
theevilsharpie is offline   Reply With Quote
Old 12-26-2012, 10:53 AM   #40
TechBoyJK
Lifer
 
TechBoyJK's Avatar
 
Join Date: Oct 2002
Posts: 13,447
Default

Quote:
Originally Posted by Acanthus View Post
I am toying with the idea of starting a small datacenter featuring 4 x 10gb bonded lease lines and i'm eyeing the price tags on some of the Cisco routers that could handle this kind of switching load...

The price is prohibitive.

Is there a reason a home built 12-core windows server with 4 dual 10gbit cards couldn't do everything that a Cisco router would be able to do? I can't imagine that performance would be an issue with that much processing power on hand, is this underpowered or overpowered for this application? Would a properly configured Windows / CentOS / FreeBSD gateway perform as well as a high end Cisco router?
Server isn't going to have the kind of backplane and integrated switching capacity to actually push 40Gbps through it. It takes some serious switching hardware to handle that, along with the bonding/bgp'ing of the nics. Add in QOS of any kind of deep packet inspection and the idea of pushing 40Gbps becomes a pipe dream.

Our Juniper switch/router in our datacenter that we got for 2 of our 10G links was like $200K+
__________________
Originally posted by: n0cmonkey
You're being difficult. You have not provided us with the information we need to troubleshoot the problem. You have not given us errors, you're vague about where the problem is, you are not answering the questions we ask. In short, you DON'T KNOW HOW TO THINK. Give back that piece of paper you think makes your intelligent, apologize, destroy your computer and go live in the woods kid.
TechBoyJK is offline   Reply With Quote
Old 12-26-2012, 11:15 AM   #41
imagoon
Diamond Member
 
imagoon's Avatar
 
Join Date: Feb 2003
Location: Chicagoland, IL
Posts: 4,713
Default

Quote:
Originally Posted by Acanthus View Post
I could use jumbo packets, and the edge topology is very flexible. There would be nothing stopping me from implementing one PC per 10gbe as a "router".
Even if you managed to get a 9218MTU to the internet core, everyone else in the world won't have that so you will be left dealing with the fragmentation issues that would add considerable overhead to the connection that your router would need to deal with. Those fragments would lower the MTU for you and then add CPU time to reassemble them.

Jumbo frames also only matter when the traffic is pretty consistently at 9218 MTU. Most Internet / TCP / UDP frames won't be.

Last edited by imagoon; 12-26-2012 at 11:31 AM.
imagoon is offline   Reply With Quote
Old 12-26-2012, 11:41 AM   #42
Acanthus
Lifer
 
Acanthus's Avatar
 
Join Date: Aug 2001
Posts: 19,916
Default

Quote:
Originally Posted by imagoon View Post
Even if you managed to get a 9218MTU to the internet core, everyone else in the world won't have that so you will be left dealing with the fragmentation issues that would add considerable overhead to the connection that your router would need to deal with. Those fragments would lower the MTU for you and then add CPU time to reassemble them.

Jumbo frames also only matter when the traffic is pretty consistently at 9218 MTU. Most Internet / TCP / UDP frames won't be.
I see. I didn't take into account that fragmentation would become an issue. I was thinking that Jumbo Frames could be used on the internal network to the edge, and then transmitted with a normal 1500MTU out to the internet.
__________________
::Intel Core i7 2600K @ 5.0ghz (100x50) 1.400v ~ 47% performance increase
::2x4GB Mushkin DDR3-1333
::Gigabyte GA-P67A-UD3-B3
::XFX Radeon 6850 1GB
::OCZ Vertex 4 128GB
::Water Cooling - Swiftech Apogee GTX, Triple 120mm rad, 120GPH pump
Acanthus is offline   Reply With Quote
Old 12-26-2012, 11:52 AM   #43
Nothinman
Elite Member
 
Nothinman's Avatar
 
Join Date: Sep 2001
Posts: 30,672
Default

Quote:
Originally Posted by Acanthus View Post
I see. I didn't take into account that fragmentation would become an issue. I was thinking that Jumbo Frames could be used on the internal network to the edge, and then transmitted with a normal 1500MTU out to the internet.
That's correct. What he's saying is that the fragmentation and reassembly that will have to happen when packets leave and come into your network will be a significant bottleneck at those speeds.
__________________
http://www.debian.org
Nothinman is offline   Reply With Quote
Old 12-26-2012, 07:35 PM   #44
freegeeks
Diamond Member
 
freegeeks's Avatar
 
Join Date: May 2001
Posts: 5,453
Default

Quote:
Originally Posted by imagoon View Post
While that is a neat looking device, it doesn't work that way.
future Mikrotik cloud routers will support sfp+, I don't see the problem of using 10 Gbit optics in one of their future products. I doubt that the op really needs 10 Gbit connectivity right away, some bonded gbit ethernet ports will do to get started
freegeeks is offline   Reply With Quote
Old 12-26-2012, 07:39 PM   #45
freegeeks
Diamond Member
 
freegeeks's Avatar
 
Join Date: May 2001
Posts: 5,453
Default

Quote:
Originally Posted by drebo View Post
Mikrotik is such garbage.

I helped a guy set up a WISP using Mikrotik at his towers and I wanted to shoot myself in the face.

Also, in their own performance metrics, note how performance tanks when you start adding in any kind of QoS or ACLs. That's going to be the case with any non-ASIC-based routing platform.
they are a pita to configure but most stuff just works when you get it going
I just configured a Mikrotik RB2011 a couple of hours ago, I was able to push 500 Mbit with a bridged configuration (so all in cpu). Not bad for a $80 device

Try that with a Cisco, for what you pay, Mikrotik offers a lot of value
freegeeks is offline   Reply With Quote
Old 12-26-2012, 10:04 PM   #46
imagoon
Diamond Member
 
imagoon's Avatar
 
Join Date: Feb 2003
Location: Chicagoland, IL
Posts: 4,713
Default

Quote:
Originally Posted by freegeeks View Post
future Mikrotik cloud routers will support sfp+, I don't see the problem of using 10 Gbit optics in one of their future products. I doubt that the op really needs 10 Gbit connectivity right away, some bonded gbit ethernet ports will do to get started
Not when he has 4 x 10GB lines. The SPF also needs to support 10GB for a 10GB SPF to work.
imagoon is offline   Reply With Quote
Old 12-27-2012, 12:04 AM   #47
jumpncrash
Senior Member
 
Join Date: Feb 2010
Posts: 416
Default

I think we have 160gbit coming in here, and I think we have 10 6500s, so there must be a reason for that
jumpncrash is online now   Reply With Quote
Old 12-27-2012, 12:50 AM   #48
fread2281
Member
 
Join Date: Sep 2012
Posts: 32
Default

Quote:
Originally Posted by Acanthus View Post
Windows
lulz.

Protip: no windows server
fread2281 is offline   Reply With Quote
Old 12-27-2012, 08:58 AM   #49
Acanthus
Lifer
 
Acanthus's Avatar
 
Join Date: Aug 2001
Posts: 19,916
Default

Quote:
Originally Posted by fread2281 View Post
lulz.

Protip: no windows server
I know I know. I wanted to get opinions more on the topology and capability of the hardware vs the operating system. If you read further down, in a realistic scenario i was thinking of OpenBSD.
__________________
::Intel Core i7 2600K @ 5.0ghz (100x50) 1.400v ~ 47% performance increase
::2x4GB Mushkin DDR3-1333
::Gigabyte GA-P67A-UD3-B3
::XFX Radeon 6850 1GB
::OCZ Vertex 4 128GB
::Water Cooling - Swiftech Apogee GTX, Triple 120mm rad, 120GPH pump
Acanthus is offline   Reply With Quote
Old 12-28-2012, 04:05 PM   #50
azev
Golden Member
 
Join Date: Jan 2001
Posts: 1,003
Default

in my opinion most home brewed computer with that kind of processor will be more than sufficient to support 40Gbps for traffic. Most of the time the limitation of a computer for a router is the amount of ports that it can support is limited by the amount of pci-e slot available. There are lots of open source linux based routing OS that uses very similar command as Cisco IOS.

The only issue, most of those are open source and its support are forum based. If you are pushing that kind of bandwidth, that means down time = $$$ lost, and you want the best support you can get to help you bring up the network.
azev is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -5. The time now is 11:36 AM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.