Discussion: Collapsed Backbone Architectures

shadow

Golden Member
Oct 13, 1999
1,503
0
0
Hello, I was just wondering if you guys wanted to start a discussion on the pros and cons of a collapsed backbone architecture. If so, please post your ideas and opinions.

thx
 

xyyz

Diamond Member
Sep 3, 2000
4,331
0
0
i'll give this a bump so that those who know anything about this will say something...

I suggest you start this topic though... say something that someone can reply to...
 

spidey07

No Lifer
Aug 4, 2000
65,469
5
76
hmmm...seems hard to describe anything as a backbone nowadays. Used to be a single piece of thick-coax or a FDDI ring, that can describe a backbone. Today however most nets are a mesh of LAN switches and routers.

Pros of a collapsed backbone - easy to troubleshoot, core communications run through one or two switches instead of dozens, high performance

cons - single point of failure, expensive

 

shadow

Golden Member
Oct 13, 1999
1,503
0
0
spidey - you are correct, a collapsed backbone network does have a spf at the core, however this can be designed away with ECMP. I see where you're coming from there, but about the expensive part - you'll have to justify that one....

you don't by any chance have comnet do you?
 

spidey07

No Lifer
Aug 4, 2000
65,469
5
76
ecmp? spf?

maybe we're talking about two different things here. Could you please give an example? I was thinking in terms of campus and LAN.
 

shadow

Golden Member
Oct 13, 1999
1,503
0
0
my bad, in my hurry I used some acronyms, I am talking in an enterprise (2000 nodes +) or campus LANs. But of course it can scale up and down.

SPF - (real sh*tty and confusing acronyms) single point of failure
ECMP - Equal Cost Multipath Routing; with this you can tie your distribution routers/switch to 2 or more core routers, hence getting rid of the single point of failure at the core of your network. Of course this means you need routers/switches at the distribution level to have ECMP

did I address all your questions?

serge
 

spidey07

No Lifer
Aug 4, 2000
65,469
5
76
ah, I see. Just never heard those acronyms before. Campus design is where I live and breathe.

The expensive part come from using higher powered switches with higer port densities and Layer3 switching. To me collapsed backone really has no bearing on today's designs because of the advantages of fast L3 routing. Since you are really routing at the distribution layers there is no true backbone - unless you consider the layer3 links with a 30 bit mask between the core and distribution layers is considered a backbone.

If you want to eliminate single points of failure then you MUST have two distrbution routers/switches. Otherwise if the dist switch goes belly up then all access points/switches will also go down (like you said, this could affect thousands of users). Also having two sets of access switches attached to separate NICs on the host truly eliminates single point of failure. Throw in some clustering to protect the host/data side and you'll never go down.

I generally stay away from equal-cost routing, causes problems with multicast. In a campus environment with all that bocou bandwidth why not fill it up with broadcast quality video? :)
 

shadow

Golden Member
Oct 13, 1999
1,503
0
0
Just to clear up something, I really should be calling the distribution layer the access layer. I fscked up before by calling it the distribution layer. I hesitated calling it the access layer because typically you have an access -> distribution -> core. My designs however only have two layers. Perhaps I should call my core the core/distribtution. More details on my design later.

Good points spidey. But as you mention, ideally you would not have routing at the distribution but rather switches. However for redundancy at the core you need to run two links from the access on up, and in the case of switches that means STP, and one of the links doesn't get used. With ECMP however you can use both links simultaneously BUT that means you need a router and not a switch. I am unsure of convergence times for STP but I do know that they are significantly higher than OSPF's ECMP. Also as far as the router/switch at the distribution I know that certain chassis can take a router management module, and a routed gigabit uplink (talking ethernet here) yet still have switched blades for distribution. I think that is the ideal solution in the case of a collapsed backbone architecture.

I have made two network designs using collapsed backbone architectures. They both only consist of two layers: USER -> Access Swithes/Routers (yes Routers) -> Core Routers (or maybe I should call it core/distribution even though they are the same device). I never implemented these, they never went beyond the analysis of my teachers as I am in college at the moment. The second design was for an insurance company of some 2000 workstations and 280 servers, I designed the network for 3000 users and 440 servers. This particular insurance company is right now moving from ATM/Ethernet to GigE, and we were able to get a hold of their design and costs. The cost of our network was around 10% less as designed than theirs. What I mean by 'as designed' is that our network is still heavily unbalanced, and could go through some streamling before arriving at a finished design. You'll see more what I mean next paragraph. Their network design is more traditional, a mesh of routers, up to 4 different paths, typically only two.

I had a pair of Extreme Networks Black Diamonds 6808s at the core/distribution with gigabit blades. For the access layer I used Extreme Networks Alpine 3804 and 3808 configured with a gigabit blade and 100BTX blades. The gigabit blade was used to connect one gigabit link to each of the core routers. The core routers were never linked - so that they could be in different buildings in case of bombs etc (this is a real threat for a big insurance company, at least the execs consider it). The 100BTX blades were used to connect to individual users or individual servers. Yes this means each server and workstation has a routed port! Yet, if you remember in the last paragraph we managed to come 10% under their proposed budget. I attribute the savings from removing the 12-16 routers they had in a mesh for our 2, and also because I used Extreme Networks while they used Alcatel.

After this experience I thought collapsed backbones were the way to go. It is also why I pondered at the added expense you indicated spidey. Since this network is way overbuilt (routed port for every user - come on!) and 256GB/s backbone speed (combined backplanes of the core routers) I could actually cut out some stuff, reduce the number of access switch/routers from 30 to 15, etc, etc, to make it even cheaper.

Questions:
What do you mean by a 30bit mask?
Did you try both ECMP by route and ECMP by packet with multicasting? More details on this would be sweet :)
Would you ever consider having two nics per workstation to eliminate SPF at access layer?

thx alot for your input, and I hope you like the discussion :)

PS - spidey you got your CCIE yet?
 

ScottMac

Moderator<br>Networking<br>Elite member
Mar 19, 2001
5,471
2
0
Maybe you should consider an ATM core. It's fast, extremely scalable, roughly the same cost as a comparable Gig solution, and well-proven. There is a minor hit for the SAR, and routing ELAN-to-ELAN is very slow (can be negated by routing ahead of the SAR).

No spanning tree in ATM....if you plug in a redundant link, it'll pretty much come as a load sharing link (depends on the vendor, and flavor of PNNI).

ATM can provide absolute end-to-end quality of service, not the smoke & mirrors Ethernet marketing fluff (like RSVP). MPLS is easily done on ATM, and if you use MPLS, the QOS can be applied and maintained across the ATM and Ethernet links.

ATM is/was a wunnerful thing. The ATM Forum et al should be shot for dropping the promotional ball. Cisco never liked it either, that didn't help (StrataCom notwithstanding).

ATM is, at the very least, a nearly perfect Core: Very fast, very scalable, very resiliant, very fault tolerant.


AND...if you're STUCK with Ethernet (at any speed), and wanna see the Oyster's Ice Skates in VLAN stuff, check out Alcatel (the former Xylan product line). Last time I looked, they were far above & beyond anyone/everyone else in policy-based VLANS. Cool stuff, a little ginchy in the CLI...but once you got used to it, it was pretty easy to set up.

I like the Extreme boxes alot. We worked with them for Networld + Interop, including the DWM modules and the Packet-Over-SONET stuff. Pretty quick, not too horrible to config. They also have been sane enough to have exactly two variants of CLI, regardless of the size of the box...from a little Summit workgroup switch to the Massive Black Diamond with 1500 (that's Fifteen Hundred) 100Meg Full-Duplex connections, same CLI, same command set, same everything. If you spring for the "Enhanced" O/S, you get the advanced routing protocols.

Nice box....the Extreme personnel get a little carried away with the Purple stuff though.....gotta wonder about guys with purple sneakers, socks, pants, tee-shirts, hats, pens, notebooks...they even eat purple M&Ms made just for them by M&M. "Extreme" is probably a good label.


FWIW

Scott


 

spidey07

No Lifer
Aug 4, 2000
65,469
5
76
the equal cost routing can cause trouble with PIMs reverse path check and locating rendevous points, not that equal cost routing is bad (it is awesome) just be aware of its limitations. More good points you brought up are spanning-tree and the inherent troubles it causes.

It is generally accepted to eliminate layer-2 forwarding and spanning tree and replace with layer-3 routing instead. As you have stated routing is much more flexible and does indeed converge much faster. So this is why I prefer to use L3 switches at the distribution layer - greater control, use routing protocol rather than spanning-tree for failover. Maybe I'm just used to a 3-tiered architecture, scales a little better because of the routing control in each building.

But then again, this topic is wrought with opinions. There isn't any real correct answer, as design goals change from network to network.

On the topic of equal-cost routing, balance by conversation and not packet-to-packet. Where's garion? he loves campus switching.

Oh, and how do I get around my firewall? :)
 

shadow

Golden Member
Oct 13, 1999
1,503
0
0
I like what I have heard of ATM, however this particular company does not like their ATM/Ethernet mix right now (Fore ATM/Alcatel Ethernet) so they don't want it for the future. I would like to see my collapsed backbone design vs an ATM/Ethernet design. Compare them based on performance, cost, scability, reliability, easability, and simplicity.

An ATM/Ethernet deisgn is never going to be as simple as a pure ethernet or pure ATM design. Also, ATM is $$$. All the extreme gear I mentioned has bandwidth throttling and layer 1,2,3,4 policy based QoS. This QoS is weighted queing on the routers, or priority, whichever way you want to call it. I don't understand completely why you would even need weighted queing on a wirespeed router..... as there is almost no queue (sp?) unless you are overloading a link - which shouldn't happen if you designed it right to begin with.. Ethernet is never going to be as good as ATM for QoS, it just won't happen, but you can design a network that has enough QoS in Ethernet to get you by and still come in way cheaper, and then there's over-provisioning.

For example, my design has 21mbps bandwidth per user and 127mbps per server in worst case, 100% network utilization, scenario. Please note that there are no shared links in my design, and I can full duplex all links to avoid collisions. Who needs QoS in an environment that never sees over 5% link utilization on any single link, and perhaps averages around 1% on all. If you ever did see over 5% utilization, then light up some dark fiber to wherever you need it and boom, 2% utilization on that link.

The power of ethernet is it's cheap bandwidth. The power of ATM is it's unbeatable QoS. To satisfy QoS with ethernet you throw alot of $$$ at it, design it well (don't put all the bandwidth hogs on the same switch etc), and use what little QoS services you have - some might actually surprise you how well they work (read about some CISCOs demoes somewhere). To satisfy QoS with ATM you configure it properly and cross your fingers that the ATM/Ethernet devices play nice with eachother.

As far as ATM beating out ethernet for resiliency, scalability, and fault tolerance; I am going to need some more convincing. I think it depends more on your design and implementation than anything.

The Ethernet QoS I was talking about above is weighted queing on the routers, or priority, whichever way you want to call it. I don't understand completely why you would even need weighted queing on a wirespeed router..... as there is almost no queue (sp?) unless you are overloading a link - which shouldn't happen if you designed it right to begin with.

But like spidey, I agree, there's a million ways to do it.
 

FFC

Member
Oct 23, 2001
100
0
0
Shadow, what you call ECMP is just an Extreme acronym. Most other routers vendors generically call it equal cost load balancing and it's on by default. Also generally in networking SPF means shortest path first rather than single point of failure, sorry I'm just nitpicking.

I haven't seen a LAN environment including campus for years that doesn't use a collapsed backbone. The bus technologies of yesteryear have gone even in education environments (traditionally the lowest spenders on IT in the UK market). The invention of 10BaseT and chassis based hubs made networks easier to troubleshoot and MUCH more reliable. Everything since then has mainly been devoted to speeding things up. Switches, faster standards etc.

If you're using Extreme kit and want resilience did you check out ESRP, Extreme's switching resilience protocol. It works fine as long as you aren't trying to do too much other stuff with the switch (Routing, rate shaping etc.).

Whether STP or layer 3 convergence is faster is entirely dependent on the network design. Lots of enterprise networks still have a variety of protocols and these aren't always catered for at layer 3, thus bridging becaomes an imperative and STP knowledge is required.

Finally I work for a reseller who used to specialise in ATM solutions. We haven't installed a new ATM based network for the past 18 months. People just don't seem to want them. The vendors are going cold on the technology and most backbone providers are busy migrating away from ATM because the available SAR speeds aren't high enough to support what they need.

Just my .02p worth.
 

shadow

Golden Member
Oct 13, 1999
1,503
0
0
No FCC, you are absolutley correct, I was in a hurry and made up some acronyms with the hope they would make sense when read.

Are you sure that most LANs, even campus sized LANs are collapsed backbone? I have only heard of one other..... The rest are FDDI/SDMS/ATM/Mesh - at least from what I have read/seen/heard about. I am not sure we have the same idea for collapsed backbone architectures....

I saw the ESRP, neat stuff, but never got that far in the design.
 

FFC

Member
Oct 23, 2001
100
0
0
Hi Shadow, by collapsed backbone I assumed (probably wrongly) that you meant collapsing a bus based backbone into a wiring closet device. In the old days we had many 10Base2, 10Base5, FDDI, Arcnet (Jeez I'm old) etc. bus based networks that were a nightmare to troubleshoot quickly but were relatively cheap. The invention of 10BaseT and it's star topology along with the big chassis based hubs (Cabletron, Chipcom, Synoptics etc) allowed the bus to "collapse" into the wiring closet and make troubleshooting easier whilst drastically increasing reliability (and initial cost).

The campus backbones can be collapsed nto switches with the advent of longer reach ethernet using fibre. With the 70Km GBIC's available now you can collapse metro networks into switches easily. Also the transmission companies are now providing ethernet handoffs to switches or routers rather than requiring that the layer2/3 kit utilise expensive SONET/SDH interfaces. With gigabit ethernet switches getting higher port densities all the time it's possible to collapse large WAN backbones should you wish to. Personally I think the layer 3 implementations on most switches today aren't close to the performance and reliability required for big WAN backbones yet but they're coming along.
 

shadow

Golden Member
Oct 13, 1999
1,503
0
0
You've got the right idea allright, and you are very correct when you say that they now _can_ collapse to a switch, but more often than not the core is actually made up of a mesh of some sort, or using a ring technology - but hey maybe I'm wrong about that - maybe more companies are using collapsed backbones than I know about. All I can speak from is my experience, and that is mabye one collapsed backbone out of the 30networks I have come into contact with.

thx for the infoz :)
 

spidey07

No Lifer
Aug 4, 2000
65,469
5
76
I guess that is what I was trying to say (thanks ffc). what used to be "backbones" are now "backplanes" in superfast high density switches.

Lot of times campus nets will look like a "stegosaurus" with two or more data center switches and many branches (dual connected) from there.

I love to debate the gigethernet vs. ATM. I love'em both. and can willingly play devils advocate just to be obnoxious. :)
Point is like FFC said the ATM forum dropped the ball and made it WAY to difficult with interoperability.
 

xyyz

Diamond Member
Sep 3, 2000
4,331
0
0
okay wth are you people talking about?

how about some giving some basic knowledge to the neophytes (me) and explaining some of the acroymns.
 

shadow

Golden Member
Oct 13, 1999
1,503
0
0
BigNick - what are all the ATM clouds used for? I can't seem to make sense out of the naming scheme, but it seems like they are connections which go to different states/schools

If I am reading it right you also have 90mbps Internet connect - nice! But I assume it's fractional or shared between yourself and the local highschools, etc.

I am typing on a frac T3 + shared T3 which supposedly comes to around 9mbps - and with some 15000 students it's still fast!!! Of course it's proxied and no one is running ftp servs etc...

-serge