Multiple Connections from One Switch to Another

[Netopia]

Hey Gents,

I've got a problem that might prove to be an impossibility, but here's the situation:

I've got one department in one area of the building, and the main server room in another. There are stacks of gigabit switches in each location.

I can take a single line (patched from one patch panel to the other) and connect the two stacks, and all is well. What I thought I'd like to do is to have multiple connections from one stack to the other in order to increase total bandwidth. I've got the lines run, so I tried it.

As soon as I plug in more than a single connection between stacks of switches, all hell breaks loose and everything on both stacks starts blinking wildly. My guess is that there is some sort of loopback is happening, but I'm not sure.

Anyone know for sure if that's the problem? If it is the problem, is there some way to make multiple connections from one place to another, or would I have to invest in a 10GB switch to make these connections?

Joe

 

[spidey07]

That is a bridging loop, sure sign. Those switches appear not to be running spanning-tree (you can check if they are by logging into them and checking it's configuration). It's typical for small office unmanaged switches to not run spanning-tree for loop-detection and prevention. Otherwise one of the switches would put a port in blocking mode to stop the loop.

The only way to get combined links is with link aggregation (LAG). And even then few switches support link aggregation across different switches. Switch model would be helpful.

 

[skyking]

I've seen that one, the lovely little buggers at a private school got into the network gear over the summer and had three loops going. Oddly enough I could remote into the server but that was all I could do

 

[spidey07]

I've seen that one, the lovely little buggers at a private school got into the network gear over the summer and had three loops going. Oddly enough I could remote into the server but that was all I could do

It's essentially a broadcast storm where every link is just full of broadcasts/multicasts because that traffic is flooded to every port. The frames (at layer2) never die, just keep going around endlessly and within even 5 seconds the network and every link in that broadcast domain is practically unusable.

This is why one needs to be VERY careful when interconnecting switches. You need to know the layout, where your root bridge is at, what redundant links are in place, how spanning-tree is configured, etc.

 

[gsaldivar]

Just a shot in the dark here -

Check the model #'s of your switches closely. I had a few old Dell 10/100 switches at a client site and I suspected that they were having a bandwidth problem. Right before ordering a bunch of new equipment, I downloaded the manuals and found that one of the ports on each switch was an unlabeled Gigabit port! Some quick rearranging of uplink cables and their lag issues were solved, without the need for any new equipment.

Good luck!

 

[seepy83]

Netopia, can you draw us a topology of your switches? Or at least describe it?

You say you've got 2 stacks of switches at each location. How are each of those stacks connected to eachother? Do you have a single switch (Core/Distribution) that all of the others (Access Switches) are plugged into?

 

[theevilsharpie]

And even then few switches support link aggregation across different switches.

Pretty much every managed switch I've used in the past five years has supported LACP.

 

[spidey07]

Pretty much every managed switch I've used in the past five years has supported LACP.

Across different switch chassis? Highly unlikely.
sw1<--->sw3
sw2<--->sw4

With those links being a single channel/LAG.

 

[Netopia]

Wow... thanks for all of the responses!

As you can probably see, I know enough about networking to get myself in trouble! Actually, I've been administering this LAN for 15 years, but never ran into this before. A lot of this change is because we were purchased by another company and they are moving all of their equipment into our building and merging the two companies... so it will only get more complicated!

Currently in my server room I have 3 Dell 2724 switches which are simply daisy chained together. The management mode has never been set up before... I never really had a need for anything but basic network connections.

(From Dell's Site)

Link Aggregation, up to six groups and up to four aggregated links per group (IEEE 802.3ad)
Port mirroring (up to four source ports)
Jumbo frame support up to 9000 Bytes (2716 & 2724 only)

At the other location are a Dell 2624 and a Dell 2824. The 2624 is unmanaged, and the manual (I just looked it up) specifically says not to connect switches together with more than one cable! Those two are daisy chained together.

The 2824 is a managed switch (never set up)... about that one, Dell site says:

Industry-standard link aggregation adhering to IEEE 802.3ad standards
Supports 4 link aggregation groups and up to 4 ports per group

There will be another location in the building where we hope to have multiple runs coming back in the same fashion, in order to increase bandwidth. I also don't know what other switches will be coming over in the next couple weeks, but I know that they are bringing at least four more gigabit switches for the server room.

After doing some quick reading on Link Aggregation, it certainly seems like what I'm after.

So... would this work (after getting the switches into managed mode):

In the server room, define ports 1-4 to be an aggregated link group on Switch #1. Daisy chain switch #1 to switch #2 and switch #2 to switch #3 like normal.

In the other department, define ports 1-4 to be an aggregated link group on the managed switch (Dell 2824) and then daisy chain that switch to the unmanaged switch like normal.

Connect Switch #1 in the server room to the Dell 2824 in the other department, using ports 1-4 on each switch, connected via 4 different runs of cable.

=========

If that would work... do I have to have link aggregation setup on the other switches in the server room? I feel like if I didn't do the link aggregation to daisy chain the switches, I'd just be creating a bottle neck at the first daisy chain.

Joe
who has to learn a lot... and QUICKLY!

 

[spidey07]

That should work. Lag is a switch to switch thing so you wouldn't need to do anything on the routers. The 2624 instructions might mean it doesn't do spanning tree which would be weird as they're decent switches.

What I was descrbing was multichassis lag. But that doesn't come into play here and nor supported. It's used for fault tolerance if you lose a single switch the lag stays up

 

[Netopia]

That should work. Lag is a switch to switch thing so you wouldn't need to do anything on the routers.

Yeah... my typo, I meant switch. I corrected it.

Again though... is there a reason that I shouldn't set up LAG on each switch in the stack (and other stacks, if I've the ability) and daisy chain them with LAG so that the worst bottle necks will be from the servers themselves? Actually... I've already been looking at changing over the dual NICs in the servers so that they are bonded too... gee... I could get a WHOLE lot more bandwidth here for cheap.

I went over and looked at the rack that the other company is bringing over. Maybe I'm wrong, but I thought that the way they have their stuff set up was dumb. The whole building is set up with GIG-E. They have all GIG-E switches, and then they run a SINGLE Cat6 cable from their switch stack into a GIG-E switch in their rack, and from that switch to the 18 servers they have in their rack. I tried to explain to them that having all those servers sharing one cable to the rest of the company was a self-inflicted bottleneck, but they kept telling "dude... it's gigabit!".

Am I off base, or is their rack set up poorly?

Joe

 

[kevnich2]

Depends what type and how much traffic is being sent/received from the servers. If your doing large file transfers or transferring traffic for several hundred computers across a single gige link - yeah that would be a bottleneck. If you're talking 100 computers accessing word documents or doing database lookups - probably works fine for them. Every network is different.

Example would be a company I used to work for - every computer was on 100mb switch - except the graphics and engineering department that was transferring large files back and forth from the server. The server was on a gige link to a gige port on the switch and the graphics/engineering had it's own gige switch but everyone else was just on 100mb. For them it worked fine.

 

[Netopia]

In this case, we are a printing and publishing company... tons of graphics and rasterized images, which can be pretty huge. The biggest single file I've seen is 17GB, but 8-10GB files are being pushed around several times a week... with moaning about how long it takes.

I hope to change that when we're merged.

Joe

 

[freegeeks]

well, the company should invest then some money in decent network design, get an outside consultant if nobody is confident or experienced enough to do it. If it falls on your head and you have to do it then you need to read up on LAG and STP. Basic advice I can give you without knowing the topology is to use LAG between the switches and enable STP so you have at least a loop free environment. Just make sure that some tiny ass switch hidden in a closet doesn't becomes the root bridge in your STP topology.

 

[spidey07]

I've done pre-press and printing/publishing networks. You need to build the highest performance network you can afford. 1 gig or mulitple 1 gig ports to end stations and multiple 10 gig between the switch closets would be the way to go. Those networks move a ton of data and the time to process jobs is most times the limitation of the network and time costs money.

You can easily justify a forklift upgrade of everything.

 

[freegeeks]

btw, by the sound how this network is setup, I would not be amazed that this is just a giant layer 2 broadcast domain. Look into VLANs to segment traffic and 802.1q to move traffic between your switches

 

[imagoon]

When creating the LAG groups, make sure to study which method works best for you.

This is VMware / Cisco but it gives you a real world example to look at.

http://kb.vmware.com/selfservice/mi...nguage=en_US&cmd=displayKC&externalId=1004048

What I am talking about is the load balancing methods mentioned here:

http://www.cisco.com/en/US/tech/tk389/tk213/technologies_tech_note09186a0080094714.shtml

Different ones have different effects on how the data is routed. You also need to find one that is supported on your gear as different brands might not have all of them.

 

[Netopia]

I've done pre-press and printing/publishing networks.

LOL.... it's Electronic PrePress that I'm trying to connect with LAG! That's the second rack that they'll be moving over. They have their RIP and storage servers in a rack, and they do just like the data guys and have a single cable running to that rack and a switch in the rack running to the servers. This is the way that Kodak set it up for them... but I'm going to be. I've tried to convince everyone that this is NOT how to do it, but a typical response has been the likes of "well... if it's slow when we move over there, we'll take a look". I keep trying to explain to them that they won't notice BECAUSE IT'S ALREADY SLOW!

The way I've had PrePress setup previously is with their own small server room and every server and all the workstations plugged into the same local switch and a single homerun going back to the main server room... but they only needed that link for email and internet access, so a single run was fine. Now that they want the PrePress servers in the main server room, the best I could do (within budget constraints) was to try this multi-gig run.

Joe

 

[spidey07]

As stated, the main problem with lag is a single conversation will normally only use a single gig connection. Sure you can adjust the balancing algorithym but a single conversation will only use a single link. For your scenario this is not a good thing. If their goal is to house the data in another closet and have the pre-press and RIP in another room, think about the data going back and forth. You can try the LAG with gigE and monitor performance and utilization.

If it were me I'd get away from stacks and move to a high performance network with multiple 10 gig links between switches. And also think about how to structure the network so there is some hiearchy to it instead of a hodge podge of closets interconnected. If it's just two closets then no big deal, just two big switches in each closet, connected by 1-8 10gig links. If you go this route make sure to have the fiber tested first to see if it can handle 10g for the distance you need. Also make room for 10gig links to the servers and stations that need it.

 

[Netopia]

Love to upgrade.... sadly, in this economy... the $$$ aren't there. That's a big part of why the merger is happening.

 

[Netopia]

Across different switch chassis? Highly unlikely.
sw1<--->sw3
sw2<--->sw4

With those links being a single channel/LAG.

I was just reading this on this Cisco link:

http://www.cisco.com/en/US/products/ps9967/products_qanda_item09186a0080a36439.shtml

Note: A limitation of Link aggregation is that all the physical ports in the link aggregation group must reside on the same switch. SMLT, DSMLT and RSMLT technologies remove this limitation by allowing the physical ports to be split between two switches.

Joe

 

[spidey07]

I was just reading this on this Cisco link:

http://www.cisco.com/en/US/products/ps9967/products_qanda_item09186a0080a36439.shtml

Note: A limitation of Link aggregation is that all the physical ports in the link aggregation group must reside on the same switch. SMLT, DSMLT and RSMLT technologies remove this limitation by allowing the physical ports to be split between two switches.

Joe

Correct. All of which are proprietary technologies. It can be done and I've used these technologies with great success. But there are limitations.

Channeling is great but you don't get thruput of the aggregated channel only a single link.

 

[Netopia]

Sort of like adding lanes to a highway. More cars can get through, but each car can only use one lane.

Joe

 

[Emulex]

which is why things like mpio and nic teaming can help if you can't afford 10gbe