Help spidey design a server farm net

[spidey07]

This should be a fun exercise!

Here's the problem - We're just about out of gigabit ethernet ports in our two core 6509 switches, all slots full (has two supervisors, one MSM, leaving 6 blade available). Most of the 24 gig ports serve the distribution layer switches in other buildings. The other gig ports and 10/100 ports are for servers and other data center devices.

So I need to provide more gigabit ethernet ports, twenty to start out and winding up at 70 at years end. I must consider 25% growth year over year. Solution must be totally physically redundant.

So my choices really are a Cisco 4006 populated to the gills with 1000Base-SX ports and a L3 engine. Or a switch fabric enabled 6509/6513 with appropriate 1000Base-SX cards. The obvious choice is a fabric enabled 6513, but good golly those fabric cards are expensive!

The two switches will then be etherchannel attached to the core switches via four or six gig ports each. Each server will be attached to both switches with NIC failover resulting in a primary switch with all traffic (or maybe I'll split it up between the two?)

Any ideas or better means? Keep in mind this is for very high powered servers (slowest ones are 4-way wintel, biggest are 32 proc 6800s for data warehouse) so performance between servers (VERY IMPORTANT HERE) and out to user community is paramount.

6519 scales and has incredible performance - expensive though. 4006 meets the bill but might suffer performance/scale side.

thanks for any input

 

[Garion]

Interesting (but not uncommon) problem.

One thing you didn't address is capacity on your existing 6509's - How much of the mem/cpu/backplane is in use?

If you have ample headroom on the chassis, consider swapping out your 8-port gig modules for the 16-port'ers. Doubles your density in one shot on the existing chassis. You can do this selectively, as you need to grow. Think you're going to need some more gig ports in a few months? Order a new card and voila! Eight free ports without any reconfiguration whatsoever.

If that fails, I'm a big fan of scaling OUT, not UP. IE, put in another 6509. Leave ONLY your absolutely critical servers on the cores and move other things to the secondary switch. Personally, I wouldn't bother with a 4006. Any 3rd switch you put in place probably doesn't need L3 capability, just trunked and routed via the primaries. That's what we did, at least. I had two core L3 6509's in the data center trunked with a 4GB channel to three others, serving the rest of the data center. Worked like a champ.

One thing - I just re-read your note. Get rid of that MSM! It's way old technology. Get yourself some supervisors with MSFC's on them. They are light-years ahead of the old MSM, more redundant, allow a lot more VLANS and are WAY faster, especially if you're talking about these kind of traffic numbers. If you've got an MSM, you probably also have a Sup1/1A. You definitely want the Supervisor 2 modules, or the Sup 3's when they come out for the 6500's. The CEF switching will give you a good performance bump.

- G

 

[spidey07]

Thanks Garion.

I have my reservations about using the 16 port gig blade as it is limited by a 8 gig connection to the backplane. Scaling out is definately the goal here.

I'd like to have layer three switching just incase, there will be lots of HSRP and redunancy. The flexiblity of IOS on each server port has its appeal.

No need to lecture me on the MSM/sup1. I've tried and tried and even forced cisco to offer substantial "technology migration" credits. The answer on the capital request is always "Well, it works fine now why would we need to upgrade it" Hard to show the technical side of "well it isn't working fine now...its limping, you just don't know it". We've got a ton of multicast to the proc on the MSM runs a little high.

Option three -
Replace current two core 6509s brains. 4xsup2 w/MSFC2. Add switch fabric module. replace 8 port blades with fabric enabled 16 port blades. replace 48 port 10/100 blades with fabric enabled ones.
Remember when using the switch fabric and 256 Gb backplane all cards must be fabric enabled - if a single blade is on the 32 Gb plane then the entire switch runs on the 32 Gb plane.

Option four (low cost)
Replace 8 port gig blades with 16 port gig blades and monitor backplane. Consider replacing sups/MSM.

I think I'm gonna throw out the 4006 option. Not really a data center kind of switch.

 

[Garion]

Have you really looked at the bandwidth that a single gig-attached server is pushing out? I found that mine RARELY pushed more than 200Mb/s. Take a snapshot of the utilization of all the ports on an 8-port blade with MRTG or Concord or VitalSuite (or whatever you use). Good money says that the average will be 1-2Mb/s and peaks might go to 3Mb/s. Of course, I could be wrong, but it's worth checking closely.

Gigabit and fast Ethernet to the desktop is cool, but really doesn't generate all THAT much traffic. I had developers doing big code builds and compiles on their sun boxes and even their desktop switches rarely hit 20% on the dual-gig channels.

Like I said.. Figure out what your backplane utilization IS, assume that's your load average for the currently-utilized number of ports. Factor in your growth factor, double it for good measure and see if you need it or not.

On the MSM/MSFC.. Just wait until your boss asks you to do something that you can't (easily) do and blame it on the old technology. *grin* Do that three or four times and they'll get the point and start pushing your request. Works every time, especially when their boss is asking THEM to do these things.

- G

 

[spidey07]

LOL,

Wanna know what else the MSM can't do? control plane manipulation. No access lists, no route-maps, you'll get basic IP switching and like it. I can't tell you how many meetings I've been in and said "Hey, I can use a route-map or ACL to help with the migration. DOH, that's right I can't. You'll have to do all that work in a 4 hour maintenance window. sorry."

Its starting to sink in *giggle*

 

[L3Guy]

So my choices really are a Cisco 4006 populated to the gills with 1000Base-SX ports and a L3 engine

Danger Will Robinson! Danger!

The 4000 series switches have a bad reputation for dropping packets.
I was testing one of them with Smartbits and with 2 ports, unidirectional traffic, and layer 2. The 4k started to drop 1518 byte frames at around 95% utilization. It would drop up to 9% of the traffic as it got close to 100% utilization.
And, it dropped some traffic at Packet sizes as low as 1024.
Clearly not a good fit for a server farm, although perfectly adequate for a access type switch.

Good Luck.

Doug

 

[spidey07]

Thanks for the info L3guy. Was your testing with the new supervisor for the 4006?

 

[L3Guy]

I tested with Ver 5.2(1) code on a
WS X4012
WS x4232-RJ-xx
WS x4248-RJ

Probably old. I haven't payed much attention.

Doug