Gigabit Layer 3 Network layout help

[shadow]

I think that a router in each building won't be that much more expensive than a medium density switch with gigabit uplinks.

24port 10/100TX layer3 wirespeed switch w/ GBIC uplink ~$3000
24port 10/100TX layer2 wirespeed switch w/ GBIC uplink ~???

it'll also be easier to set up and maintain, #$%#@%! STP burn in hell @$#%@#$%

 

[CTR]

That's correct. You can do a lot with fiber. Just be sure to run plenty of it. At least 12 strands to each remote.

I think you should explore the cheap and effective Full-Duplex Fast Ethernet solution. Personally, I would look at Cisco 3660's at each remote, and a 7513 at the core. Cisco equipment really shines in this market segment. You can start of with single FE links, and upgrade to multiple load balanced FE's as needed. In the future, you can re-use the 3660's for your VoIP solution, and the 7513 can still be used effectively when you upgrade to GigE between sites. Then your final upgrade will be the L3 switch to replace the 7513. As for you LAN switches, that is your call. Extreme (nice), Cisco, whatever.

How would you feel about revising your network diagram to reflect the cheap solution? Then you can compare it to your "ideal" diagram and come up with a network growth plan and budget. Of course, I've also worked on projects where the $$$ are available now (grants, etc) but won't ever be again. Is that the case here?

 

[CTR]

Hey too bad everybody's not in Dallas or else we COULD discuss this over some cold ones.

 

[spidey07]

Hey CTR, any jobs for CCIEs down there? I like dallas and am looking to move.

I heard the market is very dry in texas

 

[CBuxton]

Beer, yes, Beer is good...
Okay, let me try to breakdown the kind of users we have:
2,200 K-12 students - 30% are K-4th grade, and would obviously NOT be heavy users. Mostly running programs off the server for learning how to type, drawing pictures, and who know what else a 5 year old does at school! 30% are 5th-8th grade, and at this point they do file sharing, use email, research on the internet, etc, etc...medium users I'd say. The other 40% are 9th-12th grade, and I would say MOST of them are medium to heavy users. Internet, email, file sharing, many programs that run off the servers, etc. I also have probably 100 faculty. They would be the heaviest users, video conferencing between schools, led by the teachers, lots of email and internet work/research, file sharing, grades are done over the network, attendance goes to a database, etc. I would say that the High School and Junior High would generate 70% of all the traffic.
Some counts:
High School: 8 labs with 30 computers each and a printer per lab, 60 classrooms with a computer per room, maybe another 50 uncounted.
Junior High: 2 labs with 40 computers each, and a printer per lab, 50 classrooms with a computer per room, maybe another 25 uncounted, and 2 more printers.
Elementary buildings: 1 lab with 40 computers, 40 classrooms with a computer per room, and maybe 5 printers.
Total: 1500 computers total, 40 printers...alright, so maybe I overestimated a bit in the beginning! oops!

If I were to connect everything right this minute, then 90%+ of the traffic would be file, email, and internet access. From that view 100mbps would be fine, but the money is there for gigabit, and I forsee the usage really increasing in the future. Design objectives: high speed and reliability would be the two main things.

 

[CBuxton]

CTR-yes, this is a grant, half of the $$$ is going to new workstations, the other half is what I get for the network. I've got about $40k to burn. I'm game to draw up the cheaper solution, but the $$$ are enough that I could do the gigabit core switch.

 

[CTR]

Spidey:

The job market is good around Dallas, a little sketchy everywhere else. PM me with your salary req's, etc. and I'll tell you what I think your chances are. I have a few connections in the area that might help you get a decent job.

CBuxton:

$40K? I don't think that's quite enough for Gigabit. You're going to need 16 GigE interfaces just for a non-redundant WAN. Yowza. Find a Cisco reseller with a hot (female) rep and lead her on with that $40K Purchase Order until she puts out. What kind of educational discount are you getting? Cisco gives nice ones. Educational discounts, that is. You weren't still thinking about that hot sales rep, were you?

 

[CBuxton]

Hot sales rep, where!?

Check the prices on the first page of this thread to see what I was thinking.

 

[CTR]

Well one problem I see is that you still have the hubs, and no matter how much money you pour into your network those hubs are going to kill the performance. Maybe you should dedicate some of your budget to some nice LAN switches. And are you sure about that price on the Cisco L3 switch? With L3 license, etc.? And I think you'd want to go with a 4912 instead of the 4908. You don't have any room for expansion on your core with that 4908. At any rate your design is solidified in your mind, and I think it's time for some hardware pricing. Contact a Cisco VAR and get prices for all the equipment you might be interested in. Sometimes they will surprise you with those educational discounts. You might even be able to get something better than those 3coms, and also be able to afford some kind of good LAN switch. Did I mention Extreme makes good ones? Cisco's can auto-detect your VLAN's from other Cisco's though, and that is nice too. If you put together an all-Cisco network design, you can usually squeeze the VAR for some better pricing. Cisco will typically cut the VAR a sweetheart deal on the equipment to make up the difference. I used to work for a VAR, and I used to di deals like that all the time, for the clients I liked!

For the record, Dell give kickass discounts for educational institutions as well. Nice servers and the PC's aren't bad either.

 

[CBuxton]

I took my information from this page: Cisco 4908G-L3 Datasheet, if you scroll to the bottom you get the ordering information, I don't see any license for L3, only for RMON. Do you guys recommend the redundant power supply? Anything else I should go for? I would very much like to go with the Cisco 4912, but it is only a L2 switch, so I guess if I need to add ports I'll have to buy another 4908 and get a pair of stacking GBIC's. I love Dell's too, we've got about 75 workstations from them, cheap, fast, and much easier than building all the machines. I should mention that as I have extra money, the hubs are being replaced. Slowly but surely they will be gone. I'm also looking at the wiring we have right now. Like Shadow said in some earlier posts, I'm trying to get one line from the "main" switch for each building out to the switches, and eliminate the switch--hub--hub--hub type of things we have going on now!

 

[CBuxton]

Okay, I'm looking at Cisco equipment, and trying to compare it with the 3Com, because like you say, I should be able to get a deal if I go all Cisco. Anyway, I found the Catalyst 3524XL switch that I would use in place of the 3300's, question is, how can I compare the two? I know the Cisco has a fabric of 10.8Gbps, and forwarding rate of 6.5Mpps, and a forwarding bandwidth of 5.4Gbps (WTF is that!?). Where can I find the specs for the 3Com units? Their website has nothing. The Cisco's are a bit more expensive, but I could swing the price difference if they are worth it.

 

[CBuxton]

One more topic I haven't discussed yet is VLANs, at this point I don't think I will be using them, so that everybody is on the same level and has the same access, any differing opinions and why?

 

[spidey07]

You are gonna have to implement VLANs. It is through these virtual networks that you control and limit your broadcast domains and actually a L3 switch routes between VLANs. If your core switch is a L3 switch then a simple solution would be to use VLANs 10-16 for the central site, and then 20 for building A, 30 for building B, etc.

You should'nt have to trunk from the central out since each site only has 255 machines.

As far as comparing the switches, PPS forwarding and backplane speed are everything. I've yet to actually overrun a good switch.

 

[CBuxton]

I've used VLANs before, but we implemented them to specifically isolate a whole network, meaning NO communication across VLANs. For my setup, the servers would be in one VLAN, and each building in its own separate one, is this going to be possible? Good news is that I think I can use the Catalyst 3524 at each building, so they should be able to automatically download the VLANs from the core. Do you then bridge the VLANs together in the core?

 

[spidey07]

If you bridge the VLANs (very bad practice) then you eliminate the benefits of keeping broadcast domains small.

The L3 switch (router) actually routes between VLANs. This is how a L3 switch works. (alright, nobody flame me on this one...I'm NOT going to go into the differences between a switching router and a routing switch)

Check out the diagram I e-mailed you.

I still think you need a VAR or maybe even Cisco to do the design for you/with you. The questions you are asking are basic to this kind of network. If you are unfamiliar with how it works, please get some help. I'd hate to see you spend all this money and then the network run like crap because of configuration/design issues. Designing a network is much more than picking equipment. Normally you do the design first and pick the equipment that serves the application, not build a network around the equipment.

spidey

 

[CBuxton]

My configuration for the 4908G-L3 would be something like this...please correct me if I'm wrong:

On each Gigabit port:
Assign an IP address from the subnet associated with that port
Assign a VLAN to that port, and enable 802.1Q for the VLAN to allow routing between VLANs

On the switch:
Enable 802.1Q to support routing betwen VLANs
Assign a default gateway to the proxy server for internet access
Add the static routes
Enable DHCP relay

What other commands do I need to configure? Do I need to enable MLS (multilayer switching) on each port?

 

[spidey07]

Create VLANS - name them, set spanning tree priorities, all the layer2 stuff
Assign each gig port to the appropriate VLAN

Hook everything up. Now you should have separate networks for each VLAN. You will not need 802.1q trunking. All ports will be access ports assigned to a specfic VLAN.

On with the next layer - 3. Assign your VLAN routed interfaces IP addresses. These are virtual interfaces that correspond to the VLAN IDs. At this point the switch is the only router in the network and will forward frames correctly. Setup DHCP relay. Place default route into switch/routers routing table to proxy server. Make sure proxy server default gateway is correct. Watch your masks and IP address assignments.

This is my last post...please contact your VAR

 

[CBuxton]

Thanks for your help Spidey...it was MUCH appreciated!

 

[NeoSolo]

This was most interesting thread that i have read in this network forum..

 

[midnightrat]

this is why i read anandtech!!

quick question for all of you, what kind of cert's do you guys hold, and for someone starting out what kind of environment would i learn the most in, eg. isp, large corp, small corp, etc.

 

[CBuxton]

I'm still in college, will graduate one year from now. I've got an MCSE, and CCNA is next in line. I interned for a semester and a summer at Firestone, and I will tell you that working on a network of that scale give you an amazing amount of knowledge. I have worked with the best equipment money can buy (before they had all this recall business!), it's the only way I can even consider doing this project.

Neo-glad you had a good time reading..I did too, and learned a lot as well!