Network Mystery

[CubanCorona]

My apartment building has a network with about 30 computers on it. The computers are plugged into a 16 port hub and a 24 port hub which are both plugged into a router w/ a built in 4 port hub. The internet connection is a 1.5 mbit DSL line. Sometimes the connection is very, very fast, but other times, it is so slow that it is impossible to load any webapages or sign on AIM. It's not that it is extremely slow, it's that it doesn't seem to work at all during those times.

For a while I thought the problem might be a worm on one or more computer. However, looking at data from a packet sniffer and running SNORT turned up nothing. If I plug any computer directly into the router/hub it works beautifully. I'm beginning to think that one of the hubs may be bad. How can I test this theory? Does anyone else have any ideas? I'm completely stumped.

Willow

 

[deran]

You should look at the lit of the switch. If all lits are flashing very fast and non stop then sound like your network has worms. If so, it will give you the hard time to pin point which pc has worm.

 

[CubanCorona]

I've used a packet sniffer to scan the network for suspicious activity when it is slow. I didn't discover any range pining or attempted connections on tcp ports that worms generally use.

 

[Matthias99]

How are your packet collision levels? Hubs can be notoriously slow if multiple people are downloading files, etc. through the WAN connection. Moving to a switched network (like maybe 4 8-port switches rather than 2 large hubs) will help if your problem is lots of collisions -- but nothing will help if someone is hogging all the bandwidth. In particular, aDSL usually has a fairly small upload window, so if *anyone* on your network is running, say, KaZaa, you're hosed, since all your upstream bandwidth will be eaten up and it will be very hard to send any packets out to the Internet.

 

[CubanCorona]

That is something that I have wondered about. How can I monitor the collision levels?

If someone is hogging the bandwidth, would that cause this much of a slow down?? It gets to the point where a web page will not even load. Why should his packets have a higher priority than mine or anyone elses?

I believe it is a sDSL connection, but it's not my network so I don't know. I just live in the apt =).

I have quite a bit of network experience, and I'm a CS major but this has completely stumped me. I think my landlord is ordering switches as I type this, so I'm anxious to see if that will remedy the situation. In the mean time, how can I monitor the collision levels?

I can monitor the traffic from each host on the network, but how can I be sure that one host or another is "hogging" the bandwidth?

Willow

 

[deran]

It could be 1 or more NIC go bad. It was because when the bad NIC will send out alot giant & runt over the network and eat up the bandwidth.

 

[spidey07]

You either have a LAN problem or a WAN (the t1) problem.

If you plug directly into the router while other people are using the service and don't experience any trouble then I'd say you have a LAN problem. Getting rid of the hubs is the first place to start.

When hubs or network cards go bad very strange things start happening.

 

[CubanCorona]

It's not a WAN problem. If I plug directly into the router and unplug the hubs, it works beautifully. However, if I plug directly into the router and do NOT unplug the hubs, the problems persist. This would be expected, however since the router's built in hub does not break up the collision domain. Since the network is all hubs, the collision domain consists of the entire LAN. Any more ideas?

Willow

 

[Matthias99]

First off, the router almost certainly has a built-in switch. I've almost never seen a router built into a hub.

Second, almost any network monitoring software (even the crappy display in the WinXP Task Manager) should show you statistics on packet collisions. Most routers/hubs will also flash orange lights when there are collisions going on (as opposed to green for regular traffic).

As for bandwidth hogs, unless your routers or switches are doing some sort of QoS or traffic shaping, it's easy to get in trouble. Let's say there's one busy user trying to send 1000 packets/sec. out to the internet (which, we'll say, is already more than the router can handle), and you're trying to send 10/sec. Some routers, if too busy, will discard packets evenly from different interfaces. If he loses a few packets per second it's no big deal -- most likely it's just the middle of some file transfer and they'll be resent in a few seconds. If you lose a few packets per second, nothing will work from your end, since you'll have a really hard time opening connections to anything when your SYN/ACK packets are constantly getting lost. A more sophisticated router would be able to handle load balancing -- that is, making sure that the available bandwidth is shared fairly among all the connected users.

That said, if you're getting problems with nothing hooked up but the hubs, then one or both of your hubs are no good. With nobody else plugged in, they shouldn't be doing anything.

 

[deran]

What kind of the router are you use? To pin point the problem, try to connect the hub to the router with only 1 or 2 pc. If still eat up the bandwith then the hub maybe bad.

 

[CubanCorona]

The network tab of the Windows XP task manager doesn't seem to have any collision statistics. Could you point me in the right direction to find some monitoring software which would show collision statistics? I've never seen it on any software I've used. The lights aren't going to tell me much though unless I sit there and watch them, and I am not close the the room the setup is in.

I'm pretty sure it's a hub. When I plug directly into the router I can see the traffic of all the hosts on the network. A switch would not allow this.

 

[CubanCorona]

The router is a Netgear 4-port RP114.

 

[CubanCorona]

I would love for my landlord to buy a more sophisticated router with load balancing features, but I doubt he would buy into that. If bandwidth hogging IS the problem, I suppose there's no quick (cheap) fixes.

 

[Matthias99]

It's definitely a 4-port switch -- here's the manual:

ftp://downloads.netgear.com/files/Reference_Guide_for_RP114.pdf

You may be seeing *some* traffic that's being broadcast, or if more if you're directly plugged in and your NIC is in promiscuous mode. The router may also not be configured properly. I misread your earlier comment -- I thought you were plugged into the router directly with nobody else on the hubs and still having a problem. Odds are your hubs are OK. I'm about 99% sure that you have a bad host somewhere flooding the network with packets -- could be a bad NIC or a system with a virus, or just a bad bandwidth hog. The low-tech way to fix this, if it won't get you lynched, is to try pulling them one at a time (try the busiest ports first) until the problem goes away. Then find whoever owns that computer and get them to fix the problem. Getting switches for everyone will localize the issue, but you'll still need to find the broken host.

I *thought* WinXP offered collision monitoring. I'll have to check into it when I'm on my XP box. I'm on a Windows2000 box now, and it doesn't even *have* a Network tab in Task Manager.

 

[CubanCorona]

thanks for the help, the traffic I'm seeing is definitely not broadcast traffic; however the sniffer HAD set my NIC to run in promiscuous mode. Would you mind explaining this though... I was under the impression that a NIC (even in promiscuous mode) can only read the packets in its own collision domain. Isn't that the purpose of a switch? That is to send a packet only to the physical port on which its destination resides?

Well anyway... so it is a switch... I was surprised when I first thought it was hub.

When I've run the packet sniffer, though, I haven't seen anyone *flooding* the network with worm- or virus-like packets. I guess it *is* bandwidth hogging. Usually when it's slow I do notice someone downloading or uploading a lot of data at that particular time.

So I suppose there are no quick fixes for bandwidth limiting? Are there any semi-inexpensive routers that have this feature?

Thanks again for the help.

 

[CubanCorona]

^^ To the top!

 

[Matthias99]

Sorry -- XP definitely doesn't offer collision monitoring in Task Manager. I thought it was in on that screen where you can check off all the stats to look at, but it's not there. I'm surprised your packet sniffer doesn't have it, either -- what are you using?

Edit: of course, if you're plugged directly into a switch, there's also no way you could ever *see* a collision, even if they were happening elsewhere in the network. You'd have to be on a hub. They also might just be reported as lost packets in some programs, or if your hardware has no way to report it.

I'm not sure either why you're picking up other ports' traffic in promiscuous mode on a switch, as I was definitely under the impression that a switching router should, uh, route packets in a switched way! The RP114 is an older design (they don't sell it anymore), so maybe it's got some quirky behavior in there. Or perhaps it's configured in some strange fashion? Have you looked at the web interface and checked for any unusual port forwarding, etc.?

Setting up enterprise-level networks is a bit beyond my expertise, unfortunately, and that's usually the sort of stuff you have to look at in terms of a hardware solution. I know that if you replace the router with a PC equipped with dual NICs and run Linux on it to do your routing, you can do stuff like that (limiting bandwidth by IP, etc.) in software. Unfortunately, I don't have much experience setting that kind of thing up. I was on the client side in tech support, so I never dealt with anything past the switch level in a whole lot of detail.

Maybe someone else can point out some good links? Or has advice? Please?

 

[Abzstrak]

you need a true hub for sniffing it. Chances are if its something like the Nachi worm its sending a crap load of ICMP packets and they are not on the same subnet, so the router is getting flooded..... Sniffing from the built in switch will not show this. Only way I know to sniff on a switch is if its managable and you can use STP.

I agree with the comments above to unplug the systems one at a time to figure out which system(s) is doing it, should only take you about 15minutes that way.

 

[CubanCorona]

Well if I setup the network like this:

Me & Everyone Else
_|__|__|_
|__HUB__|
__| _________ <-- Uplink
|__ROUTER__|

Then I should be able to see all frames on the medium. Correct?

P.S. Still looking for a sem-inexpensive router with load balancing!

 

[Genx87]

To me it is obvious the problem lies in the hubs. Especially since you said the problem is resolved by connecting directly into the router. Remember with a hub you create a large collision domain. With an ethernet network the frame is spammed on the collision domain looking for the correct MAC addy. Each nic will recieve every frame and check if the mac addy. If the mac addy matches the nic's mac addy then the nic accepts the frame. If it is not the correct addy it simply discards the frame. Chances are there is so much traffic going on you are getting a lot of collisions. When a collision occurs the frame is destroyed and has to be resent.

If you get a switched network up and running I would expect the problem to go away as the switch eases the load on the network by sending the frame down the appropriate wire. How many users do you have on this network? I dont believe hubs work very well above 15-25 users. Just too much dead traffic eating up bandwidth.

Getting a router with load balancing wont help in this situation as the problem lies behind the router.

 

[CubanCorona]

Thanks for all the help guys. My landlord has ordered 24 port Netgear switches, so I'll let you know what happens!

 

[CubanCorona]

Look at this!

Pinging www.google.akadns.net [216.239.41.99] with 32 bytes of data:

Reply from 216.239.41.99: bytes=32 time=488ms TTL=52
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Reply from 216.239.41.99: bytes=32 time=71ms TTL=52
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Reply from 216.239.41.99: bytes=32 time=60ms TTL=52
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Reply from 216.239.41.99: bytes=32 time=119ms TTL=52
Request timed out.
Request timed out.
Reply from 216.239.41.99: bytes=32 time=58ms TTL=52

I mean serisouly, WHAT IS GOING ON!?!

 

[Genx87]

Try pinging just the router and see what happens

 

[gunrunnerjohn]

You obviously have a lot of packet loss. As mentioned, try pinging the router and see if it's inside the network, or on the other side of the router.

 

[spidey07]

That's really bad. We need to start isolating the problem to a particular hub. Focus all your attention on the LAN now (meaning ping the router, the gateway address).