• We’re currently investigating an issue related to the forum theme and styling that is impacting page layout and visual formatting. The problem has been identified, and we are actively working on a resolution. There is no impact to user data or functionality, this is strictly a front-end display issue. We’ll post an update once the fix has been deployed. Thanks for your patience while we get this sorted.

Network Speed woes - Analyzation Tools?

T

Transition

Banned
Ok so here's the situation.... Please try to bare with me as my networking knowledge is limited...

I work for a relatively small company (150 people approximately), and we are having some network slowdown issues... Now, our main location is divided into two branches (the buildings are accross the street from each other - and from what i understand both branches are operating off the same server.) Now, for some reason one of the branches connection is horribly slow accessing our server-side application. Each branch is from my understanding using a different gateway which routes us to wherever (forgive me for my lack of network knowledge). Branch 1 is acheiving great speeds both to the internet and for our server side applications, while Branch 2 nearly chokes itself to death and our server side application gets so slow during periods of the day that you practically can't work (this never happens to branch 1). Now, our network admin is NOT a guru by any means and he has no clue as to what the issue is (so i figure i'd investigate a little). Where do i start? I mean i have access to EVERYTHING here - no boundaries.. What tools or how can i start to analyze where our network is being tied up at?

P.S. While pinging a computer located on our internal intranet it's not uncommon to lose 50-75% of the data. On the intranet!

Any help is highly appreciated!
 
50-75% packet loss when pinging across buildings! ouch. bad...very bad.

Next question - how are these two buildings connected? Are there any routers in the picture?

qcheck - by netIQ

I feel your pain - that much packet loss is completely unnacceptable. You might want to hire someone to come in and take a look at it.

My guess - cabling between buildings. Saturated link between buildings.
 
Tools you might want:

MRTG - Multi Router Traffic Grapher is a freeware tool that will run on windows & unix to graph statistics on your routers and switches. It can be rather complex, unfortunately, but it's an awesome tool.

Ping Plotter will do a traceroute to a destination and show you where packets are being lost.

LanSpy Network Monitor is one I haven't used, but looks interesting and there's a 14-day demo.

A few questions..

You have Branch 1 that works fine and Branch 2 that doesn't. Where is the physical server? Branch 1, I'm guessing.

Any idea how the two branches are connected together? T1? Ethernet?

Let us know, and we'll try to help.

- G
 
Actually, i'm pinging computers that are located in the same branch and i'm now losing 100% of all data. I'm sitting abotu 50 feet from one of the computers that's currently being used and when i try to hit the IP from my workstation, i get a 100% loss.. Hrm......

I appreciate all the suggestions guys! I'll run some of the diagnostics and see what i can come up with. If i find anything interesting i'll be sure to share it.

Thanks a lot ....

- Ryan Graf
 


<< You have Branch 1 that works fine and Branch 2 that doesn't. Where is the physical server? Branch 1, I'm guessing. >>



Correct, we have a brand spanking new IBM server sittting in Branch 1 - i'll try to get some details about it.





<< Any idea how the two branches are connected together? T1? Ethernet? >>



To the best of my knowledge we are connected together via Ethernet, but i could be wrong.. I'll double check





<< Let us know, and we'll try to help. >>



Thank you for the help!

 
Ok here's some food for thought.....

Ping Plotter Data & Graph

Let me just do a brief run-down of what everything is...

29.1.4.1 - The gateway
10.10.4.1 - mystery. not even the sys admin knows what it is (well at least we call him system admin - as to what he is in actuality is a whole other story)
29.1.1.100 - the server address located in Branch 1 - i am in Branch 2

This is my first time using this utility, but what i do see is a big latency between the gateway and 10.10.4.1 ...... Once again you'll have to forgive my ignorance - but could the 10.10.4.1 be a router? I'm going to have to run the Ping Plotter over in division one to see if we experience the same latency between the gateway and 10.10.4.1- to get some better results...

Anyone have any insight to share?
 
Ok more info...

I was just over in the other branch and i ran a basic trace route in DOS, and to my surprise going from anywhere in Branch 1 to 29.1.1.100 is a direct connection - only 1 hop straight to 29.1.1.100 with a very solid connection and no data loss.

arg.. i'm confused - but loving this!
 
Alright here's some good information....

10.10.4.1 is the serial side of the router.. i talked to a tech support person for a company whom's software we're running on our server and he said he's going to attempt to re-prioritize tasks to set Eclipse (our server side software) to be priority over the internet.. Hrm ok...

His diagnosis is there might possibly something wrong with the hub or he mentioned something about a CSU?

Any insight?
 
If your server is in Branch1, there's no network "hops" because it's on the same subnet. Client PC's just connect to it directly via the LAN, no need for routers, etc.

I'd guess than 10.10.4.1 is probably a router interface. If you have a connection between buildings it's likely done through routers. The router-to-router link is just an IP segment that needs to have IP's on both ends. If you ping from Branch 1 to Branch 2, you'll probably pass through 10.10.4.2, which is expected.

The latency (delay) you're seeing is definitely NOT normal. Even if you're on a T1, the latency should be < 20 MS and you're 40 times that.

This could be caused by a few things:

A small connection between branches (small being T1 or lower) that isn't adequate to handle the traffic of your network
Errors on the network that cause retransmission problems.

A few very simple things to check:

Check the switch port that the router is plugged into. Make very sure that it's hard-set to the same speed and duplex that the router is using. It sounds ironic, but sometimes routers and switches don't autonegotiate very well together and this can cause total havoc on a network.

Given that it's the day after christmas, how many people are around? If you see this same kind of response when there's nobody else in the office, you're probably seeing errors on the line or some kind of automated process (workstation backups?) using traffic.

In your office, do you use a switch or a hub? If you're on a hub, there are some network tools you could use to look at the traffic see how much is in use.

Also, can you physically trace down and find the router that connects the buildings? Let us know what model it is and what is plugged into it. This might help us determine how fast it is and how you might find out.

Next note.. Are you guys part of the US Government, specificially the Defense Information Systems Agency? They are the group that's listed owning the 29.x.x.x IP's that you are using. If not, whoever setup your network didn't do you any favors.. It's generally not a good thing to use someone elses' real IP's.. In this case, it's not that big of a deal, since you probably won't talk to the DOD much, but.. Soemthing to keep in mind.

I hate to say this, but you might want to get a pro in to help figure out what's going on.

- G
 
Not many people are around actually the day after Christmas.. This network problem (to my understanding - has been present for years). I found out another possible contributing factor which is when they ran cable from building to building they apparently only ran 75 pair instead of the proposed 120.

We use hubs in the office - specifically in Branch 2 we're using what appears to be an old "CentreCOM 3024 TR Hub - 10-Base T" hrm only 10-base T.. Along with that we're using an Adrian TSE ACE and a Motorola Vanguard 320 Router. Essentially in Branch 1 we have the same equipment, but the hub is a brand new 3COM, the CSU is a new AMerican Technologies (i believe so), and the router is the same.

The set of IP's we're using are internal.. Strictly for use on our intranet - that is except for our server which is running on 29.1.1.100..

We got someone yesterday that flashed the firmware on our routers so he could re-prioritze the ports for our server software - and make other ports less priority (hope that makes sense).

Otherwise, we have an extra CSU sitting around (American Technologies) that we're gonna try swapping out for the Adrian one in case this re-prioritizing of the routers doesn't allot us the bandwidth we need.

I don't know if i told this before but here's why i'm so concerned about bandwidth over in Branch2.. Branch 1 & 2 are both operating off of the same connection.. Yet, branch 1 gets about 5-6 times faster speeds on the internet and intranet. Branch 2 can only muster a lousy 9kbs/sec while anyone sitting in branch 1 can do an easy 60-70/kbs all day long.. In addition, Branch 1 has twice the users.

I understand that since Branch 2 has to interface with the serial side of the router before hitting the server that's an extra "hop" (or am i using the wrong terminology) to go. Looking at the ping plotter graphs though, the huge slow down is between the router & server..
 
My guess still is a saturated WAN connection. What is the speed of the connection between buildings and how many people are in building2 (the problem one).

Garion has the best bet - if you are getting pitiful response when nobody is in the office then it is probably errors. If you get good response then it is too much traffic.
 
Whats the best way to determine speeds between the buildings?

Branch 1 = Approx. 35 people with networked computers
Branch 2 = Approx. 14 people with networked computers

Could you elaborate a little on what it means when the connection is saturated? As in all the available bandwidth is being used, or.......?

Thanks again for all the help
 
9 people is not very many. By a saturated WAN connection I mean the line is running at 100% utilization, all bandwidth being used. When this happens a router is queuing up frames but can't get them onto the wire fast enough. Kindof like a bucket with a hole in it, if you fill the bucket up faster than water can exit the hole then water flow over the rim onto the floor. That water flowing onto the floor are packets getting dropped. Severe packet loss or drops will make your applications run like poo.

Maybe this is a 56 kbit connection, starting to look that way? What is the exact model number of the ADTRAN?

We're trying to figure out if this is a LAN or a WAN problem, trying to narrow down possibilities. OH! One more thing, when you ping across the WAN connection make sure you set your timeout to very high, that way you can see if the packets are trully getting lost or just held up in the routers queues.

ping x.x.x.x -t -w 10000

-t means constant
-w msec timeout, above would be 10 seconds.

if you get replies in the 2000-5000 msec range then at least your frames are being delivered and you can start looking at bandwidth utilization and queue depth on the routers. 9 people would be really hard pressed to fill up a T1. Get into the routers if you can and look at the serial port.

 
Here's how the breakdown is on the equipment.......


Branch 1
- Centrecom 3024 TR Hub (10-Base T)
- Adtran TSU ACE
- Motorola Vanguard 320 Router

Branch 2
- [2] 3COM Super Stack 3 Switch 3300XM Hub's
- American Technology 1510 T1 DSU/CSU
- Motorola Vanguard 6435 Router

EDIT: Adrian = Adtran .. Oops Also, the only thing i could find was the P/N on the Adtran which is 1200295L1
 
OK, so you are running a T1 connection between the two buildings, not a lot of bandwidth. I would try and find some kind of a sniffer program (such as Ethereal) which is free in Branch 2. Try to figure out what traffic is going across the network to the other side. Someone might have done something dumb like configure workstation backups, etc. across the T1.

If your buildings really are right across the street from one another, you might be able to do some kind of a wireless link between them. if you were to spend some small cash, you could use 802.11b (but that's got some serious security implications), larger cash could get you some faster and/or more secure connections.

It is possible that your application is simply not designed well for low-speed connections - If it depends on a mapped drive in any way, you're going to have a problem. Remember a T1 is 1.5Mb/s, shared among all the users of Branch B for all network activities. Not very much!

- G
 
Here's what happens when i try to ping someone on the WAN...

C:\WINDOWS>ping 29.1.1.27 -t -w 10000

Pinging 29.1.1.27 with 32 bytes of data:

Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for 29.1.1.27:
Packets: Sent = 5, Received = 0, Lost = 5 (100% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms

or... here's another IP accross the WAN (which i KNOW is active and in use as we speak)

C:\WINDOWS>ping 29.1.1.27 -t -w 10000

Pinging 29.1.1.27 with 32 bytes of data:

Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for 29.1.1.27:
Packets: Sent = 5, Received = 0, Lost = 5 (100% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms

EDIT I posted the same IP address twice.. While i actually did try 2 - and had the same results...

Fawwwk. 🙁
 
and here's what a typical tracert looks like across the WAN...

C:\WINDOWS>tracert 29.1.1.27

Tracing route to 29.1.1.27 over a maximum of 30 hops

1 16 ms 9 ms 11 ms 29.1.4.1
2 559 ms 493 ms 351 ms 10.10.4.1
3 * * * Request timed out.
4 * * * Request timed out.
5 * * * Request timed out.
6
 

Could do a check at the router/firewall/proxy to see what the gateway, network mask & DNS is.

Also check the router/firewall rule by running ip route

It is very strange that your trace hit the ip and it is an external IP, but then you get an internal IP after that.
 
Odd... Check this out

C:\WINDOWS>tracert 29.1.1.5 -w 10000

Tracing route to 0.0.39.16 over a maximum of 30 hops

1 9 ms 9 ms 9 ms 29.1.4.1
2 10.10.4.1 reports: Destination net unreachable.

Trace complete.


I believe i'm now in well over my head.. It's kind of an odd situation how our network is administered (or rather how it isn't being administered) because of the system we're using. Our server runs some software (Eclipse) to handle all of our customer information, purchase orders, etc etc.. Basically, it's the backbone of our business. The server is setup and maintained by Eclipse along with all of our network configurations. Ok great, the company is in Colorado while we're in Minneapolis. The hard part is we need to keep Eclipse in the loop so to speak. I'd love to go contract someone out to do some network analyzation and upgrading but that fact is if we knock Eclipse out of the loop and do some updates / changes their not aware of - then they won't support us anymore (which means we're screwed).
 

I don't think "Eclipse" has something to do with your network, but if it does then it has to be some kind of software firewall/router.

You might want to setup a computer on an external IP from work and one at home for VPN testing. Then switch over to a secure tunnel once you feel comfortable with it and run your Eclipe software as if it is on a LAN.
 


<< I don't think "Eclipse" has something to do with your network, but if it does then it has to be some kind of software firewall/router.

You might want to setup a computer on an external IP from work and one at home for VPN testing. Then switch over to a secure tunnel once you feel comfortable with it and run your Eclipe software as if it is on a LAN. >>



Oh i don't think it has anything to do with Eclipse rather then them not setting up our network properly.. It definitely sounds like either lack or bandwidth as stated before - or some of the hardware is mis-configured.. Like others have stated, there are a lot of variables and potential problems
 

Ping from the computer the top of the IP range (.254 etc...) and one at the bottom of the IP range to see if you can get out.

can you ping all computers with real IP address at Branch1 internally or from Branch2?
 


<< Ping from the computer the top of the IP range (.254 etc...) and one at the bottom of the IP range to see if you can get out.

can you ping all computers with real IP address at Branch1 internally or from Branch2? >>



How do i do this? And, what do you mean by "can you ping all computers with real IP address at Branch1 internally or from Branch2?"

- RJ
 
Find out the leased ip at your DHCP server, Router/Firewall is the easyest way.

router/firewall: at command promt type ip neigh to see if you could see all leased IPs.

at each individual station: at command promt run netstat -r or ipconfig to get the IPs then ping your gateway.

The "real IPs" are IPs that was given to you from the ISP provider.



 
You must log in or register to reply here.

TRENDING THREADS

Back
Top