network slowness

[dphantom]

Hoping some of the network/server guys here can help me out. I have been leading a team of server and network folks for the last two weeks looking into a "login slowness" problem at two of our sites. Please note this has been a very long term problem (2+ years) and only now is being addressed. Don't ask why only now...please.

The details:
Site R connects to Site P via a 10MB Charter cable link. Site P connects to our main site via a DS-3. Domain authentication, DNS, WINs is all provided from the main site. No Domain Controllers are located at either Site R or Site P.

Site R has perhaps 15 simultaneous users while Site P may have up to 30.

A computer and user configured exactly as any other user/computer at Site R or Site P can login in at the main facility in a normal time. When the same user/computer is brought to either Site R or Site P, anywhere from 15-30 seconds is added to the log in time. There is no specific time when logins are slower. They are always consistently slower 7x24.

Other off site locations with similar numbers of users/computers, similar connections and no Domain Controllers do not experience any difference in login speeds from the main site.

Network guys see no interface issues nor bandwidth problems. WireShark captures from Site R and Site P show nothing unusual that we have been able to see.

The DS-3 does cross LATAs.

We have considered placing a DC at either or both locations but do not believe this will resolve the "slowness" as we use a 3rd party DNS appliance located at our main facility to provide DNS and DHCP services and other similar off site locations do not need experience any login slowness.

Wireless or wired connections make no difference at Site R or Site P.

I would appreciate any ideas you may have. We have a good team here, but are quickly running out of ideas to test to identify the smoking gun.

 

[imagoon]

Any reason why you use a 3rd party DHCP and DNS? The most common issue you are listing is a DNS issue however...

How is the latency to the site? Is it set up properly in sites and services. Are you 100% sure that there is not an erroneous DNS entry in DHCP? In most cases DNS and DHCP should be provided by the domain controllers.

How is routing handled? Is the network converged? Is the charter link a any:any vpn link, VTI point to point? "etc" What is handling routing at Site P that handles main to site R transit?

 

[dphantom]

no choice but the Infoblox appliance for DNS/DHCP is quite good.

Site and services is correct. Latency is not an issue. The netowrk is converged. Details on the Charter link I will have to get back to you but I believe it is point to point. It is not a VPN link.

Site P and Site R have 3750s handling L3. The network is converged.

We have checked and double-checked DNS entries in DHCP. That too was my first thought. In fact, there was an incorrect DNS entry in DHCP that when corrected saved about 5 seconds on average off login times. Leaving us where we are today.

 

[imagoon]

no choice but the Infoblox appliance for DNS/DHCP is quite good.

Site and services is correct. Latency is not an issue. The netowrk is converged. Details on the Charter link I will have to get back to you but I believe it is point to point. It is not a VPN link.

Site P and Site R have 3750s handling L3. The network is converged.

We have checked and double-checked DNS entries in DHCP. That too was my first thought. In fact, there was an incorrect DNS entry in DHCP that when corrected saved about 5 seconds on average off login times. Leaving us where we are today.

I have never worked with infoblox. Does it properly support and hand out the SRV records for the domain? If those are wrong, you get these issues. Windows DNS is so dynamic, is this thing pulling a feed from a DC someplace?

 

[dphantom]

yes, infoblox handles all srv records. I had never worked with one either before I got here but for the last 3 years have had no problems with it. I have been an AD admin since 2000 and always used AD integrated DNS. But I have high confidence in these appliances and all DCs are running nominally. Our last Microsoft AD health check was excellent.

Infoblox does handle all 10k+ users for us across all our locations.

 

[dphantom]

and latency from ping test is 9ms or less, but generally 9ms to Site R from main or from Site R to main

 

[imagoon]

Is there anything in the Event logs? I had a site that had an issue caused by a filter preventing a protocol from the workstation to the DC's from talking properly... I can't remember... gray matter I/O error. However it was pretty obvious by digging through the event viewer. Does GPO apply properly?

edit:

Now I remember it had to do with the VPN blocking fragmented ICMP causing slow link detection to fail. GPO wouldnt apply because it thought the DC was "down" since the packets were lost rather than failed out.

Maybe ICMP fragments are blocked some place?

It popped up in the netdiag.log

 

[dphantom]

GPO does apply properly. Nothing at all in the event viewer. I will check netdiag though.