• We’re currently investigating an issue related to the forum theme and styling that is impacting page layout and visual formatting. The problem has been identified, and we are actively working on a resolution. There is no impact to user data or functionality, this is strictly a front-end display issue. We’ll post an update once the fix has been deployed. Thanks for your patience while we get this sorted.

Odd network problem, some private IP can get outside and some cannot (medium sized Cisco network) *SOLVED thanks alrox*

M

Mucman

Diamond Member
Wow! Been a while since I've done a panic post in here for a while 😛

I come in to work today after being off for a week, and I am now in charge for the rest of the week while my boss goes on vacation. Of course he is the
only one who really knows his network stuff, while I am still learning. I login to my machine and notice that it can't get outside our network. Our private IPs
are delivered via DHCP from our Windows Domain Controllers. I don't have a problem getting an IP and I can ping all of the public IPs we manage... The only
reason I can post this message is because I went to another machine in the office and it can get out!!!! I tried a couple other machines, and some can get out
and some cannot. I'm really not sure where to start so I will begin by listing some info, please let me know what else you need, I have enabled access to all of
our Cisco equipment.

whole whack of network info


ipconfig /all

Windows IP Configuration

Host Name . . . . . . . . . . . . : smuc
Primary Dns Suffix . . . . . . . : intranet.pacific.ca
Node Type . . . . . . . . . . . . : Hybrid
IP Routing Enabled. . . . . . . . : No
WINS Proxy Enabled. . . . . . . . : No
DNS Suffix Search List. . . . . . : intranet.pacific.ca
pacific.ca



Ethernet adapter Local Area Connection

Connection-specific DNS Suffix . : intranet.pacific.ca
Description . . . . . . . . . . . : 3Com EtherLink XL 10/100 PCI For Complete PC Management NIC (3C905C-TX)
Physical Address. . . . . . . . . : 00-04-76-24-45-05
Dhcp Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes
IP Address. . . . . . . . . . . . : 192.168.2.2
Subnet Mask . . . . . . . . . . . : 255.255.252.0
Default Gateway . . . . . . . . . : 192.168.1.2
DHCP Server . . . . . . . . . . . : 192.168.1.253
DNS Servers . . . . . . . . . . . : 192.168.1.253
192.168.1.254
Primary WINS Server . . . . . . . : 192.168.1.253
Secondary WINS Server . . . . . . : 192.168.1.254
Lease Obtained. . . . . . . . . . : Saturday, December 28, 2002 9:16:22 AM
Lease Expires . . . . . . . . . . : Sunday, January 05, 2003 9:16:22 AM


router print

===========================================================================
Interface List
0x1 ........................... MS TCP Loopback interface
0x10003 ...00 04 76 24 45 05 ...... 3Com EtherLink XL 10/100 PCI For Complete PC Management NIC (3C905C-TX)
===========================================================================
===========================================================================
Active Routes:
Network Destination Netmask Gateway Interface Metric
0.0.0.0 0.0.0.0 192.168.1.2 192.168.2.2 20
127.0.0.0 255.0.0.0 127.0.0.1 127.0.0.1 1
192.168.0.0 255.255.252.0 192.168.2.2 192.168.2.2 20
192.168.2.2 255.255.255.255 127.0.0.1 127.0.0.1 20
192.168.2.255 255.255.255.255 192.168.2.2 192.168.2.2 20
224.0.0.0 240.0.0.0 192.168.2.2 192.168.2.2 20
255.255.255.255 255.255.255.255 192.168.2.2 192.168.2.2 1
Default Gateway: 192.168.1.2
===========================================================================
Persistent Routes:
None

66.51.160.60 is our mailserver
Pinging 66.51.160.60 with 32 bytes of data:

Reply from 66.51.160.60: bytes=32 time<1ms TTL=128
Reply from 66.51.160.60: bytes=32 time=1ms TTL=128
Reply from 66.51.160.60: bytes=32 time<1ms TTL=128
Reply from 66.51.160.60: bytes=32 time<1ms TTL=128

Ping statistics for 66.51.160.60:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 1ms, Average = 0ms

Tracing route to smtp.paconline.net [66.51.160.60]
over a maximum of 30 hops:


1 2 ms 1 ms 1 ms 192.168.1.2
2 1 ms <1 ms <1 ms smtp.paconline.net [66.51.160.60]

Trace complete.

24.77.232.81 is my home

Pinging 24.77.232.81 with 32 bytes of data:

Request timed out.

Ping statistics for 24.77.232.81:
Packets: Sent = 1, Received = 0, Lost = 1 (100% loss),
Control-C

Tracing route to h24-77-232-81.vf.shawcable.net [24.77.232.81]
over a maximum of 30 hops:

1 2 ms 1 ms 1 ms 192.168.1.2
2 2 ms <1 ms <1 ms gateway.paconline.net [66.51.160.1]
3 * * * Request timed out.
4 * *

pinging from our router

c7204-vxr#ping
Protocol [ip]:
Target IP address: 24.77.232.81
Repeat count [5]:
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface: 192.168.1.1
Type of service [0]:
Set DF bit in IP header? [no]:
Validate reply data? [no]:
Data pattern [0xABCD]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 24.77.232.81, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
c7204-vxr#

running config for private network

interface FastEthernet0/0
ip address 192.168.1.1 255.255.252.0
ip access-group 100 in
no ip unreachables
no ip proxy-arp
ip route-cache flow
no ip mroute-cache
full-duplex
no cdp enable
!

c7204-vxr#show access-lists 100
Extended IP access list 100
permit ip 192.168.0.0 0.0.3.255 any (6786152 matches)
deny ip any any (179 matches)
c7204-vxr#

The only other device I can think of being a problem is a Cisco 3600 which is running as a firewall (hence the gateway 192.168.1.2 in my ipconfig). I know
absolutely nothing about how it works, but I can get telnet to it if you need info from that machine... We also have implemented VLANS but I don't think
it has to do with that because I can get a network connection and ping all of our other IPs.

Here is some info from the machine I am typing this post :

ipconfig /all

Windows IP Configuration

Host Name . . . . . . . . . . . . : blum
Primary Dns Suffix . . . . . . . : intranet.pacific.ca
Node Type . . . . . . . . . . . . : Hybrid
IP Routing Enabled. . . . . . . . : No
WINS Proxy Enabled. . . . . . . . : No
DNS Suffix Search List. . . . . . : intranet.pacific.ca
pacific.ca

Ethernet adapter Local Area Connection:

Connection-specific DNS Suffix . : intranet.pacific.ca

Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Network Connection
Physical Address. . . . . . . . . : 00-08-74-24-BA-22
Dhcp Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes
IP Address. . . . . . . . . . . . : 192.168.2.4
Subnet Mask . . . . . . . . . . . : 255.255.252.0
Default Gateway . . . . . . . . . : 192.168.1.2
DHCP Server . . . . . . . . . . . : 192.168.1.253
DNS Servers . . . . . . . . . . . : 192.168.1.253
192.168.1.254
Primary WINS Server . . . . . . . : 192.168.1.253
Secondary WINS Server . . . . . . : 192.168.1.254
Lease Obtained. . . . . . . . . . : Saturday, December 28, 2002 9:17:46 AM
Lease Expires . . . . . . . . . . : Sunday, January 05, 2003 9:17:46 AM

route print
===========================================================================
Interface List
0x1 ........................... MS TCP Loopback interface
0x20002 ...00 08 74 24 ba 22 ...... Intel(R) PRO/1000 MT Network Connection
===========================================================================
===========================================================================
Active Routes:
Network Destination Netmask Gateway Interface Metric
0.0.0.0 0.0.0.0 192.168.1.2 192.168.2.4 20
127.0.0.0 255.0.0.0 127.0.0.1 127.0.0.1 1
192.168.0.0 255.255.252.0 192.168.2.4 192.168.2.4 20
192.168.2.4 255.255.255.255 127.0.0.1 127.0.0.1 20
192.168.2.255 255.255.255.255 192.168.2.4 192.168.2.4 20
224.0.0.0 240.0.0.0 192.168.2.4 192.168.2.4 20
255.255.255.255 255.255.255.255 192.168.2.4 192.168.2.4 1
Default Gateway: 192.168.1.2
===========================================================================
Persistent Routes:
None

Thanks in advance! (sorry for the crappy formatting of some of the shell output)


 
check routers for any routing activity or a flapping route, check serial interface errors or resets.

debug ip routing
debug ip bgp peer
sh ip bgp peer

Is there a single point or device that is the start of an "unreachible" problem? Firewall access lists maybe? NAT statements on the firewall?
 
Thanks spidey07

"debug ip bgp peer" wasn't an option so I just did "debug ip bgp".

c7204-vxr#show ip bgp peer

BGP peer-group is GT, remote AS 6539
BGP version 4
Minimum time between advertisement runs is 30 seconds

For address family: IPv4 Unicast
BGP neighbor is GT, peer-group external, members:
216.18.31.99
Index 1, Offset 0, Mask 0x2
Inbound soft reconfiguration allowed
Incoming update prefix filter list is gt-in
Outgoing update prefix filter list is bgp-out
Incoming update AS path filter list is 15
Outgoing update AS path filter list is 10
Update messages formatted 12, replicated 0

BGP peer-group is AT&T, remote AS 15290
BGP version 4
Minimum time between advertisement runs is 30 seconds

For address family: IPv4 Unicast
BGP neighbor is AT&T, peer-group external, members:
209.82.86.101
Index 2, Offset 0, Mask 0x4
Inbound soft reconfiguration allowed
Incoming update prefix filter list is at&t-in
Outgoing update prefix filter list is bgp-out
Incoming update AS path filter list is 16
Outgoing update AS path filter list is 10
Update messages formatted 6, replicated 0
c7204-vxr#

I don't have a clue what you mean about flapping routes 🙁, nor do I know how to check for errors on the serial interfaces... I hate to say it, but I need
a little hand holding here. In the tracert to my home IP you can see that I reached our gateway router (which would be past our firewall). Like I said, I know
diddily about our firewall configuration... just tell me the commands and I will show you the output... at least after this, I will know a little more about it.
 
I am starting to wonder how I am getting out at all!!! If I cannot ping from the 192.168.1.1 gateway on the 7200, how the heck is this machine getting out???
 
Originally posted by: alrox
What router/interface is bound to 192.168.1.2/22?
Click to expand...

192.168.1.2 is define on the 3600 (NAT/firewall)

show run
interface Ethernet0/0
ip address 192.168.1.2 255.255.252.0
ip access-group 101 in
no ip redirects
no ip unreachables
no ip proxy-arp
ip nat inside
ip inspect firewall in
no ip mroute-cache
full-duplex
no cdp enable
!
interface FastEthernet1/0
ip address 66.51.160.254 255.255.255.0
ip access-group 102 in
no ip redirects
no ip unreachables
no ip proxy-arp
ip nat outside
no ip mroute-cache
duplex auto
speed auto
no cdp enable
!
ip nat translation timeout 3600
ip nat pool Pacific 66.51.160.129 66.51.160.158 prefix-length 27
ip nat inside source list 10 pool Pacific
ip classless
ip route 0.0.0.0 0.0.0.0 66.51.160.1
no ip http server
ip pim bidir-enable
!
logging trap debugging
access-list 1 permit 192.168.2.0 0.0.0.255
access-list 1 deny any log
access-list 101 permit ip host 192.168.1.1 host 192.168.1.2
access-list 101 permit ip host 192.168.2.50 any
access-list 101 permit ip host 192.168.1.50 host 192.67.9.219
access-list 101 permit icmp any any
access-list 101 permit udp 192.168.2.0 0.0.0.255 any eq domain
access-list 101 permit tcp 192.168.2.0 0.0.0.255 any eq 443
access-list 101 permit tcp 192.168.2.0 0.0.0.255 any eq www
access-list 101 permit tcp 192.168.2.0 0.0.0.255 host 66.51.160.50
access-list 101 permit tcp 192.168.2.0 0.0.0.255 any eq ftp
access-list 101 permit tcp 192.168.2.0 0.0.0.255 any eq whois
access-list 101 permit tcp 192.168.2.0 0.0.0.255 any eq smtp
access-list 101 permit tcp 192.168.2.0 0.0.0.255 any eq telnet
access-list 101 permit tcp 192.168.2.0 0.0.0.255 any eq 1433
access-list 101 deny ip any any
access-list 102 permit icmp any 66.51.160.128 0.0.0.31
access-list 102 deny ip any any
no cdp run


Hmm... I just tried pinging my home computer from the 3600 and I cannot do it.



 
Are the deny counters incrementing for access-lists 101 and 102? Just from a glance, the subnet mask on access-list 101 appears to be wrong, your DHCP server is serving out a /22, the ACL's are only allowing 192.168.2.0/24. You can try to disable both acl's for a second and see if your pings go through then. If that works, just review your ACL's usage.
 
How about a tracert from the computer you are on right now (the one that can get out) to your house? Also, what is 192.168.2.4 and 192.168.2.2, the addresses that shows up in most of your routing tables?
 
Originally posted by: rbayer
How about a tracert from the computer you are on right now (the one that can get out) to your house? Also, what is 192.168.2.4 and 192.168.2.2, the addresses that shows up in most of your routing tables?
Click to expand...

192.168.2.4 is the the address of the computer that I am on now (which can get out), 192.168.2.2 is the IP of my computer in my office which cannot get out.

C:\Documents and Settings\smuc>tracert 24.77.232.81

Tracing route to h24-77-232-81.vf.shawcable.net [24.77.232.81]
over a maximum of 30 hops:

1 2 ms 2 ms 1 ms 192.168.1.2
2 3 ms 3 ms 3 ms gateway.paconline.net [66.51.160.1]
3 6 ms 3 ms 6 ms 209.82.86.101
4 4 ms 3 ms 4 ms pos5-1.core2-van.bb.attcanada.ca [216.13.203.13]
5 4 ms 3 ms 3 ms srp2-0.gwy1-van.bb.attcanada.ca [216.191.65.235]
6 3 ms 3 ms 3 ms rc1wh-pos3-1.vc.shawcable.net [66.163.69.61]
7 4 ms 3 ms 3 ms rd1wh-pos15-0.vc.shawcable.net [66.163.69.33]
8 5 ms 4 ms 7 ms 24.77.232.1
9 66 ms 35 ms 29 ms h24-77-232-81.vf.shawcable.net [24.77.232.81]

Trace complete.

c3620#show access-list
Standard IP access list 1
permit 192.168.2.0, wildcard bits 0.0.0.255 (18 matches)
deny any log
Extended IP access list 101
permit ip host 192.168.1.1 host 192.168.1.2 (1458 matches)
permit ip host 192.168.2.50 any
permit ip host 192.168.1.50 host 192.67.9.219 (156 matches)
permit icmp any any (304 matches) <------ this is incrementing (my machine is doing `ping 24.77.232.81 /t` and they aren't responding)
permit udp 192.168.2.0 0.0.0.255 any eq domain
permit tcp 192.168.2.0 0.0.0.255 any eq 443 (64426 matches)
permit tcp 192.168.2.0 0.0.0.255 any eq www (8984 matches)
permit tcp 192.168.2.0 0.0.0.255 host 66.51.160.50
permit tcp 192.168.2.0 0.0.0.255 any eq ftp
permit tcp 192.168.2.0 0.0.0.255 any eq whois
permit tcp 192.168.2.0 0.0.0.255 any eq smtp
permit tcp 192.168.2.0 0.0.0.255 any eq telnet (1397 matches)
permit tcp 192.168.2.0 0.0.0.255 any eq 1433 (36 matches)
deny ip any any (6901 matches)
Extended IP access list 102
permit tcp host 216.239.51.119 eq www host 66.51.160.133 eq 2947 (40 matches)
permit tcp host 216.239.51.119 eq www host 66.51.160.133 eq 2946 (28 matches)
permit tcp host 66.51.160.61 eq www host 66.51.160.133 eq 3181 (406 matches)
permit tcp host 66.51.160.61 eq www host 66.51.160.133 eq 3180 (355 matches)
permit tcp host 24.77.232.81 eq 443 host 66.51.160.133 eq 2888 (93923 matches) <---- this is my ssh connection to my home machine from the computer I am at now
permit icmp any 66.51.160.128 0.0.0.31 (16009 matches)
deny ip any any (109859 matches)
c3620#

alrox, the reason for the /24 being firewalled is that all the other private networks are denied outside access. The 192.168.2.0 portion of the /22 is the only allowed
section of the subnet that is allowed out (the whole block is allocated via DHCP).

So can we rule out the Cisco 3600 firewall from this problem? It seems like my pings are going past it (perhaps the replies aren't coming back???)
 
it could be your 'ip nat inside source list 10 pool Pacific'. You haven't mention acl 10 yet, perhaps you mean access-list 1 for the ip nat inside command?
 
Originally posted by: alrox
it could be your 'ip nat inside source list 10 pool Pacific'. You haven't mention acl 10 yet, perhaps you mean access-list 1 for the ip nat inside command?
Click to expand...

Ahh... I think you are right! Especially after looking at the examples
here. Now how do I drop the existing
config and replace it with the new one? I tried going "conf t" and then "ip nat inside source list 10 pool Pacific" but I got an error "%Pool Pacific in use, cannot replace"

Some more show commands

c3620#show ip nat translations
Pro Inside global Inside local Outside local Outside global
--- 66.51.160.129 192.168.2.5 --- ---
--- 66.51.160.130 192.168.2.130 --- ---
--- 66.51.160.133 192.168.2.4 --- ---

According to this it looks like only three machines in our network have working NAT translations!

c3620#show ip nat statistics
Total active translations: 3 (0 static, 3 dynamic; 0 extended)
Outside interfaces:
FastEthernet1/0
Inside interfaces:
Ethernet0/0
Hits: 30822261 Misses: 386
Expired translations: 726
Dynamic mappings:
-- Inside Source
access-list 10 pool Pacific refcount 3
pool Pacific: netmask 255.255.255.224
start 66.51.160.129 end 66.51.160.158
type generic, total addresses 30, allocated 3 (10%), misses 0
c3620#
 
configuring from a terminal, do a 'no ip nat inside source list 10 pool Pacific', then 'ip nat inside source list 1 pool Pacific', then 'exit' and type 'write memory' from the enable prompt.
 
Originally posted by: alrox
configuring from a terminal, do a 'no ip nat inside source list 10 pool Pacific', then 'ip nat inside source list 1 pool Pacific', then 'exit' and type 'write memory' from the enable prompt.
Click to expand...

c3620(config)#no ip nat inside source list 10 pool Pacific
%Dynamic mapping in use, cannot remove
c3620(config)#ip nat inside source list 1 pool Pacific
%Pool Pacific in use, cannot replace
c3620(config)#

 
hmm, might have to clear all the nat mappings and then redo the source list. Not 100% sure, but you can probably find a good example on cisco.com.
 
Thanks! It looks like it has been solved!

The commands that fixed it :

c3620#clear ip nat translation *
c3620#conf t
Enter configuration commands, one per line. End with CNTL/Z.
c3620(config)#no ip nat inside source list 10 pool Pacific
c3620(config)#ip nat inside source list 1 pool Pacific

Not sure what my boss was doing, but I am going to leave a message with him asking what the heck he was doing... So why were 3 IPs still
working? Would they have not worked once their DHCP lease expired?
 
I don't know why 3 IP's from 192.168.2.0/24 were working, it seems that they shouldn't have. Were any nat config changes made recently? maybe those IP's had previous nat mappings, the config changed, and those 3 were allowed to get through still, but no new ones. *shrug*
 
I wonder if after the "clear ip nat translation *" command, all the machines who could go outside would break? Can't wait to tell my boss that
he broke our network 😛. I am not sure what he was up to since I was away for the week... and he is very bad at letting people know what changes
have been made.
 
You must log in or register to reply here.

TRENDING THREADS

Back
Top