Two Cisco 2800's can't ARP each other over same 3560 switch

[Cooky]

Recently ran into an issue, where two routers are physically connected to the same switch:
R1---SW---R2

I triple checked the interface config, the router interfaces are up/up, and have the exact same netmask.
The switchports have the exactly the same and correct vlan assignment.
I also checked the switchports to make sure no storm-control for broadcast was configured.

From all three devices I can see them in "show CDP neighbor", w/ correct ports, and info.
Yet R1 and R2 can't ping or ARP each other.
Their "show arp" indicate each other's IP as "Incomplete", which indicates a L2 arp issue.

Can anyone think of any reason why the ports are up/up, but ARP wouldn't work?

 

[Gryz]

up/up has nothing to do with IP.
If I remember well (it's been many years since I did cisco support) the first up means that the interface is configured to be up ("no shutdown"). And the second up means that the "line protocol" is up. For ppp and hdlc that means the router sees keepalives from the neighbor. For ethernet, I'm not sure what it meant Ethernet does not have keepalives. I think it just means that the interface sees that there is voltage on the wires (there is a device on the other end of the cable). In any case, it says nothing about layer3 (IP) functionality.

I guess it might be a typo/mistake in the IP addresses. Where the combination addresses/masks put them on a different subnet. We can't say more, unless you copy/paste part of the config here.

Have you tried "debug arp" ? Don't enable it if you don't know what you're doing, as the wrong debug on the wrong router in the wrong situation can bring your router to its knees very quickly.

 

[Cooky]

Thanks for the reply.

up/up was mentioned to show interface was not in shutdown state.
Ethernet actually uses keepalives.
IP & netmask were checked multiple times to make sure no typo was made, and they're in same subnet.
Very tempted to do debug, but it's not allowed on these particular devices due to some reasons I won't mention here.

Below are the config and arp table.
R1 can communicate w/ another router through vlan 12 just fine, so problem is isolated to vlan 14, possibly between R2 and SW1.

R1#sh arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.12.12.1 - 0025.849d.d500 ARPA GigabitEthernet0/0.12
Internet 10.14.14.1 - 0025.849d.d500 ARPA GigabitEthernet0/0.14
Internet 10.14.14.2 0 Incomplete ARPA
======

interface GigabitEthernet0/0
no ip address
duplex auto
speed auto
interface GigabitEthernet0/0.12
encapsulation dot1Q 12
ip address 10.12.12.1 255.255.255.224
interface GigabitEthernet0/0.14
encapsulation dot1Q 14
ip address 10.14.14.1 255.255.255.224
===========
SW1
interface GigabitEthernet0/1
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 12,14
switchport mode trunk
spanning-tree portfast trunk
interface GigabitEthernet0/5
switchport access vlan 14
switchport mode access
spanning-tree portfast
===================
R2#sh arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.14.14.1 0 Incomplete ARPA
Internet 10.14.14.2 - 001e.f725.ba00 ARPA GigabitEthernet0/0
==
interface GigabitEthernet0/0
ip address 10.14.14.2 255.255.255.224
duplex auto
speed auto

 

[freegeeks]

check your cabling, I had weird issues where everything looked allright and in the end it was a faulty UTP cable. Interfaces were up, config was right, .... just like you have. You can also make a static arp entry and try to ping the other side. if you are not allowed to run a debug, you can setup a span port on the switch and debug that way

 

[spidey07]

That means you've got bad L2 vlan mismatch. Cables are always a possibility. Check spanning-tree for the switch to ensure it's forwarding on boths ports for the correct vlan.

I'm not sure if this is a trick question - you've got a native VLAN/tagging problem from what I can quickly glean. This looks like CCIE lab *cough*.

show int trunk
show int Gx/x switchport
show spanning-tree int Gx/x

interface GigabitEthernet0/0 ***router will tag vlan 12 and 14, native vlan is something else***
no ip address
duplex auto
speed auto
interface GigabitEthernet0/0.12
encapsulation dot1Q 12
ip address 10.12.12.1 255.255.255.224
interface GigabitEthernet0/0.14
encapsulation dot1Q 14
ip address 10.14.14.1 255.255.255.224
===========
SW1
interface GigabitEthernet0/1
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 12,14
switchport mode trunk ***no native vlan specified, that means it's VLAN 1 and vlan 12 and 14 are tagged***
spanning-tree portfast trunk
interface GigabitEthernet0/5
switchport access vlan 14
switchport mode access ***not trunking, access mode, if other end is trunking or tagging this assumes native vlan is 14***
spanning-tree portfast
===================
R2#sh arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.14.14.1 0 Incomplete ARPA
Internet 10.14.14.2 - 001e.f725.ba00 ARPA GigabitEthernet0/0
==
interface GigabitEthernet0/0 ***not tagging/trunking, ensure switchport is access vlan 14 or trunk native vlan 14***
ip address 10.14.14.2 255.255.255.224
duplex auto
speed auto

 

[freegeeks]

That means you've got bad L2. Cables are always a possibility. Check spanning-tree for the switch to ensure it's forwarding on boths ports for the correct vlan.

I'm not sure if this is a trick question - you've got a native VLAN/tagging problem from what I can quickly glean.

where? I had a quick look at the config, one router is running a trunk, the other one is connected on an access port, vlan 14. vlan 14 is allowed on the trunk

from what I see, his config looks ok. Didn't do a lot of cisco anymore the last year but it looks fine to me

EDIT: saw spideys updated post, interesting

 

[Cooky]

From what you guys can see though, vlan 14 is used for R1 and R2 to communicate w/ each other, through the SAME switch.

Switch was not in VTP transparent mode, so it's not like vlan 14 was configured as isolated private vlan so that the two hosts can't talk to each other.

Is there any special configuration on the switch or router that would prevent ARP from working?

BTW, I'm not just asking for fun...I'm dying to find out what the answer might be.

 

[spidey07]

No. Nothing special going on here. Map it out on paper on what is tagged and how. You'll figure it out. Look over my post again and think it through.

Think about how a frame leaves one device and how it will received by the other. What vlan will the sender and receiver believe it to be.

 

[drebo]

Did you make sure to add VLANs 12 and 14 to the switch's VLAN database?

 

[sactwnguy]

This is a pretty easy set up. Native vlan does not matter as long as the frames are tagged. You can have mismatched native vlans on a trunk and it will still work properly for tagged frames, cdp will grumble about it constantly though. If you do sh int trunk on the switch what do you see? Also this may seem like a stupid question but did you no shut the sub-interfaces on the router? Try doing a sh ip route on the both routers, do you see a interface route for 10.14.14.0/27?

 

[Cooky]

I made sure both vlan's 12 & 14 are in the vlan DB, and show up as "active", and not suspended or shutdown.
Also ensured all interfaces, sub-interfaces were up.

I no longer have access to that network, but thank you all for the responses.
Will post if I ever figure out what the issue was.

 

[ScottMac]

Did you ever do a trace? i.e., Hook up Wireshark and see what the traffic actually looked like?

Also, did you verify that both ends were 802.1q? Lots of older Cisco stuff/firmware defaults to the older Cisco protocol.

 

[Cooky]

thanks Scott.

Time and resources were rather limited, so no, I didn't have a chance to do SPAN and see what it looks like in Wireshark.
Although I did verify both ends (router trunk interface, and switch trunk interface) were doing .1q.
The 2800 ISR's don't support ISL, and .1q was hardcoded on the 3560 side.