Cisco 3600 at 100% CPU! 800ms ping time! *update new question*

[spidey07]

<< Spidey r u CCNP or CCDP? or which cert. if u have one? or how many years in this? >>


both, about 8 years - ccie lab scheduled for oct. I WILL fail.

sorry to be so short but this guy is in trouble. no firewall with a rampant virus/worm on the loose and a core router that is goin four-paws on him.

 

[Damaged]

heh. see, I'm thinking:

sh ip arp
sh proc cpu

ip arp shows lots of lovely arp request unresolved.

sh proc cpu shows lots of IP FLOW.

Nimda BS. Much the same affect on the router as CodeRed had. arp table is maxed, chewing up beacoup mem and some cpu just trying to resolve. And every attempt creates a new a IP FLOW (attempted connection, a new tcp flow). Eats up cpu and ram like nuts. Lovely sh!t.

 

[spidey07]

so D,

as far as the new one goes...look for sh!t load empty arps and emptly flows?

 

[Damaged]

John...you got it! It was no different with CodeRed though.

 

[spidey07]

clear ip arp *
clear ip flow *

debug ip arp
debug ip flow
debug ip packet

watch router go bye-bye

<edit> 'lest we not forget the most important command for troubleshooting "debug all". this will help you isolate your problem. thanks be to DAMAGED for reminding us of this holy grail of troubleshooting. that single command truly will allow you to see god.

😉

 

[Garion]

Wait a second.. Do you happen to have the HTTP server turned on? If so, you might just be getting nailed by the latest virus on the block. If you turn off the HTTP server you might be able to clean it up.

Worst case is an access list to deny * sending port 80 traffic to the IP of the router.

A few other comments:

If you need ram "right now", go to Crucial and get it fed-exed. It's dirt cheap compared to Cisco parts (1/10th the price) and works fine. I've put 3rd party memory in routers - Just don't tell Cisco!

Also, there's two kinds of memory - straight SIMMS for DRAM and card-based FLASH memory. The Flash card might lOOK like a PCMCIA card but it's really not. If you need more RAM you need a DRAM upgrade, not a flash upgrade. Putting it in PC terms, think of flash as your hard drive and DRAM as your RAM.

Guys, you might lay off of him JUST a bit on the firewall thing. If this is an edge router (like one running his T3/T1's) you can't have a firewall there yet. Also, a lot of ISP's really don't have firewalls for their core networks - After all, they ARE the Internet when you really look at it. Yeah, we're all for big companies that firewall the hell out of their networks (I work for a bank - We are the WORST!) but not all ISP's do.


- G

 

[Mucman]

Garion, you don't need to tell them ease off 🙂 btw about HTTP server being turned on, do you mean on the router? I told my boss about crucial and he was very impressed with their prices and service!

I am not responsible for our network, I am just tech support for our customers, but I would like to get involved. Let me explain our situation a little more. We are a web hosting company as well as an ethernet customer for companies in our building. We only have 7 web-servers, 1 SQL server, 2 name-servers, 2 DC's, Backup server, ~5 co-lo's and ~25 ethernet customers. We have two 10MBit feeds. In a month we will have an OC3 starting at 20-30MBit.

Oh well, live and learn 🙂

 

[Garion]

Yes, the routers have a built-in HTTP server. That's what got most of the whacked during the code red "invasion" a few weeks ago. Turn the HTTP server off and you'll get rid of most of the bogus traffic that's killing the router.

- G

 

[Mucman]

OK, HTTP server was off... I am only assuming this because we only connect to the router via telnet, or null-cable.

 

[Jonathan93]

<< We don't have a firewall! >>




It wouldn't be too hard to set that Router up as a firewall..... Check out the Access lists......

 

[Mucman]

Yeah, we decided to do that... Too bad we decided AFTER all of this mess began.

 

[dbcrossfire]

Alright, maybe someone already mentioned this... (i got sick of all that reading) We had a problem similar to this earlier this year with my 2501.

If the person that works on this router is a good little IT he would've backed up the ios image into a remote server, or onto a disk via console connect. Just wipe that 3600 clean and reinstall the IOS image, worked for us...