Cisco router processor keeps hitting 95% - 99%

[err]

Hi,

I am managing a cisco router that keeps hitting 95 - 99% in processor usage. I am having a latency problems and there is a huge package loss problem.

The cisco router is 2621, 48 MB RAM and has 2 WIC with T1 attatched to it. It is currently running 3 Class C network.

I am suspecting several things:

1. I am getting DDOS and it is killing my router and my latency. However the strange thing is my router bandwidth is not maxed out. (It is only currently using 25% of the bandwidth which is normal).
2. My router is maxed out because it is running 2 T1 and 3 Class C network ?

I am almost out of ideas on this problem. Can anyone please help give out any ideas ?

Thanks !

eRr

 

[Tallgeese]

Originally posted by: err
I am managing a cisco router that keeps hitting 95 - 99% in processor usage.

This might help you troubleshoot the problem as Cisco would.

Also, keep in mind that full logging and debug option can bring even the largest routers to their (proverbial) knees.

 

[ScottMac]

If you do a "show processes" it should give you a list of processes, and how much CPU each is using.

Also check "Show Buffers" and look for drops, misses, etc

"Show interface" will give you stats per interface.

Which IOS version are you running?

Run 'em & post 'em (edited as necessary).

-Scott

 

[err]

Here's the CPU Utilization.

CPU utilization for five seconds: 99%/40%; one minute: 97%; five minutes: 84%
PID QTy PC Runtime (ms) Invoked uSecs Stacks TTY Process
1 Csp 8028C15C 2616 9638 271 2648/3000 0 Load Meter
2 M* 0 172 121 142110064/12000 66 Virtual Exec
3 Msa 8046B088 280 3238 86 5692/6000 0 BGP Open
4 Lst 8027932C 76496 5658 13519 5740/6000 0 Check heaps
5 Cwe 80270714 0 1 0 5600/6000 0 Chunk Manager
6 Cwe 8027EA3C 22208 15054 1475 5640/6000 0 Pool Manager
7 Mst 801FE494 0 2 0 5604/6000 0 Timers
8 Mwe 8001B084 4 4 1000 5620/6000 0 Serial Backgroun
9 Msi 802BF3B8 160 1609 99 5608/6000 0 Environmental mo
10 Lrd 802D1200 34172 57786 591 5304/6000 0 ARP Input
11 Mwe 80440068 0 4 0 5612/6000 0 DDR Timers
12 Mwe 804568CC 0 2 011616/12000 0 Dialer event
13 Lwe 805A4998 4 2 2000 5652/6000 0 Entity MIB API
14 Mwe 8001F43C 0 1 0 5652/6000 0 SERIAL A'detect
15 Cwe 802841C0 0 1 0 5644/6000 0 Critical Bkgnd
16 Mwe 8024F49C 7020 13021 53910596/12000 0 Net Background
17 Lwe 801F3A08 36 117 30711468/12000 0 Logger
18 Msp 80211CA0 10444 47091 221 5536/6000 0 TTY Background
19 Msp 8024EA8C 5660 47109 120 5832/6000 0 Per-Second Jobs
20 Mwe 800959BC 0 2 0 5560/6000 0 Hawkeye Backgrou
21 Msi 802C95EC 3960 47095 84 5640/6000 0 Partition Check
22 Hwe 8024ECC4 2008 9116 220 5632/6000 0 Net Input
23 Csp 80256180 3324 9639 344 5636/6000 0 Compute load avg
24 Msp 8024EADC 24376 806 30243 5624/6000 0 Per-minute Jobs
25 Mwe 800698A0 0 4 0 5620/6000 0 Service-module a
26 Mrd 802F91E0 35590012 11800152 301610776/12000 0 IP Input
27 Mwe 80397258 4400 7939 554 5452/6000 0 CDP Protocol
28 Hwe 8040A2B4 0 1 0 5740/6000 0 Asy FS Helper
29 Mwe 803E4BF0 0 1 0 5624/6000 0 PPP IP Add Route
30 Lwe 806580B8 0 1 0 5328/6000 0 X.25 Encaps Mana
31 Hwe 80B68DDC 0 1 0 5664/6000 0 MPC Router Proce
32 Mwe 80357538 3808 994 3830 8320/9000 0 IP Background
33 Mwe 801B9D48 652 808 806 5612/6000 0 Adj Manager
34 Mst 802E24AC 12 32 37511476/12000 0 TCP Timer
35 Lwe 802E7254 8 3 266611344/12000 0 TCP Protocols
36 Lwe 803312D0 0 1 0 5644/6000 0 Probe Input
37 Mwe 8033252C 4 1 4000 5652/6000 0 RARP Input
38 Mwe 80340DD0 0 1 0 5792/6000 0 HTTP Timer
39 Hwe 8034B9C0 0 1 0 5780/6000 0 Socket Timers
40 Mwe 802D760C 4 2 2000 5480/6000 0 DHCPD Receive
41 Lsi 8038F6B0 60 804 74 5772/6000 0 IP Cache Ager
42 Hwe 80662FE8 0 1 0 5616/6000 0 PAD InCall
43 Mwe 8062C950 0 2 011608/12000 0 X.25 Background
44 Mwe 8072468C 0 1 0 5796/6000 0 Inspect Timer
45 Mwe 8076356C 0 1 0 5792/6000 0 Authentication P
46 Mwe 807681B4 0 1 0 5796/6000 0 IDS Timer
47 Hwe 809D08F0 0 2 011624/12000 0 ILMI Input
48 Mwe 805A002C 0 1 0 5764/6000 0 SNMP Timers
49 Mwe 809C5988 0 2 0 5636/6000 0 ILMI Request
50 Mwe 809C5A14 0 2 0 5632/6000 0 ILMI Response
51 Mwe 809D0E9C 0 1 0 5764/6000 0 ILMI Timer Proce
52 Mwe 809DA3A4 48 2 2400011592/12000 0 ATM PVC Discover
53 Hwe 809FD2A8 4 2 2000 5608/6000 0 SSCOP Input
54 Mwe 809FD568 0 2 0 5612/6000 0 SSCOP Output
55 Mst 809FD9BC 136 807 168 5616/6000 0 SSCOP Timer
56 Mwe 80110DC4 0 1 023788/24000 0 ISDN Timer
57 Mwe 802A3368 0 1 0 5768/6000 0 Time Range Proce
59 Mwe 80999FEC 4 2 2000 5576/6000 0 CCVPM_HDSPRM
60 Mwe 8098C998 4 1 4000 5620/6000 0 CCVPM_HTSP
61 Mwe 809BE5FC 0 1 0 5588/6000 0 CCSWVOICE
62 Mwe 80B99288 0 1 011652/12000 0 Encrypt Proc
63 Mwe 80B99EE0 0 5 0 7616/8000 0 Key Proc
64 Mwe 80BA3088 0 1 0 2636/3000 0 Crypto Support
65 Mwe 80B7DED4 22624 48336 46811608/12000 0 Crypto SM
66 Mwe 80BEA8EC 8 3 2666 7472/8000 0 Crypto CA
67 Mwe 80BCEE5C 20 163 12211564/12000 0 Crypto IKMP
68 Mwe 80BC62C0 952 5373 177 5740/6000 0 IPSEC key engine
69 Mwe 80BC6A24 0 1 0 5716/6000 0 IPSEC manual key
70 Mwe 803A11FC 0 1 0 5772/6000 0 ISDNMIB Backgrou
71 Mwe 803AB860 0 1 0 5768/6000 0 CallMIB Backgrou
72 Lwe 803D66C0 0 1 011660/12000 0 SNMP ConfCopyPro
73 Mwe 80714A6C 0 1 0 5664/6000 0 Syslog Traps
74 Mwe 809C41D0 8 1 800023636/24000 0 CCSWVOFR
75 Lwe 80328F60 4560 650 701511132/12000 0 IP SNMP
76 Mwe 805A0344 4 2 200011432/12000 0 SNMP Traps
77 Mwe 806BABE8 0 2 0 5636/6000 0 IP NAT Ager
79 Lwe 802E5FD0 0 1 0 5792/6000 0 TCP Listener
80 Mwe 805F84D4 7536 48430 155 4964/6000 0 NTP
82 Mwe 8034E474 72 403 178 5760/6000 0 DHCPD Timer
83 Msi 803554B0 936 13636 68 5036/6000 0 DHCPD Database
84 MS 80474D00 5908 94262 62 8348/9000 0 BGP Router
85 ME 8046B5A8 0 1 0 5804/6000 0 BGP I/O
86 Lsi 80470080 24128 815 29604 4160/6000 0 BGP Scanner

Look at IP input. what does that mean ?



Any input ?

Thanks !

eRr

 

[ScottMac]

Generally, your stats seems to be in order / proportion with what I can bring up on some 2600s I have access to from home.

How many people are using the VPN?
Is it possible that you have some people "camped out" on the VPN (Up, but the computer is idle) ... or worse, running a "pinger" to keep 'em up, even when they're not using them ... to save login / call time ?

Crypto processes take ALOT of CPU time .... as in a SH!Tload.

I'm thinking the IP Input counters are high because of Access-list processing. If your access lists are long or complex, perhaps they should be reviewed to see if they can be economized somehow (rearrange the logic a little).

Same thing for your BGP filters.

The ATM interface doesn't look like it's active. If you have any interfaces that aren't doing anything, pull 'em if you can. Their presence will cause the router will assign processes/resources unnecessarily (looking to see if they're active, assign buffers, etc).

That's it for me on a quick tip-toe through the processes ... I might be off a little.

How long was the router up when you took that snapshot?
Is this a common (normal) traffic load?

There's some time being spent on the SNMP processes, is this router being managed remotely (like an OpenView /CiscoWorks/etc management platform)?

Can you get us a "show interface" and "Show buffers" - that would show how well the interfaces and memory resources are handling the traffic.

FWIW

Scott

 

[err]

This is the output of sho buffers

Buffer elements:
498 in free list (500 max allowed)
128354514 hits, 0 misses, 0 created

Public buffer pools:
Small buffers, 104 bytes (total 155, permanent 50):
148 in free list (20 min, 150 max allowed)
74636164 hits, 362116 misses, 86113 trims, 86218 created
21903 failures (0 no memory)
Middle buffers, 600 bytes (total 150, permanent 25):
145 in free list (10 min, 150 max allowed)
21119552 hits, 187 misses, 25 trims, 150 created
1 failures (0 no memory)
Big buffers, 1524 bytes (total 62, permanent 50):
62 in free list (5 min, 150 max allowed)
2009774 hits, 19 misses, 3 trims, 15 created
0 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 10, permanent 10):
10 in free list (0 min, 100 max allowed)
1088 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Large buffers, 5024 bytes (total 0, permanent 0):
0 in free list (0 min, 10 max allowed)
0 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Huge buffers, 18024 bytes (total 0, permanent 0):
0 in free list (0 min, 4 max allowed)
0 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)

Interface buffer pools:
CD2430 I/O buffers, 1524 bytes (total 0, permanent 0):
0 in free list (0 min, 0 max allowed)
0 hits, 0 fallbacks

Header pools:
Header buffers, 0 bytes (total 137, permanent 128):
9 in free list (10 min, 512 max allowed)
125 hits, 3 misses, 0 trims, 9 created
0 failures (0 no memory)
128 max cache size, 128 in cache

Particle Clones:
1024 clones, 0 hits, 0 misses

Public particle pools:
F/S buffers, 256 bytes (total 384, permanent 384):
128 in free list (128 min, 1024 max allowed)
256 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
256 max cache size, 256 in cache
Normal buffers, 1548 bytes (total 512, permanent 512):
320 in free list (128 min, 1024 max allowed)
328 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
128 max cache size, 128 in cache

Private particle pools:
FastEthernet0/0 buffers, 1548 bytes (total 192, permanent 192):
0 in free list (0 min, 192 max allowed)
192 hits, 0 fallbacks
192 max cache size, 128 in cache
FastEthernet0/1 buffers, 1548 bytes (total 192, permanent 192):
0 in free list (0 min, 192 max allowed)
192 hits, 0 fallbacks
192 max cache size, 192 in cache
Serial0/0 buffers, 1548 bytes (total 32, permanent 32):
0 in free list (0 min, 32 max allowed)
32 hits, 0 fallbacks
32 max cache size, 16 in cache
Serial0/1 buffers, 1548 bytes (total 32, permanent 32):
0 in free list (0 min, 32 max allowed)
32 hits, 0 fallbacks
32 max cache size, 16 in cache

 

[err]

FastEthernet0/0 is up, line protocol is up
Hardware is AmdFE, address is 0002.4b76.0a20 (bia 0002.4b76.0a20)
Description: LAN Interface
Internet address is 1.1.1.1/24
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Queueing strategy: fifo
Output queue 0/40, 0 drops; input queue 3/75, 38717837 drops
5 minute input rate 478000 bits/sec, 317 packets/sec
5 minute output rate 220000 bits/sec, 221 packets/sec
29832804 packets input, 934567450 bytes
Received 123342 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast
0 input packets with dribble condition detected
34617883 packets output, 3574815395 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 98992 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
Serial0/0 is up, line protocol is up
Hardware is PQUICC with Fractional T1 CSU/DSU
Description: T1 - A
Internet address is 1.1.1.1/30
MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
reliability 255/255, txload 37/255, rxload 20/255
Encapsulation HDLC, loopback not set
Keepalive set (10 sec)
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Queueing strategy: fifo
Output queue 0/40, 56 drops; input queue 1/75, 2987440 drops
5 minute input rate 125000 bits/sec, 127 packets/sec
5 minute output rate 228000 bits/sec, 158 packets/sec
33059042 packets input, 2805900252 bytes, 0 no buffer
Received 7766 broadcasts, 0 runts, 0 giants, 0 throttles
1749 input errors, 630 CRC, 1098 frame, 0 overrun, 0 ignored, 21 abort
14874919 packets output, 2448053909 bytes, 0 underruns
0 output errors, 0 collisions, 109 interface resets
0 output buffer failures, 0 output buffers swapped out
1 carrier transitions
DCD=up DSR=up DTR=up RTS=up CTS=up
FastEthernet0/1 is administratively down, line protocol is down
Hardware is AmdFE, address is 0002.4b76.0a21 (bia 0002.4b76.0a21)
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Half-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters never
Queueing strategy: fifo
Output queue 0/40, 0 drops; input queue 0/75, 0 drops
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast
0 input packets with dribble condition detected
0 packets output, 0 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
0 output buffer failures, 0 output buffers swapped out
Serial0/1 is up, line protocol is up
Hardware is PQUICC with Fractional T1 CSU/DSU
Description: T1 - B
Internet address is 1.1.1.1/30
MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
reliability 255/255, txload 37/255, rxload 19/255
Encapsulation HDLC, loopback not set
Keepalive set (10 sec)
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Queueing strategy: fifo
Output queue 0/40, 38 drops; input queue 2/75, 5249778 drops
5 minute input rate 120000 bits/sec, 129 packets/sec
5 minute output rate 224000 bits/sec, 160 packets/sec
34541117 packets input, 2899149922 bytes, 0 no buffer
Received 7527 broadcasts, 0 runts, 0 giants, 0 throttles
1463 input errors, 527 CRC, 920 frame, 0 overrun, 0 ignored, 16 abort
14754098 packets output, 2435133039 bytes, 0 underruns
0 output errors, 0 collisions, 189 interface resets
0 output buffer failures, 0 output buffers swapped out
1 carrier transitions
DCD=up DSR=up DTR=up RTS=up CTS=up
0 output errors, 0 collisions, 189 interface resets
0 output buffer failures, 0 output buffers swapped out
1 carrier transitions
DCD=up DSR=up DTR=up RTS=up CTS=up

Loopback0 is up, line protocol is up
Hardware is Loopback
Description: Peering point for BGP sessions
Internet address is 1.1.1.1/32
MTU 1514 bytes, BW 8000000 Kbit, DLY 5000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation LOOPBACK, loopback not set
Last input 22:39:44, output never, output hang never
Last clearing of "show interface" counters never
Queueing strategy: fifo
Output queue 0/0, 0 drops; input queue 0/75, 0 drops
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
5 packets output, 0 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 output buffer failures, 0 output buffers swapped out

 

[err]

Scott, thanks for your input so far.

No one is using VPN via the router., but our router is constantly pinged for monitoring purposes. I believe some people ping their server every 2 minutes or so to check uptime.

The router was just booted. It was up for 13 hours. Right now, the processes on the router isn't so bad and I am seeing good response time:

CPU utilization for five seconds: 57%/25%; one minute: 43%; five minutes: 47%

Yes we're using snmp to monitor mrtg traffic

As far as ATM, how would you disable that ?

Also, do you think the IP input is normal for this kind of environment? Unfortunately we do not monitor the router in the past, so we don't really know how much it has been abused.

Thanks again for the input scott, email me and let me know when you're in town. I'll buy you a beer

eRr

 

[spidey07]

Output queue 0/40, 0 drops; input queue 3/75, 38717837 drops

something is very wrong, that is a tremendous amout of drops on an interface...38 million. Also looks at your small buffer pools, lots of misses and failures.

how about post output of "show proc cpu" as it will give a percentage output. check if any debuging is going on with "show debug", or better yet "undebug all". Check "show lines" to make sure there are lots of terminal sessions.

SNMP isn't that much of a hog, but you should set your polling to every five seconds.

show proc cpu will atleast let us know what is taking up the most. Maybe even post a show run to see if we have any glaring problems. Post show version as well.

Last but not least you can call cisco.

troubleshooting input/output drops

with only two t1s I can't see this router running out of processor. but if there is something else going on (like trunking on the Fe interface or other lan routing then for sure you can run into problems.)

 

[Tallgeese]

Originally posted by: spidey07
Maybe even post a show run to see if we have any glaring problems.

Just a general reminder for folks tuning in:

If you post configurations, please please please make sure and remove any passwords or other sensitive info that you don't want anyone else to see.
KIM, that sensitive info may even include ip addresses in use (both external and internal)

Carry on,
Tallgeese

 

[Santa]

Type "undebug all" or "u all"

Then see if processor use time drops.

Another note about show run displaying.. even if it looks like the password is encrypted completely remove the line because it can be decrypted and then they will be able to get into your router if it is publically availble.

I second a showing of the "show proc cpu" but as I have seen in my past troubleshooting some things do not show up on this list such as Serial Mainframe Controllers which are a big hog on resources.

Check all the interfaces and do a shutdown on the ones you know you aren't using along with shutting off all debugging and that should at least eliminate those two memory hogs out of the equation.

 

[err]

Hey all, first of all I would like to thank you guys for your tremendous conrtibution.

The router is back to its normal level for now. Processor usage has dropped to 20-25% level.

I did NOT do anything on the router.

This is really strange and I would like to prevent this from happening again in the future.

I have turned all debugging and all logging options. I have also shutdown the interface that I am not using.

Anyway. I'll post the sho proc cpu below. I think the result will be skewed as the router is running on 20-25% utilization now.

Thanks all again !!
Btw, any ideas why the router is jammin' ? Any sign of possible attacks ?

eRr

 

[err]

sho proc cpu

CPU utilization for five seconds: 30%/11%; one minute: 34%; five minutes: 44%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 4648 22797 203 0.00% 0.00% 0.00% 0 Load Meter
2 356 119 2991 1.80% 0.30% 0.08% 66 Virtual Exec
3 572 7625 75 0.00% 0.00% 0.00% 0 BGP Open
4 172820 13362 12933 0.00% 0.11% 0.11% 0 Check heaps
5 0 1 0 0.00% 0.00% 0.00% 0 Chunk Manager
6 34668 24358 1423 0.00% 0.00% 0.00% 0 Pool Manager
7 0 2 0 0.00% 0.00% 0.00% 0 Timers
8 4 7 571 0.00% 0.00% 0.00% 0 Serial Backgroun
9 364 3803 95 0.00% 0.00% 0.00% 0 Environmental mo
10 70696 126883 557 0.00% 0.06% 0.07% 0 ARP Input
11 0 7 0 0.00% 0.00% 0.00% 0 DDR Timers
12 0 2 0 0.00% 0.00% 0.00% 0 Dialer event
13 4 2 2000 0.00% 0.00% 0.00% 0 Entity MIB API
14 0 1 0 0.00% 0.00% 0.00% 0 SERIAL A'detect
15 0 1 0 0.00% 0.00% 0.00% 0 Critical Bkgnd
16 12916 25824 500 0.00% 0.00% 0.00% 0 Net Background
17 564 503 1121 0.00% 0.00% 0.00% 0 Logger
18 19212 112040 171 0.00% 0.00% 0.00% 0 TTY Background
19 10204 112058 91 0.00% 0.00% 0.00% 0 Per-Second Jobs
20 0 2 0 0.00% 0.00% 0.00% 0 Hawkeye Backgrou
21 7120 112044 63 0.00% 0.00% 0.00% 0 Partition Check
22 4436 21726 204 0.00% 0.00% 0.00% 0 Net Input
23 6152 22797 269 0.00% 0.00% 0.00% 0 Compute load avg
24 55548 1903 29189 0.00% 0.03% 0.00% 0 Per-minute Jobs
25 0 4 0 0.00% 0.00% 0.00% 0 Service-module a
26 67986416 29097132 2336 14.23% 17.25% 24.69% 0 IP Input
27 9560 18782 508 0.00% 0.00% 0.00% 0 CDP Protocol
28 0 1 0 0.00% 0.00% 0.00% 0 Asy FS Helper
29 0 1 0 0.00% 0.00% 0.00% 0 PPP IP Add Route
30 0 1 0 0.00% 0.00% 0.00% 0 X.25 Encaps Mana
31 0 1 0 0.00% 0.00% 0.00% 0 MPC Router Proce
32 8972 2291 3916 0.00% 0.00% 0.00% 0 IP Background
33 1768 1907 927 0.00% 0.00% 0.00% 0 Adj Manager
34 24 105 228 0.00% 0.00% 0.00% 0 TCP Timer
35 8 6 1333 0.00% 0.00% 0.00% 0 TCP Protocols
36 0 1 0 0.00% 0.00% 0.00% 0 Probe Input
37 4 1 4000 0.00% 0.00% 0.00% 0 RARP Input
38 0 1 0 0.00% 0.00% 0.00% 0 HTTP Timer
39 0 1 0 0.00% 0.00% 0.00% 0 Socket Timers
40 4 2 2000 0.00% 0.00% 0.00% 0 DHCPD Receive
41 136 1901 71 0.00% 0.00% 0.00% 0 IP Cache Ager
42 0 1 0 0.00% 0.00% 0.00% 0 PAD InCall
43 0 2 0 0.00% 0.00% 0.00% 0 X.25 Background
44 0 1 0 0.00% 0.00% 0.00% 0 Inspect Timer
45 0 1 0 0.00% 0.00% 0.00% 0 Authentication P
46 0 1 0 0.00% 0.00% 0.00% 0 IDS Timer
47 0 2 0 0.00% 0.00% 0.00% 0 ILMI Input
48 0 1 0 0.00% 0.00% 0.00% 0 SNMP Timers
49 0 2 0 0.00% 0.00% 0.00% 0 ILMI Request
50 0 2 0 0.00% 0.00% 0.00% 0 ILMI Response
51 0 1 0 0.00% 0.00% 0.00% 0 ILMI Timer Proce
52 48 2 24000 0.00% 0.00% 0.00% 0 ATM PVC Discover
53 4 2 2000 0.00% 0.00% 0.00% 0 SSCOP Input
54 0 2 0 0.00% 0.00% 0.00% 0 SSCOP Output
55 296 1904 155 0.00% 0.00% 0.00% 0 SSCOP Timer
56 0 1 0 0.00% 0.00% 0.00% 0 ISDN Timer
57 0 1 0 0.00% 0.00% 0.00% 0 Time Range Proce
59 4 2 2000 0.00% 0.00% 0.00% 0 CCVPM_HDSPRM
60 4 1 4000 0.00% 0.00% 0.00% 0 CCVPM_HTSP
61 0 1 0 0.00% 0.00% 0.00% 0 CCSWVOICE
62 0 1 0 0.00% 0.00% 0.00% 0 Encrypt Proc
63 0 5 0 0.00% 0.00% 0.00% 0 Key Proc
64 0 1 0 0.00% 0.00% 0.00% 0 Crypto Support
65 41136 114218 360 0.00% 0.00% 0.00% 0 Crypto SM
66 8 3 2666 0.00% 0.00% 0.00% 0 Crypto CA
67 48 382 125 0.00% 0.00% 0.00% 0 Crypto IKMP
68 1612 12695 126 0.00% 0.00% 0.00% 0 IPSEC key engine
69 0 1 0 0.00% 0.00% 0.00% 0 IPSEC manual key
70 0 1 0 0.00% 0.00% 0.00% 0 ISDNMIB Backgrou
71 0 1 0 0.00% 0.00% 0.00% 0 CallMIB Backgrou
72 0 1 0 0.00% 0.00% 0.00% 0 SNMP ConfCopyPro
73 0 1 0 0.00% 0.00% 0.00% 0 Syslog Traps
74 8 1 8000 0.00% 0.00% 0.00% 0 CCSWVOFR
75 10136 1527 6637 0.00% 0.00% 0.00% 0 IP SNMP
76 4 2 2000 0.00% 0.00% 0.00% 0 SNMP Traps
77 0 2 0 0.00% 0.00% 0.00% 0 IP NAT Ager
79 0 1 0 0.00% 0.00% 0.00% 0 TCP Listener
80 13940 114386 121 0.00% 0.00% 0.00% 0 NTP
82 136 952 142 0.00% 0.00% 0.00% 0 DHCPD Timer
83 2192 32234 68 0.00% 0.00% 0.00% 0 DHCPD Database
84 10836 224111 48 0.00% 0.00% 0.00% 0 BGP Router
85 0 1 0 0.00% 0.00% 0.00% 0 BGP I/O
86 54808 1919 28560 0.00% 0.03% 0.00% 0 BGP Scanner

 

[Santa]

Are you doing any sort of L2TP tunneling?

Check this post out to see if it helps you any..

L2TP high CPU utilization

Make sure you have "no ip directed-broadcast" on all your interfaces in use with IP addresses (BIG MUST)

You may be involved in a Smurfing of some poor individual if this is not set.

You may also want to tone down your SNMP polling or turn it off altogether to see if that changes your results.

One last thing before work..

Display "show ip cache flow | include 0050"

The 0050 is a hex (0x50) value for 80 as in port 80.

If you see alot of the same source address but many many different destination address you may have an infected or multiple infected Code Red devices on your internal network. Track em down and eliminate them.

g'luck..

 

[err]

Just a quick update, I think I know what is happening to our network.

We've been infected with the Apache_mod_ssl virus that allows a network host to be a DDOS station. This scary virus exactly matches the problem we're having on our network.

More info about this virus can be found here:

http://securityresponse.symantec.com/avcenter/security/Content/2002.09.13.html

We'll be very busy for a while now as we need to patch basically all of our linux servers.

Well Thanks for all the input so far appreciate everybody's effort.

Cheers

eRr

 

[Tallgeese]

Originally posted by: err
More info about this virus can be found here:
http://securityresponse.symantec.com/avcenter/security/Content/2002.09.13.html

Nasty little bastard you caught there...good luck with the cleanup!

 

[Mucman]

Dayum, that's a scary bug... I saw the cert.org notice and it said apache + mod_ssl in linux servers... So this doesn't affect unix machines or any of the BSD's?

 

[n0cmonkey]

Originally posted by: Mucman
Dayum, that's a scary bug... I saw the cert.org notice and it said apache + mod_ssl in linux servers... So this doesn't affect unix machines or any of the BSD's?

As with all buffer overflows the exploit will have to be modified for each operating system, distribution, and hardware paltform. The worm out there is linux based and will remain that way for now. Dont let that make you feel all warm and cuddly.

Not to be a dick or anything, but this is stupid. This right here is what we have been complaining about for over a year when it comes to IIS microsoft MCSE admin wannabes. The exploit Code Red v1 used was patched several months I believe before the worm was released. This exploit was patched in late July. What have your webserver admin been doing for the past month and a half?! Hopefully he wasnt cashing a paycheck because by the looks of things he surely wasnt working. I realize that what Im saying is extremely mean and possibly hurtful to some people, but dammit Im tired of seeing month+ old vulnerabilities turn into a major disaster and bad press for admins of any type!

Be vigilant.

EDIT: Just wanted to say, if any of you do need help with stuff like this, feel free to pm, start a thread, and whatnot. I will help as best I can.

 

[Tallgeese]

Originally posted by: n0cmonkey
This right here is what we have been complaining about for over a year when it comes to IIS microsoft MCSE admin wannabes.

So this time it is the Apache Linux admin wannabes who are the problem, eh? *ducks*

Seriously n0c, I'm with ya on this all the way. When you get right down to it, the platform isn't the issue. Doing the necessary work for the given platform is. And that goes for a HECKUVA lot more than security issues (altho the consequences tend to show up much faster when it comes to security).

 

[Mucman]

n0cmonkey, you can't help but be a dick, but that's why we love you

Looks like I have an OpenBSD machine to patch then...

btw, Tallgeese will no longer be posting anymore.... if he changes his post count I will have to kill him
On a side note, how come he isn't Elite yet?

 

[ScottMac]

I dunno, the "Tallgeese and n0cmonkey for Elite" threads have been resurrected twice (on the FI board) since the original round, no action.

Maybe we gotta send the mods some flowers or sumthin' ....

I'm all for it

Scott

 

[spidey07]

Well, I don't know. monkey and TG's rants are not very "elite" like. You have to be more subtle when you dumass somebody. phrases like "It appears you don't understand", "I and 6 others have tried to inform you, I fear our efforts were in vain"

stuff like that.



J/K

 

[Tallgeese]

Originally posted by: spidey07
You have to be more subtle when you dumass somebody. phrases like "It appears you don't understand", "I and 6 others have tried to inform you, I fear our efforts were in vain"

ROFLMAO!

BTW: And with his humor, spidey coldly sentences me to death at the hands of Mucman, since my post count is no longer identical to the title of a Rush album.

 

[Tallgeese]

Originally posted by: ScottMac
"n0cmonkey for Elite" threads have been resurrected twice (on the FI board) since the original round, no action.

No action?
Scott...you might wanna check n0c's ranking again. He made it (and deservedly so, I might add).

I still say the only fitting rank for me is "Incurable threadjacker EXTRAORDINAIRE."
Mainly cuz it drives all the ladies WILD!

 

[spidey07]

And just when is RUSH going to relase 2112 on SACD?

<---still waiting.