Okay, log files compressed, update STILL taking over 40 minutes.

[Russ]

In debug, these are the areas that seem to be the problem:

Resolving eMail to host and host to eMail participation

This one is the worst, and took 22 minutes to complete.

Entering Write_Bymail

During this one it also coughs up "Use of uninitialized, blah, blah, at Line 1852." This one is taking 5 minutes to complete.

Entering Write_Byhost

This took 10 minutes to complete and also coughs up the uninitialize error, at line 4426.

Any ideas? And, what exactly does "Use of uninitialized Value" mean? This appears several times during the running of the script.

Total time was 42 minutes, so the rest of the process is going through quickly.

Until I get this figured out, stats updates are cut back to once an hour.

Russ, NCNE

 

[Adul]

once, how will you ever manage. :Q

 

[Russ]

Bump.

Russ, NCNE

 

[Adul]

Russ, did you try runing the log compressor for only the previous days stats.

Also, your pproxy version is a very old 313. You should go up tp 3.19. It should help with avoiding assigning old WU that may sit in you buffers for who knows how long. Bphantom made mention of this in your other thread.
Also, if you want to speed up the stats run, you will need to run the log compressor on every log file you have on your server.

Has for the stats run Russ, may I suggest trying what I did.
I have 8 different set of stats on our pproxy.

daily for rc5/ogr <- runs every 10 minutes
Weekly for rc5/ogr <- runs every 15 minutes
yearly for rc5/ogr <-- runs every hour
one yearly with host not consolidated rc5/ogr <-- runs every hour

I run the strip cache every 2 hours
log compressor runs 0:40 UTC

Since yearly takes so long, I run it only once an hour. .
Daily only processes the stats for that day's stats, and it only takes a few seconds to run.
Weekly only processes stats for the last 7 days, and it takes less then a minute to run.

Every hour I have 8 proccess run at once. Every 30 minutes I have 4 process run at once, and 10 and 15 minutes only have 2.


Hope this helps somewhat.

 

[Adul]

Any news russ?

 

[Dougal]

Hi Adul,

I'm probably being really dim here but how do you split the stats runs up? I though they were all generated from a single script.

 

[Hawkeye_(BEL)]

Russ > you're running MDK7.2 on your server, right ?

If you haven't updated the kernel to a 2.4.x one, your harddrives are probably running in PIO mode if you haven't tweaked them. This GREATLY hinders performance if this is the case. Even if you checked 'HD optimalisations' during install, you can still get a lot more out of your HDs.

Test how fast your drives are by running 'hdparm -Tt' as root. If the values look really low, run 'hdparm -d1 -c1 -m16 -X66 -k1 /dev/hd?', where ? is the letter that refers to your IDE channel your HD is running on (a for IDE1-1, b for IDE1-2, etc). Browse through 'man hdparm' first though, to make sure you know what you're doing.

Just discard my message if you've already done this

 

[Adul]

great tip hawkeye. I was wondering if disk I/o has anything to do with it.


Dougal, what I mean by spliting it up, is only process certain amount of days from the logs files.

Daily will only process that days log file
weekly will process only the last 7 days log files

etc..

It is much faster to process files this way and I can get more frequent stats updates.

 

[Russ]

Hawkeye,

That wouldn't be it. Drives are SCSI.

Russ, NCNE

 

[Adul]

damn.... must be a way to help russ.

 

[Dougal]

Adul,

I think I see what you mean but I can't see a way to actually do that (being really dim here I know). Is it an option you change in the INI file or do I have to have multiple INI files or something? Sorry for asking so many questions but if I could streamline the stats on my PPRoxy a bit more it's more keys for TA.

Thanks,

Martin

 

[Hawkeye_(BEL)]

Russ, I still think it's a disk I/O problem of some sort.

You can check if your SCSI drives are running at their full potential by using Bonnie (http://www.textuality.com/bonnie/).

Make a new dir, put the download in that dir, untar it, and run 'bonnie -s 512'.

I hope this might give some clue at least...

 

[Adul]

I have mulitple ini files. basicly i seperated all mr runs into multiple directories and i am running multiple schedules.