Grepping the last 30 minutes of a log file...

[jtelep]

I need to know if anyone can assist me on how to grab the last (we'll just say "x" minutes) of a log file. How do you tell the grep command without specifying an exact window of time? (So relative instead of absolute.)

The log format looks like this:

03-05-2008 09:33:34 blah blah blah

Thanks,

Jon

 

[Nothinman]

I'm sure it's possible with regexp but frankly I'd probably just do "grep 03-05-2008 09:" and look through the whole hour.

 

[jtelep]

The problem is that I need to be able to do this every hour twice an hour (24/7) so I can't just look at one particular date or hour as it has to keep happening.

 

[Nothinman]

Well then you're looking at a more complicated script and not just grep. I'd probably use perl since I already know it pretty well, but straight bash scripting would probably work fine too.

 

[kamper]

Well for the full hour you could try this: grep "`date '+%d-%m-%Y %H'`"

That gives you a pattern including the current hour so it'd work better towards the end of the hour. If you script knows it's being run at, say, :00 and :30, for the first case you could subtract one from the hour:
HOUR=`date '+%H'`
HOUR=`expr $HOUR - 1`
if [ $HOUR -eq -1 ]; then HOUR=23; fi
if [ $HOUR -lt 10 ]; then HOUR=0$HOUR

Now your grep becomes: grep "`date '+%d-%m-%Y '`${HOUR}"

Then add [345] for the minutes: egrep "`date '+%d-%m-%Y '`${HOUR}:[345]"

And the one running at :30 can be: egrep "`date '+%d-%m-%Y '`${HOUR}:[012]"

A little further trickery would allow the script to run at any time. And I'm sure this can be done much more elegantly than I'm doing it

 

[kamper]

Ick, if you subtract one when the hour is already 0, you also have to move the day back by one, which means you might have know how many days were in the previous month, when the leap years are and whether or not it's January Better to use some kind of intelligent calendar handling library (perl?) or just run the script at :29 and :59

 

[kamper]

How about this:
NOW=`date '+%s'`

minutes_ago() {
DIFFERENCE=`expr $1 '*' 60`
SECONDS=`expr $NOW - $DIFFERENCE`
# cut chops off the least significant digit of the minutes
echo `date -r $SECONDS '+%d-%m-%Y %H:%M'` | cut -c1-15
}

egrep "`minutes_ago 10`|`minutes_ago 20`|`minutes_ago 30`" thelogfile

One caveat I can think of is that if you're running this right on a 10 minute barrier and the NOW line might run just before or just after, you could lose a chunk of 10 minutes. Also, if you ran it at, say :05, the entries between :00 and :05 wouldn't get picked up until next run. Here's what I get running it at 19:35: egrep "06-03-2008 19:2|06-03-2008 19:1|06-03-2008 19:0"

 

[Brazen]

Seriously, go over the lessons here: http://pine.fm/LearnToProgram/

and write your script in Ruby. Or learn Python or Perl, but this Learn To Program lessons made is seriously easy to learn Ruby, for me anyway.

I have a Ruby script that basically does this. I'm going to bed though so maybe I'll post it tomorrow.

 

[silverpig]

Originally posted by: Nothinman
I'm sure it's possible with regexp but frankly I'd probably just do "grep 03-05-2008 09:" and look through the whole hour.

set up a cron job to call a bash script. Have the bash script get the date, pull the hour and the minute off the time, then issue the following commands:

grep 03-05-2008 <hour>:<minute>
grep 03-05-2008 <hour>:<minute>+10
grep 03-05-2008 <hour>:<minute>+20

where <hour> is pulled from date and <minute> is pulled from date and is set to 00 if the minutes are less than 30, or set to 30 if the minutes are between 30 and 60.

edit: a little more formatting could be had to get the entire date off the date command

 

[Brazen]

well here is my script for subtracting a day (in ruby):

def yesterday(today)
ydate = today
ydate = ydate - ( 24 * 60 * 60 ) until ( ydate.day != today.day )
return ydate
end

the "until ( ydate.day != today.day )" is in case it is run at like 24:30 on a day that daylight savings takes effect, so subtracting 24 hours doesn't result in a Time object of the same day at 01:30 (maintaining the same hour and seconds was not necessary but could have been done by pulling from the orginal Time.now). I know this explanation probably doesn't make much sense if you don't know Ruby and how Time objects work, but I could explain better if the OP wanted to go this route (or read the link I provided earlier).

This can be modified to something like this:

def 30_minutes_ago(now = Time.now)
newtime = now
newtime = newtime - ( 30 * 60 )
return newtime
end

This would return a new time object that already takes into account your system's time zone, daylight savings changes, and rollbacks spanning across a day, month, year, etc. edit: and by the way, the "now = Time.now" means you can pass in a Time object or it will default to using Time.now (the time when the method is run).

My script then reads in the log files, uses regexp with a method to pull out the lines based on the date, and then email me the results. This is all done in Ruby and I could give the code snippets if interested. Even the email is done with Ruby's own smtp method and requires no configuration of smtp on the server, it's all done in Ruby in the script.