OT: Automated stats download proggy

[GeoffS]

I want to grab the contents of a few webpages automatically around midnight every night and save them to a file. What is the easiest way to do this, aside from staying up until midnight every night?

Thanks!
Geoff

 

[soni]

Sent you a PM

 

[ProviaFan]

Originally posted by: GeoffS
I want to grab the contents of a few webpages automatically around midnight every night and save them to a file. What is the easiest way to do this, aside from staying up until midnight every night?

Thanks!
Geoff

wget in a cron job?

 

[n0cmonkey]

Originally posted by: jliechty

Originally posted by: GeoffS
I want to grab the contents of a few webpages automatically around midnight every night and save them to a file. What is the easiest way to do this, aside from staying up until midnight every night?

Thanks!
Geoff

wget in a cron job?

Thats what I did for eccp109, well kind of. I wanted .txt files instead of .html files so I used lynx -dump instead of wget.

I could paste the simple scripts I used here if anyone wants them. Nothing spectacular, but it took me atleast 5 minutes to figure out the syntax for date

 

[GeoffS]

That stuff sounds like Linux I'm not well versed in Linux... yet

I understand that cron (*nix) = at (Windows), and the pages are pure text, so no complications there. I wouldn't mind a Windows solution, but if someone wants to take the time to give me the *nix basics, I'd appreciate that too!

Geoff

 

[Sqiz]

You could give SetiTEAM 2.1h a try if you are running Windows

http://www.sqiz.co.uk/Seti/history.htm

Sorry, just noticed that you want to capture several different pages

 

[n0cmonkey]

Originally posted by: GeoffS
That stuff sounds like Linux I'm not well versed in Linux... yet

I understand that cron (*nix) = at (Windows), and the pages are pure text, so no complications there. I wouldn't mind a Windows solution, but if someone wants to take the time to give me the *nix basics, I'd appreciate that too!

Geoff

at is another Unix program

$ more stats.sh
#!/bin/sh

#start in ~/eccp109 directory
cd /home/n0c/eccp109/

#make a directory for the day
mkdir `date "+ %d.%m.%Y"`

#move into that directory
cd `date "+ %d.%m.%Y"`

#lynx -dump to get the stats for the day
lynx -dump http://www.nd.edu/~cmonico/eccp109/team19.html > team19.txt

Thats what I was using. It can easily be modified to grab mroe pages and whatnot.

The cron entry:
#30 1 * * * /bin/sh ~/statprep.sh

 

[GeoffS]

Well then... guess it's time to fire up Linux on one of these D2OL boxes.... I should anyway since the client seems to run so much faster on that OS

Geoff

 

[ProviaFan]

Just a quick question now; what's the UNIX utility that can do search & replace on files? I know grep can be used for searching for stuff, but what does the replacing? I mean, stuff like stripping out the heading and footing HTML, removing the table row and data tags, and stripping commas from the numbers so they won't be confused with the commas that I insert in place of the <td>...</td> tags to separate the data items before importing to Access. All this I do now in UltraEdit-32, but I must do many manual search&replace operations, each time filling in the dialog box with a different thing to remove or replace. That's a big PITA to do early each morning, and I often screw it up and have to start over; a script would be much easier.

 

[n0cmonkey]

Originally posted by: jliechty
Just a quick question now; what's the UNIX utility that can do search & replace on files? I know grep can be used for searching for stuff, but what does the replacing? I mean, stuff like stripping out the heading and footing HTML, removing the table row and data tags, and stripping commas from the numbers so they won't be confused with the commas that I insert in place of the <td>...</td> tags to separate the data items before importing to Access. All this I do now in UltraEdit-32, but I must do many manual search&replace operations, each time filling in the dialog box with a different thing to remove or replace. That's a big PITA to do early each morning, and I often screw it up and have to start over; a script would be much easier.

sed or awk, both of which I know very little about.

 

[ProviaFan]

Originally posted by: n0cmonkey

Originally posted by: jliechty
Just a quick question now; what's the UNIX utility that can do search & replace on files? I know grep can be used for searching for stuff, but what does the replacing? I mean, stuff like stripping out the heading and footing HTML, removing the table row and data tags, and stripping commas from the numbers so they won't be confused with the commas that I insert in place of the <td>...</td> tags to separate the data items before importing to Access. All this I do now in UltraEdit-32, but I must do many manual search&replace operations, each time filling in the dialog box with a different thing to remove or replace. That's a big PITA to do early each morning, and I often screw it up and have to start over; a script would be much easier.

sed or awk, both of which I know very little about.

Hmm, guess it's time for me to learn some regular expressions and Read Some ****ing Man Pages.