writing a forum post-counter...

[Deeko]

Ok, we have a message board that is very primative, no post counts or anything, and its been requested that I write a post counter for it. Well, I wrote up a quick program that accepts a text file of posts and goes through and counts them...but thats horribly inefficient because then I'd have to copy and paste every thread into a text file. Is it at all possible to write something that will navigate to each thread automatically, download the text to that text file, and then run my original script on the file?

 

[notfred]

It's possible to write software that does anything you want.

What you should do is change your forum to increment a post counter every time someone posts a message.

 

[Deeko]

That would make life most simple....but its not my forum. Its a message board on another website.

The main question....how do I go about having the software navigate to a page, download the text(or the html, that would work just as well), and then navigate to the next?

 

[kamper]

Parsing html for information is not cool (of course it's been done, but only for much more complicated things than post counts). Is there no way you can query their database?

 

[mugs]

What type of storage is used for the forum? Database or flat files?

 

[notfred]

Originally posted by: Deeko
That would make life most simple....but its not my forum. Its a message board on another website.

The main question....how do I go about having the software navigate to a page, download the text(or the html, that would work just as well), and then navigate to the next?

Something like this:

String start = "http://www.somesite.com";
lookupPages(start);

void lookupPages(string startPage)
{
String html = downloadPage(startPage);
string [] links = getlinks(html);
for(int ii = 0; ii < links.length; ii++)
{
lookupPages(links[ii]);
}
}

 

[Deeko]

Its a database I'd imagine. I highly, highly doubt I'll get access to it though.

Here's the premise. There is a website called 'thefacebook.com'. It is a giant college directory basically. It supports 'groups' within your school...we have a group on there, that we use for its message board. The message board is, as I stated earlier, very primative. However, due to the scope of this entire website, I doubt I'd ever be allowed access to their db.

 

[Deeko]

Originally posted by: notfred

Originally posted by: Deeko
That would make life most simple....but its not my forum. Its a message board on another website.

The main question....how do I go about having the software navigate to a page, download the text(or the html, that would work just as well), and then navigate to the next?

Something like this:

String start = "http://www.somesite.com";
lookupPages(start);

void lookupPages(string startPage)
{
String html = downloadPage(startPage);
string [] links = getlinks(html);
for(int ii = 0; ii < links.length; ii++)
{
lookupPages(links[ii]);
}
}

looks easy enough. downloadPage...is a function of what?

 

[notfred]

Originally posted by: Deeko

Originally posted by: notfred

Originally posted by: Deeko
That would make life most simple....but its not my forum. Its a message board on another website.

The main question....how do I go about having the software navigate to a page, download the text(or the html, that would work just as well), and then navigate to the next?

Something like this:

String start = "http://www.somesite.com";
lookupPages(start);

void lookupPages(string startPage)
{
String html = downloadPage(startPage);
string [] links = getlinks(html);
for(int ii = 0; ii < links.length; ii++)
{
lookupPages(links[ii]);
}
}

looks easy enough. downloadPage...is a function of what?

downloadPage() and getlinks() are excercises left to the reader.

In perl there is "get()" from the LWP::Simple module, In java there is URL.openStream(), and I'm sure there are other similar functions in whatever other language you might use to retrieve data from webpages. I don't know of any library functions that will parse your page for links, though.

Also, you'll need to create some method of storing links you've already visited, so that you don't load the same page twice.

 

[WannaFly]

notfred gave some good pseudocode - what you are looking for is basically a screen scraper...I've written a few, can be written in any language just about