I need help from you stats people

[TallBill]

How do you use scripts to do it? I might be able to get microsoft sql server.. and i run linux so i could easily run a sql server on that... but where do i even start?

 

[ProviaFan]

I want to make a script to do F@H stats, too, so perhaps some information could help both of us.

I think the easiest and best way would be to do something like this, as is illustrated by the pseudocode below (but I'm open to suggestions for improvement before I actually try to learn awk / perl / sql just to implement this stuff):

1. Download the stats from the project website (cron script with wget would work fine)
2. Run the stats through a filter to strip the HTML tags if necessary and convert the stats to a comma-delimited list (sed or awk would be quite up to the task)
3. Use the numbers in the list to go through a database and update the production info of each member (noting when milestones occur and reporting any members which did not previously exist in the database)
4. Use a SQL queries to extract the data necessary
5. Execute a simple script to take the data from the queries of #4 and assemble them into a text file for posting on the forums (a simple BASH script would work here AFAIK, but the gurus might prefer Perl instead)

AFAIK, numbers 1 and 5 should be very easy (I could probably set up those two parts within minutes if I wanted to try). Number 4 would involve learning some SQL; it doesn't seem like it would be that hard to do. Number 2 would be somewhat harder, but a little knowledge of regular expressions and sed or awk would go a long way there. Number 3 would be by far the hardest to do, and I have no idea where to start on that one. Thus, if any UNIX / database guru happens to read this, and feels like offering me some pointers, that would be the area in which I need a shove in the right direction the most.

 

[Baldy18]

I have no idea how it works. Meremortal made the D2OL stats script and I just run. MM and OD are geniouses though.

 

[Confused]

Originally posted by: Baldy18
I have no idea how it works. Meremortal made the D2OL stats script and I just run. MM and OD are geniouses though.

I'd like to add Soni to that list, too

All of this stuff is totally over my head!

Confused

 

[TallBill]

unfortunatly i wont be learning how to do all this in school for another year or two.. so i need to skip ahead

 

[ProviaFan]

I'll probably just do what I can by myself from reading tutorials and documents, and when I get stuck (presumably on the database manipulation part), I'll just come here and scream for help.

 

[CyGoR]

Same story here like jliechty, just reading some tutorials..
I'm busy now with MySQL, Apache and PHP which is a very very good combination like most people know..
I'm now learning the language first before I start with big projects, because I want to be able to do everything on my own.
But it takes a lot (read a LOT) of time tu understand the language of PHP and SQL..
But hey, it's a nice hobby

Good luck!

 

[TallBill]

alright.. sql shall be my nice project then

 

[MereMortal]

There are many ways to skin a cat...um...I mean write stats scripts. I use PHP for scripting and MySQL for a database, because I know they work well together, and because I had a sample script in these languages to learn from. About a year ago, I had very little knowledge of PHP or MySQL, but taught myself. Granted I have a number of years of programming experience, and after a while all languages look the same.

Actually trying to write a script (and learning how do do things along the way) will teach you much. It's like a motivated practice problem. The way I write scripts is based to a large degree on my background. Coming from a scientific programming field, I tend to like to manipulate all my data with the scripting language using multidimensional arrays. Other folks like to do a bunch of the calculations in the database. So use a style that you feel comfortable with; unless you are manipulating large volumes of data, or are scripting for real-time stats, efficiency isn't all that crucial.

 

[GeoffS]

I use an Excel spreadsheet... dump in the data, let it do the calcs, and do a filter to get just our team, et voila!

Geoff

 

[ProviaFan]

Originally posted by: MereMortal
There are many ways to skin a cat...um...I mean write stats scripts. I use PHP for scripting and MySQL for a database, because I know they work well together, and because I had a sample script in these languages to learn from. About a year ago, I had very little knowledge of PHP or MySQL, but taught myself. Granted I have a number of years of programming experience, and after a while all languages look the same.

Actually trying to write a script (and learning how do do things along the way) will teach you much. It's like a motivated practice problem. The way I write scripts is based to a large degree on my background. Coming from a scientific programming field, I tend to like to manipulate all my data with the scripting language using multidimensional arrays. Other folks like to do a bunch of the calculations in the database. So use a style that you feel comfortable with; unless you are manipulating large volumes of data, or are scripting for real-time stats, efficiency isn't all that crucial.

Thanks for the tips, MereMortal. I'm pretty knowledgable in Visual Basic, but that's kind of a beginner's language, so I don't know if it will help much if / when I try to learn PHP and SQL.

Originally posted by: GeoffS
I use an Excel spreadsheet... dump in the data, let it do the calcs, and do a filter to get just our team, et voila!

Geoff

Right now I'm using an Access database to do the Folding@Home stats. It's kind of cumbersome, and I have to manually type the milestone makers and top 10 daily producers into the stats post after I sort them out of the database. Hopefully a script will help in this regard.

 

[GeoffS]

I've got the calcs making the text of the MM post, with the markup-tags, in the sheet itself. All I do is copy & paste

For example, for a line that looks like this:

TA_GeoffS crossed 50000 and climbed 1 spots overall! :Q:Q:Q:Q:Q

my calc looks like this:

=IF(I10="Y","[ B]"&A10&"[/b] crossed "&IF(AND(I10="Y",OR(J10<>0,K10<>0)),(J10*10000+K10*1000),IF(L10="X","750",IF(M10="X","500",IF(N10="X","250"))))&IF(E10>0," and climbed [ B]"&E10&"[/b] spots overall! "," ")&REPT(":Q",J10)&REPT("",K10)&IF(L10="X","",IF(M10="X","",IF(N10="X",""," ")))," ")

Looks ugly, but it's quite simple If there is a milestone, the text is generated, otherwise, nothing is generated. Obviously, it's been tailored to meet my specific needs, but it's pretty simple to change. Each morning, I import 3 files into my spreadsheet and the cut & paste the resulting text into the post

Geoff

 

[soni]

Next step, a script that make the Post here

 

[ProviaFan]

Originally posted by: soni
Next step, a script that make the Post here

That might not be too hard to do, after all, if it were possible to find out how the message posting form works and somehow imitate it without actually requiring any human intervention in a web browser.

 

[TallBill]

you could do a keyboard and mouse macro

 

[OhioDude]

I'm pretty knowledgable in Visual Basic, but that's kind of a beginner's language

I beg your pardon... :disgust:

That's virtually all we develop in. That and Java. We develop business intelligence and data warehousing applications that interface with legacy OLTP systems and transfer and store legacy data in ODBC/OLEDB compliant industrial strength databases like SQL Server and Oracle. Beginner's language, indeed! 95% of what we need to do we can do in VB. The other 5% we may need to write some Java or some C++, but the majority of the nuts and bolts stuff is VB.

The SETI race app is an Excel workbook with VB code that connects to Berkeley, downloads the team stats, messages, calculates, pushes, pulls, and otherwise manipulates the data and spits out my race thread every morning. It's waiting for me in Notepad when I get to work every morning. I just add smilies, commentary, and post.

 

[ProviaFan]

Originally posted by: OhioDude

I'm pretty knowledgable in Visual Basic, but that's kind of a beginner's language

I beg your pardon... :disgust:

That's virtually all we develop in. That and Java. We develop business intelligence and data warehousing applications that interface with legacy OLTP systems and transfer and store legacy data in ODBC/OLEDB compliant industrial strength databases like SQL Server and Oracle. Beginner's language, indeed! 95% of what we need to do we can do in VB. The other 5% we may need to write some Java or some C++, but the majority of the nuts and bolts stuff is VB.

Woah, sorry. I thought, since VB is what is commonly taught to beginners, and since it has a tendency to be slower than C++ for certain (most?) things, that it was a "beginner's" language. As far as the language goes, I've never gotten in to using it with SQL servers and stuff like that, mainly since I can't afford those kind of things for my computers at home. The only database work I've done with VB is using the ADO controls.

 

[TallBill]

hehe.. i actually just picked up a book on VB (i know some, need to know more), and a sql book gimme a few weeks

 

[InverseOfNeo]

Originally posted by: Confused

Originally posted by: Baldy18 I have no idea how it works. Meremortal made the D2OL stats script and I just run. MM and OD are geniouses though.

I'd like to add Soni to that list, too All of this stuff is totally over my head! Confused

I guess you truly are Confused about that huh? LOL

 

[m0ti]

Currently I do it all by hand... don't have the time to set anything more complicated up.

Though this is also on the list for me when I get more time.

I'm not going to go to the trouble of learning PHP or Perl, I'll just stick with Java and set something up in JSP. Database will either be SQL Server or MySQL. Doesn't make much of a difference to me.

 

[ProviaFan]

Maybe this thread (subscriber link) would be of interest to those who want to automatically post their stats.