Any projects need some stats?

[GeoffS]

I've written some stats pages for DPAD and D2OL. I'm open to creating stats pages for other projects that the TeAm is involved in. Here's where you can find my work

D2OL Stats Page

DPAD Stats Page

The DPAD stats update hourly, the D2OL status update daily except for the node information, which updates hourly.

I guess the only criteria is that the TeAm would gain something from the effort required to adapt my stats to <project-x>, meaning that the 'official' stats for <project-x> don't already have everything I have to offer, and that the project has a significant following here.

LMK what you guys think, and if you want to work out a priority list of what order to do them in, that would be great too!

Geoff

 

[Freewolf]

I would say TSC but there wouldn't be much much changes in the updates

 

[n0cmonkey]

If you just want something to work on, try ecc2. Their stats page is half broken when I check it.

 

[ProviaFan]

I realize that there are probably more pressing needs, but F@H would benefit from TeAm-"sponsored" stats. I've had the intention to make some web-based stats that would also aid in the production of the daily stats posted here (I used to do it by hand, aka Copy & Paste with some tables in Access - don't know how Hyperfocal does it now). Unfortunately, interfacing PHP with MySQL is not one of my strong points (that's a severe understatement), and I've not been able to figure out how to efficiently do the storage of score count history to generate graphs and stuff. Being able to see some code from stats for another project would help, but so far no one has been willing to either GPL and/or publicly release some code for me to see, or work on it themselves, but I'd love to help in this area if I could get so little as a push in the right direction.

 

[n0cmonkey]

Originally posted by: jliechty
I realize that there are probably more pressing needs, but F@H would benefit from TeAm-"sponsored" stats. I've had the intention to make some web-based stats that would also aid in the production of the daily stats posted here (I used to do it by hand, aka Copy & Paste with some tables in Access - don't know how Hyperfocal does it now). Unfortunately, interfacing PHP with MySQL is not one of my strong points (that's a severe understatement), and I've not been able to figure out how to efficiently do the storage of score count history to generate graphs and stuff. Being able to see some code from stats for another project would help, but so far no one has been willing to either GPL and/or publicly release some code for me to see, or work on it themselves, but I'd love to help in this area if I could get so little as a push in the right direction.

There needs to be an "Open stats" project

 

[CyGoR]

Nice gesture Geoff!

jliechty, the hardest part is usually the fetching and inserting of the data in the database. When it's in there, getting it out and displaying it is easy, you just have to learn a few lines of code of how to get stuff out of a database. After that the rest is just plain html.. And as you gain more experience with PHP you can add functions, graphs etc..
I can't help with writing a script for you to fetch the needed data of the site with PHP, for that cause I only have Perl experience..

 

[n0cmonkey]

Originally posted by: CyGoR
Nice gesture Geoff!

jliechty, the hardest part is usually the fetching and inserting of the data in the database. When it's in there, getting it out and displaying it is easy, you just have to learn a few lines of code of how to get stuff out of a database. After that the rest is just plain html.. And as you gain more experience with PHP you can add functions, graphs etc..
I can't help with writing a script for you to fetch the needed data of the site with PHP, for that cause I only have Perl experience..

WARNING: I have no experience with this and am a complete moron most of the time!

I'd think formatting the data would be the toughest part. Getting the actual data from the website could be as simple as lynx -dump www.ebsite.com Formatting that into something usable and then injecting it into the db server sounds tougher.

EDIT: Stupid fusetalk.
EDIT2: stupid fusetalk again. Preview must be broken :|

 

[GeoffS]

jliechty - I don't mind sharing... Some of the D2OL stats are found HERE. The loader code I use is as follows (unfortunately all the indenting is lost because we can't have a CODE tag here :|) :

// Load team current daily totals

echo date("H:i:s")." > Loading topTeamsAll(yesterday) data...";
$filename = "http://d2ol.sengent.com/stats/topTeamsAll.jsp?t=yesterday"; // URL
$line_array3 = file($filename); // Load the contents of URL into an array
echo sizeof($line_array3)." elements\n"; // How big is it?

// Parse and insert team current daily totals

echo date("H:i:s")." > Parsing and loading topTeamsAll(yesterday) data...\n";
foreach ($line_array3 as $array_line) { // For every element in the array...
$pos = strpos($array_line, "|"); // check if there is a delimiter

if ($pos === false) { // if not, do nothing...
} else { // else...
$line_vals = explode("|",$array_line); // blow out the delimited data into another arrary
if ($line_vals[0] != "Rank") { // if this is not a head record, load it into the db

$Query = "INSERT INTO temp_team_daily
(td_team, td_cands)
VALUES ('".addslashes($line_vals[1])."',$line_vals[2])";

if (mysql_db_query ($_SESSION['dbname'], $Query, $_SESSION['Link'])) {
} else {
echo date("H:i:s")." > *ERROR* Team daily insert failed for ".$line_vals[1]."\n";
}
}
}
}
unset ($line_array3); // free up the memory

Pretty simple, hunh?

I've just started looking at how to parse XML data... LifeMapper data is in XML format instead of a nice & easy delimited format

Geoff

 

[GeoffS]

Where are the downloadable F@H stats?

 

[InlineFive]

GeoffS will soon be our all around stats admin.

 

[GeoffS]

heh... there's a good title to replace Diamond Member with!

 

[ProviaFan]

Originally posted by: GeoffS
Where are the downloadable F@H stats?

Raw Team Stats
Raw User Stats

Beware that (IIRC) a user may appear multiple times in the user stats if s/he has contributed to more than one team.

Hey, if you want to do these F@H stats, that's fine with me. I was just offering because I wouldn't mind something to motivate me to learn some more PHP and some MySQL, and I didn't know if you would want to put your efforts into something else that didn't have any third party stats at this time (there are some for F@H, but I think having our own would be better). In any case, let me know what you decide, so we won't be duplicating effort.

BTW, what I was wondering, more specifically, was how the database should be set up to store results over a period of time, to be able to generate graphs? Does one use multiple fields for the past x periods, shifting them over by one and inserting a new value for each run of the stats? Or is there a more efficient way that you prefer to do this?

 

[GeoffS]

Originally posted by: jliechty

BTW, what I was wondering, more specifically, was how the database should be set up to store results over a period of time, to be able to generate graphs? Does one use multiple fields for the past x periods, shifting them over by one and inserting a new value for each run of the stats? Or is there a more efficient way that you prefer to do this?

Well... that load is into a temporary table, so it gets reloaded each time the stats run. The data ends up in a table that looks like this:

CREATE TABLE user_total (
ut_userid varchar(255) NOT NULL default '',
ut_date bigint(20) NOT NULL default '0',
ut_cands bigint(20) NOT NULL default '0',
ut_nodes bigint(20) NOT NULL default '0',
PRIMARY KEY (ut_userid,ut_date),
KEY ut_date (ut_date)

That table has a date in it (in this case, it's not a real date/timestamp, just a really big number and the size of the field is way overkill) and contains a record for each user for each day.

For ease of selection purposes, you would want to add a record daily instead of keeping a single record for a team or user and adding a new column to that record each time you have new data.

Even for the rolling 24-hour node information in the D2OL stats, I keep a separate record for each node for each hour. I add the new hour records and drop the oldest hour records each hour instead of manipulating a record with 24 buckets on it. In this case, the 24 buckets may actually work better for me than the way I am doing it now, however if you don't know how many buckets you need (like continually adding new data for every day), then additional records would seem to me to work better.

Geoff

 

[GeoffS]

Originally posted by: jliechty
Hey, if you want to do these F@H stats, that's fine with me. I was just offering because I wouldn't mind something to motivate me to learn some more PHP and some MySQL, and I didn't know if you would want to put your efforts into something else that didn't have any third party stats at this time (there are some for F@H, but I think having our own would be better). In any case, let me know what you decide, so we won't be duplicating effort.

Be my guest! LMK if you want to go ahead with that or not.

The first (and only) book that I bought on the topic of php is the PHP Bible 2nd Edition (Tim Converse and Joyce Park, ISBN 0-7645-4955-3)... are $50 new, but it covers just about every topic you can think of. That, and a help reference that I found online are all I use.

If you don't have a database background, you'll want to find a book on relational databases to get an understanding of how they work and data normalization, etc before you really delve into creating databases. A poorly designed database will be inefficient and difficult to maintain and query I don't have any suggestions in that area as I've been working in relational databases since dBase II was all the rage.


I'm still waiting to hear from folks about which projects could use some more stats... surprisingly, not too many bites here!

Geoff

 

[MereMortal]

Originally posted by: GeoffS

I've just started looking at how to parse XML data... LifeMapper data is in XML format instead of a nice & easy delimited format

[grumpy old man voice] Back in my day, we didn't have any of these fancy delimited stats files. We did regex searches, and we liked it! You young whippersnappers have it far too easy! *cough* *cough* *hack* Now see what you've done? You've stirred up my pneumonia! *cough* You damn kids get out of my yard!![/grumpy old man voice]

 

[amdxborg]

lol Well...you know me bit first...Lifemapper...I wasn't going to say it again...but since you aren't getting enough work...I'm gladly mentioning it again...

 

[GeoffS]

... had to walk 5 miles, uphill both ways, in the driving snow, with cut up milk cartons on my feet for boots, to get the stats and had to bring them home in a bucket and transcribe them to punchcards!

 

[amdxborg]

:Q

 

[ProviaFan]

I guess I would like to try my hand at some F@H stats, unless the F@H team would rather have GeoffS do it...

 

[GeoffS]

Originally posted by: jliechty
I guess I would like to try my hand at some F@H stats, unless the F@H team would rather have GeoffS do it...

Go ahead... if you want a peek at some of my code I'll pass it on to you. I'll tend to the LifeMapper project for the moment

Geoff

 

[BlackMountainCow]

LM would be great indeed ... seems to be a promising project for TA!

 

[RaySun2Be]

Excellent stats, Geoffs!

I don't have any suggestions in that area as I've been working in relational databases since dBase II was all the rage.

DBaseIII+ & Clipper....now those were the days! ah, the memories.

 

[GeoffS]

Yeah... forgot about Clipper!