WCG - 16th Birthday Challenge

Terminator_1

Member
Jul 14, 2013
34
35
91
Dear members of ANANDTECH,

For Nov 16th to Nov 22nd 2020 SETI.Germany is arranging a challenge at World Community Grid on the occasion of the grids 16th birthday:

16thbirthday_logo_720.png



We would like to invite ANANDTECH to participate in the challenge and hope that your team and many members of your team will be joining and participting this event.
!!Be aware, only your WCG-Team-Captain can join your team to the challenge!!
https://www.worldcommunitygrid.org/team/challenge/viewTeamChallenge.do?challengeId=10277

More information and statistics regarding the challenge can be found here.http://www.seti-germany.de/wcg/1_en_Welcome.html


Your participation is very much appreciated.

Best regards
Terminator
SETI.Germany
 

StefanR5R

Elite Member
Dec 10, 2016
5,459
7,718
136
For anyone wondering, these are the precise start and end times of the challenge:
Start:
00:00 UTC on Monday, November 16​
19:00 EST on Sunday, November 15​
16:00 PST on Sunday, November 15​
End:
00:00 UTC on Monday, November 23​
19:00 EST on Sunday, November 22​
16:00 PST on Sunday, November 22​
 

StefanR5R

Elite Member
Dec 10, 2016
5,459
7,718
136
Less than 10⅔ days left until start of the challenge.
Bunkering can commence in less than 16 hours from now if you select subprojects with 10 days deadline.

(Note, from what I remember, you may receive some tasks with a shorter deadline than normal for the respective subproject. These are 2nd or higher replicas from workunits of which the 1st task needs a validation or was not completed. When you load your bunkers, check for such special tasks; you may abort + report them right away if you require the standard deadline for all tasks in your bunker.)

I am now running WCG in a number of client instances in parallel, in order to get them recognized as reliable hosts by the project servers. (Edit: This is necessary for a high quota of tasks in progress per host, i.e. for the servers to allow for deep work queues of the clients.) When I get home from work, I will check the average run times, since I haven't been runnig WCG for a while now. I will certainly have to create more instances on my newer computers and prime them in the same way. A little late to do that now, but not too late.
 
Last edited:
  • Like
Reactions: TennesseeTony

StefanR5R

Elite Member
Dec 10, 2016
5,459
7,718
136
Less than 10⅔ days left until start of the challenge.
Bunkering can commence in less than 16 hours from now if you select subprojects with 10 days deadline.
As far as I am concerned, it did commence. :-)

Meanwhile, the Thor challenge will still receive some results from me, as my preparation of client instances during yesterday left me with 199 pages of results pending validation.
 

crashtech

Lifer
Jan 4, 2013
10,521
2,111
146
I was having trouble finding a page on WCG's site that gives all the task deadlines.
 

Icecold

Golden Member
Nov 15, 2004
1,090
1,008
146
What qualifies a host as being 'trusted' ? Is it a set number of completed tasks? I should have planned ahead better but I have a machine that won't download much more than the amount of threads it has.
 

StefanR5R

Elite Member
Dec 10, 2016
5,459
7,718
136
@Icecold, I had looked around at the WCG site, but haven't found answers. Things like that are probably adjusted from time to time too. It's perhaps a minimum number of validated results, returned within a specific time frame. Also, the 'reliability' property may not just be per host, but per host and per application.

A host which is not yet deemed reliable seems to have a quota of tasks in progress of # of active CPUs + 2, from what I have seen.

In this list:

I don't see OpenPandemics COVID-19. Is it not part of the challenge?
This page has not been updated after last year's challenge.

As @Ken g6 noted, the challenge is across all subprojects.
 
  • Like
Reactions: Icecold

Markfw

Moderator Emeritus, Elite Member
May 16, 2002
25,482
14,434
136
Stupid question, how do you know if a host is trusted or not ?
 

Icecold

Golden Member
Nov 15, 2004
1,090
1,008
146
Stupid question, how do you know if a host is trusted or not ?
If it's not trusted it will only download just barely more than enough work to run on the amount of cores/threads you have. I had the same results as @StefanR5R - it downloads the number of active CPU's + an additional 2 tasks. I'm not sure of any other way of identifying if it's trusted or not. The trusted hosts download enough tasks to run for days.
 

Markfw

Moderator Emeritus, Elite Member
May 16, 2002
25,482
14,434
136
If it's not trusted it will only download just barely more than enough work to run on the amount of cores/threads you have. I had the same results as @StefanR5R - it downloads the number of active CPU's + an additional 2 tasks. I'm not sure of any other way of identifying if it's trusted or not. The trusted hosts download enough tasks to run for days.
When you have 128 cores/threads, then 130 vs 1000 may be harder to see.
 

Icecold

Golden Member
Nov 15, 2004
1,090
1,008
146
When you have 128 cores/threads, then 130 vs 1000 may be harder to see.

I'm using just the standard BOINC manager so it might be different if you're using BOINC tasks or something, but I just sorted by status and it seems fairly clear(mind you this is 56 threads, not 128 and you could be running other projects on the same machine, etc.). I would think regardless of what you're using, though, that it would be clear if there are a bunch of tasks waiting to run, or just 2.

Edit - is there no way to see tasks in progress by host on the WCG site? I thought that might be easier for Markfw on a 128 thread machine but I couldn't find it. I have trouble navigating their site since it's different than the standard BOINC site, but I could only find the page that shows results by computer it doesn't show specific tasks.
 

Attachments

  • wcg.PNG
    wcg.PNG
    1.1 MB · Views: 25
Last edited:

biodoc

Diamond Member
Dec 29, 2005
6,257
2,238
136
Edit - is there no way to see tasks in progress by host on the WCG site?
It's cumbersome but go to My Contribution: Results Status and then you can filter results status by device name and project.
 
  • Like
Reactions: Icecold

Icecold

Golden Member
Nov 15, 2004
1,090
1,008
146
I don't have an exact answer on what qualifies as a trusted host, but the bar seems fairly low which is fortunate. I let an instance run and report tasks for the last 5 hours or so and it's now picking up plenty of new tasks. The side benefit being those tasks reported today will go towards the Thor Challenge.
 

Markfw

Moderator Emeritus, Elite Member
May 16, 2002
25,482
14,434
136
It's shown OK in BoincTasks. But I prefer a script which I posted elsewhere, and will post again now elsewhere.
Its put together, but since xfinity is due here any minute, I don't want to be distracted. I will know tonight.
 

cellarnoise

Senior member
Mar 22, 2017
709
394
136
I keep running out of tasks while bunkering. I am now on a third new instance trying to get enough tasks to get through to the start of the race. Am I reading it right that WCG will only give 70 tasks per CPU thread?
 

Skillz

Senior member
Feb 14, 2014
911
929
136
I keep running out of tasks while bunkering. I am now on a third new instance trying to get enough tasks to get through to the start of the race. Am I reading it right that WCG will only give 70 tasks per CPU thread?

You need to give new clients (and instances) time to be trusted with WCG before you can get a lot of tasks. The max limit is ~1000 tasks per instance.
 

cellarnoise

Senior member
Mar 22, 2017
709
394
136
Thanks. I am new to all of this.
Who has this kind of time?! I want to set these machines up to do my bidding. I think in the end I am doing their bidding...

Good to learn how to do all this stuff, but it takes time. Now how do I get my TCL T.V. segregated off my network without new hardware? All these I.O.Things I now fear.

More time learning.
 

Markfw

Moderator Emeritus, Elite Member
May 16, 2002
25,482
14,434
136
Thanks. I am new to all of this.
Who has this kind of time?! I want to set these machines up to do my bidding. I think in the end I am doing their bidding...

Good to learn how to do all this stuff, but it takes time. Now how do I get my TCL T.V. segregated off my network without new hardware? All these I.O.Things I now fear.

More time learning.
Take my word for it (this is my first time bunkering) Its a lot of work, but once all done, the second time per instance is easy, and they are trusted. So it just a lot of work the first time, and I did not do it all now, I will have to spend a little time every week to set up more instances.
 

StefanR5R

Elite Member
Dec 10, 2016
5,459
7,718
136
Start and end of the challenge
For anyone wondering, these are the precise start and end times of the challenge:
Start:
00:00 UTC on Monday, November 16
19:00 EST on Sunday, November 15
16:00 PST on Sunday, November 15​
End:
00:00 UTC on Monday, November 23
19:00 EST on Sunday, November 22
16:00 PST on Sunday, November 22​
Well, my statement was not completely correct.

Each day at 00:00 UTC, Worldcommunitygrid takes a database snapshot, and then starts a statistics run to convert the then current boinc credits and time stats into their permanent record of WCG points, result counts, and CPU times. I suspect that the database snapshot happens reasonably quickly. The statistics run takes some time. I remember that the WCG web pages which are statistics related showed sort of a maintenance-in-progress message for the duration of this run, and it could sometimes last for hours.

The race starts
sometime around 00:00 UTC on Monday, November 16,
when WCG have taken the database snapshot.​
The race ends
sometime around 00:00 UTC on Monday, November 23,
when WCG begins to take the database snapshot.​

I don't know of a way to observe these two points in time.
Hopefully the snapshotting is an atomic operation, and maybe it is pinpointed to 00:00 UTC with a sub-second precision even. But I don't know whether this is true.


Statistics pages

Master statistics

The canonical statistics are WCG's own, at @Terminator_1's first link:
This table is updated only once a day, at some random time during the above mentioned statistics run. That is, while the actual acquisition of the raw stats data happens more or less precisely at 00:00 UTC, the processed results (team totals derived from user totals) appear on this page with an undetermined delay.

The header line "Statistics Last Updated: mm/dd/20 23:59:59 (UTC) [x hour(s) ago]" refers to the acquisition of the raw stats data which are basis of the currently shown page content.


Hourly statistics

SETI.Germany provide much enhanced statistics on their own web site at @Terminator_1's second link:
From what I understood, these statistics work like this:
Thanks to a special arrangement between SETI.Germany and WCG, these pages are updated hourly from boinc server data, converted by SG's site code into WCG points. In addition, SG's site synchronizes itself with the team statistics files from WCG's file exports or API exports (I don't know which one), which result from WCG's daily stats run.

Due to this daily synchronization with the canonical 24h stats, SETI.Germany's record of 1h stats may contain sudden jumps, even negative updates if the hourly private stats overshot the official daily stats.


Third party statistics

FreeDC:
BoincStats:
These statistics are derived from WCG's file exports or API exports (I'm not sure which one), once a day for teams, and twice a day for users.

These stats are showing boinc credits (a.k.a. cobblestones), which are 1/7th of WCG points. (WCG's canonical stats and SETI.Germany's hourly stats are both showing WCG points = 7 times as much as FreeDC and BoincStats are listing, for historical reasons.)

Perhaps some of our more active WCG contributors can chime in and give the times of day when the 3rd party stats are usually updated.
 

StefanR5R

Elite Member
Dec 10, 2016
5,459
7,718
136
I keep running out of tasks while bunkering. I am now on a third new instance trying to get enough tasks to get through to the start of the race. Am I reading it right that WCG will only give 70 tasks per CPU thread?
You are right that this is what the FAQ says. But it is not the complete picture, and I haven't checked whether this FAQ is up to date.

Additional limits:
You need to give new clients (and instances) time to be trusted with WCG before you can get a lot of tasks. The max limit is ~1000 tasks per instance.
Right, hosts which are not (yet) deemed reliable by the project server currently have a server-side enforced limit of tasks in progress of active logical CPUs + 2, from what I have seen.

The ≈1000 tasks limit is a client-side limit: The client no longer sends requests for new work as soon/ as long as there are ≥1000 runnable tasks present on the client.

Further, you can define additional limits on tasks in progress for yourself at the WCG web site in "Settings" -> "Device Manager" -> "Device Profiles" -> profile name -> Project Limits. These user-defined limits are then enforced by the server.