BOINC Benchmarks

[mrwizer]

In the Seti Classic "Whats wrong" thread, cheating is being discussed. So it made me think about BOINC and cheating.

On a lot of Forums, optimized clients have been debated as possibly cheating. I have not heard many mentions of it here... so what is the consensus on this? Does the majority feel that using optimized clients to inflate benchmarks scores is a way of cheating? Or is it simply a way to make your machine run more efficient?

I have also heard it put that it is aligned to overclocking your machine, and something that everyone could do.

Perhaps the BOINC gurus could also chime in on whether this is even a concern. And I also heard that the BOINC team was going to change this in the future??? (And remember, this is optimized BOINC clients, not the science apps).

So not trying to start a flame war, or heated discussion with this. Please be nice.

 

[Wolfsraider]

I am using them, it is optimized and available for the projects, not unlike everyone using the 3.03 version of seti classic over the 3.08, why because it was faster

There isn't anyone saying "you won't get credit from our project if you use them", and a lot of teams are using them.

they still crunch the work and only turn in 1 result.

This was discussed a few times and there are folks on both sides. Clearly no lines have been drawn, unless you count the silence from the projects as their acceptance of the use of these optimized clients.

Unless I see that that they are ruling against it (the projects) I will continue to use them.

My .02 worth

 

[mrwizer]

I thought I heard rumors of some projects saying the optimized clients skewed results. Is this not true?

 

[BlackMountainCow]

There was a problem with optimized cleints that had a fixed benchmark value in them. So, no matter how fast your PC was, you were able to set a fixed benchmark score --> higher credits for each WU. I don't know if they're still around or if they are blocked somehow.

 

[mrwizer]

Originally posted by: BlackMountainCow
There was a problem with optimized cleints that had a fixed benchmark value in them. So, no matter how fast your PC was, you were able to set a fixed benchmark score --> higher credits for each WU. I don't know if they're still around or if they are blocked somehow.

Perhaps this is why they are re-thinking the credit system?

 

[rise]

i'd like to hear more peoples opinion about/knowledge of the optimization.

 

[BlackMountainCow]

If you use an optimized client to make up for the fact that your linux box is scoring much lower than a windows box does - and that's a fact - then I think it's ok to use an optimized client to make up for a flaw of the project/BOINC. If you want to get the science done quicker, even better, use an optimized client. If you're a lame cheater, go home and stand in the corner.

But the easiest way against any cheating is the quorum. It's very unlikely that with a quorum, all WUs will be send to optimized computers. So, if you have one cheated WU with 80 creds and two others with only 50, the cheater won't get the high score but the average one. (right?).

But there are several ideas around these days:

• Introduction of GFlops --> no cheating. The time you crunch is credited (like FaD did it)
• Fixed credit --> no cheating. That's how CPDN handles their credit system.

AFAIK, SETI is already working on the GFGlops system.

 

[lane42]

I will say there Fast !!!

Just downloaded to my A64 3000+ laptop @ 1.9

Before, 2 and a half hour wu times, 10 a day.
After, 47 min. wu times, 30 a day :Q

IMO, The only one that can say it's cheating is Berkeley, and they have not..

 

[petrusbroder]

Lets get the terminology straight: all the below is AFAIK and I may be wrong:
BOINC in it self is a client-server system (It was such a long time i used that expression ... ).
The system consists of the servers, a client and a application, where the server is the project's server (or servers as in Seti) and the client resides in each computer which crunches WUs and (specifically for BOINC) several applications (which crunch WUs).
The client is the BOINC-manager etc itself. The client does no scientific calculations at all. It does the house keeping, score keeping, benchmarking, etc. Optimized clients present sometimes trouble with fixed Whetstone/Drystone-numbers, incorrect reporting, inflated/deflated scores etc.
The application does the crunching and is different from every project - Rosetta has its application, Seti has several applications, CPDN has its etc.
Using optimized applications, which crunch to the project acceptable results, but much faster (that is the case with Seti@home only AFAIK; but there are several good ones for the Macs and PCs), is good for the science and perhaps not so good for the scores: if you crunch faster you claim less credit, and you may get less credit. But that is often not the case because the longest crunching time and the shortest are discarded. If the crunching times are very similar to each other, the granted credit is usually the average of the 3 - 4 times reported. However, this may hit the other way too, that happens if the other results have reported shorter times than yours then you get less credits.

IMHO running optimized application is not only OK but should be encouraged: if the application is good then the results will be acceptable, and more results will be crunched (I increased my seti-crunching by 30-50% in WUs crunched and by 20 - 30% in scoring). That is good for science and for the stats because more WUs crunched should give more credits. However, the scoring system is not "used" to optimized application, and therefore the credits obtained are not quite as good as they could be.

I see no problems running optimized clients as long as the functions of the client are not harmed. The client - if "optimized" for maximal scoring - is one way to cheat and in my opinion clients with "fixed" numbers for Whet- and Drystones are not fair. This introduces also the ethical problem of two crunchers running the computers which give very similar Whet- and Drystones (when benchmarked) getting different credits in the same project. That is not fair play.

I know that the devs always work for optimizing the server-side and that is very good indeed.

 

[Rattledagger]

Crediting in BOINC is, at same time wu Validated crediting is decided after these rules:
1; If only 2 results passed validation, lowest claimed to all.
2; If 3 or more results, remove highest and lowest claimed and average the rest.

Any results reported later and also passing validation gets the same credit as the other results, no re-calculation of credit is done.


SETI@Home/BOINC uses both rules, but normally #2.



As for optimized core client, used on non-windows or paired with optimized seti-application to get similar claimed credit it's not a problem.

Even in other projects that don't have an optimized science-application it doesn't really matter if some users are using an optimized core client, since due to credit-system only if 2 has artificially inflated benchmark-scores will it affect granted credit. Due to the large variations in claimed credit the granted credit will also variate a lot, but average granted credit evens-out any spikes.


But, one project runs without any redundancy, this is Rosetta@home. Since no redundancy, users gets whatever their claimed credit is, this meaning anyone using an optimized core client gets artificially inflated benchmark, and this is directly translated into granted credit.

CPDN and Folding@home/BOINC haven't any redundancy either, but these projects uses fixed credit for each wu-type, so can't cheat on credit anyway.



Anyway, Seti_Enhanced "soon" to be released "counts flops", and therefore doesn't rely on the huge variations in BOINC benchmark.

 

[rise]

But, one project runs without any redundancy, this is Rosetta@home. Since no redundancy, users gets whatever their claimed credit is, this meaning anyone using an optimized core client gets artificially inflated benchmark, and this is directly translated into granted credit.

when did that start? i thought you were crazy when i first read it but i just browsed through my results and indeed all the granted credit is the same as the claimed.

i know this wasn't like this before when the optimzed clients came up.( a month or so ago iirc?) or maybe i'm nuts, always a possibility

also, just so i understand better, the theory behind these optimizations are that they make the client think the computer is slower than it actually is so that it crushes what the client assumed the time woulod be to finish the unit?

or is it making the client think its running on some super computer?

sorry for the dumb questions all. :Q


edit-typos

 

[Rattledagger]

Rosetta@home has always run without redundancy, but there's been various discussions on how to improve things...


Optimized science-application means runs faster, and takes less time. Since claimed credit = benchmark * cpu-time, to get same claimed credit as before they're trying to do the same optimizations on the benchmark to keep things in line.


In a project that don't have any optimizations in science-application, it means anyone running optimized benchmark will have too high benchmark, and therefore also claimed credit.

So it's basically to get the client to think it's running on a faster computer that it's really running on...

 

[mrwizer]

Originally posted by: rise4310

But, one project runs without any redundancy, this is Rosetta@home. Since no redundancy, users gets whatever their claimed credit is, this meaning anyone using an optimized core client gets artificially inflated benchmark, and this is directly translated into granted credit.

when did that start? i thought you were crazy when i first read it but i just browsed through my results and indeed all the granted credit is the same as the claimed.

i know this wasn't like this before when the optimzed clients came up.( a month or so ago iirc?) or maybe i'm nuts, always a possibility

also, just so i understand better, the theory behind these optimizations are that they make the client think the computer is slower than it actually is so that it crushes what the client assumed the time woulod be to finish the unit?

or is it making the client think its running on some super computer?

sorry for the dumb questions all. :Q


edit-typos

To further elaborate, optimizations take into account special features on your processor such as SSE2, Velocity, etc. It is not fooling your computer in the real since. The goal with the optimizations is to give you a better score on the sythetic benchmarks, which are then used to calculate credit as Rattle has shown.

Again, teh reason why I bring this whole thing up is what is cheating and what is not (and I would like to make it clear again that I am not claiming it is or isnt, just asking what others think). Others do things to make their system run better for 3DMark, is this cheating?

Perhaps rather than calling it cheating, some in the community find it unfair. And maybe it just brings up an issue with crediting style. All of this makes me sympathetic towards the BOINC devs, there are always going to be issues.

And Rosetta has ot valiated since the start. There were threads about it on the forum, but I have not checked the progress of them for some time now.

 

[rise]

thanks for elaboarating guys :beer:

i wonder if there is a comparison between (or correlation may be a better word?) to the client benchmark and say, sandra cpu benchmark?

 

[petrusbroder]

BTW, thanks for starting this thread, mrwizer. I really like the discussion so far.

 

[BlackMountainCow]

But there's one major difference with R@H: Only now, while we're still testing the search algorithm of R@H and are not yet doing some "real" science aka looking for cures, ther's no quorum. Once the step will be taken to cure search, R@H will introduce the quorum.

 

[Freewolf]

http://www.free-dc.org/forum/showthread.php?t=10312

 

[mrwizer]

Originally posted by: BlackMountainCow
But there's one major difference with R@H: Only now, while we're still testing the search algorithm of R@H and are not yet doing some "real" science aka looking for cures, ther's no quorum. Once the step will be taken to cure search, R@H will introduce the quorum.

At one point I heard talk about only using a system of 2 units to validate. Although this will check for scientific consistency, it will not be able to do anything with the credit. When using multiple units, you can throw out the high and low, and average the rest. But with only two, you can just average as you would not know what to throw out.

The largest problem with validation is that less science is done. So I understand where Rosetta is coming from.

 

[mrwizer]

Originally posted by: Freewolf
http://free-dc.org/forum/showthread.php?t=10312

Thanks for the link.

 

[rise]

ok, in the linked thread some are claiming that this just levels the playing field. but i don't see how that is.

if we take RD's formula of cpu time * benchmark= granted credit then slower cpus wouldn't be rewarded the same points as my x2 in the first place. so by running the optimized client this actually would be artificiially inflating my points.

edit- well, that is unless they're not benching my system properly to begin with.

unless i'm misunderstanding something (again)

i don't think the "tweaking for 3dmark" analogy works because in that case you are in fact making your system run faster by oc'ing your card, stopping other processes, lowering iq etc.

 

[Rattledagger]

Originally posted by: mrwizer
At one point I heard talk about only using a system of 2 units to validate. Although this will check for scientific consistency, it will not be able to do anything with the credit. When using multiple units, you can throw out the high and low, and average the rest. But with only two, you can just average as you would not know what to throw out.

The largest problem with validation is that less science is done. So I understand where Rosetta is coming from.

As already mentioned, if only 2 results used for validation, lowest claimed credit is given to both.

This system would stop anyone artificially increasing their benchmark, but the problem is on some computers the claimed credit is very low...


As for using redundancty to validate, for most projects it's easy since if not validated the result isn't usable, meaning no science at all without redundancy.
The reason Rosetta@home and Folding@home can get away without redundancy is likely since they're looking on molecules and a wrong calculation will show up as an "impossible" configuration or something...

But, in example SETI@Home that basically looks on random noise, you'll of course know something is wrong if a wu looks from 10 to 20 and you're reporting a signal at 25, but a signal at 15 on the other hand must be verified. Also, since a wrong calculation can mean a signal that otherwise would have been reported is not reported, must likewise make sure "no detected signal" is verified meaning must re-crunch same wu multiple times.

 

[Rattledagger]

Originally posted by: rise4310
if we take RD's formula of cpu time * benchmark= granted credit then slower cpus wouldn't be rewarded the same points as my x2 in the first place. so by running the optimized client this actually would be artificiially inflating my points.

edit- well, that is unless they're not benching my system properly to begin with.

unless i'm misunderstanding something (again)

claimed credit = benchmark * cpu-time.

Meaning, a 486-33MHz using 100h on a wu and a 486-33GHz using 0.1h on the same wu will both claim the same credit.

But of course, the 2nd computer will crunch 1000 results at the same time the 486-33MHz crunches only one result, so after 100h the 2nd computer will have 1000x as much credit.



In reality the benchmark isn't exactly linear, since a project can use other functions more heavily than the benchmark is using, and also other parts of the system like size of cache-memory and memory-speed can influence the crunching-speed.

Still, if uses redundancy and all results is from the same wu, computer A that crunches 10 results/day will have roughly 2x as much credit as computer B crunching only 5 results/day.

In a project that uses redundancy, even if A runs an optimized client giving 50% higher benchmark and therefore 50% more claimed credit, the granted credit likely will not increase significantly so will still get roughly 2x as much credit as computer B.

But, in Rosetta@home that does not use redundancy, 50% higher benchmark gets immediately translated into 50% higher granted credit, meaning A now suddenly gets roughly 3x as much credit as computer B...


So yes, in Rosetta@home running an optimized client will give artificially inflated credit...

 

[Insidious]

Thanks for the thread mrwizer. A topic that does need to remain transparent and it's teriffic to see people able to discuss this without a bunch of flames and sarcasm. KUDOS to all so far.

I have been using the 5.3.1 boinc client. When I first heard of it, I went to the Rosetta and started a thread asking if this was cheating. The answers were very ambiguous and certainly a strong point of view was not brought into the thread by the project devs. (If there was... I missed it.) The thread then got hijacked onto a discussion of "better scoring methods" which had little to do with my question so I quit watching it.

The main theme seemed to be that the people who used it had a list of reasons why it was OK for them (usually supporting SETIBoinc) but they considered themselves to be special cases. My reaction to that was.. BUNK!

It would be so incredibly simiple for BOINC to disallow any version of the BOINC client they chose to. (EVERY result you send in also indicates what client version was used.)
I have made a personall choice to assume if they are going to allow it, then how silly would I be to intentionally crunch for less credit than was available.

To further satisfy my curiosity, I started randomly looking at the result IDs of a lot of users (random... I said ) and found the use of the 5.3.1. client was VERY widespread. I used this fact to further my own personal rationalizations about using it myself.

Finally, I specifically poked around in the results of TeAm AnandTech Roseys and found a lot of people here are using it too. Again, I really have been persuing a "don't rock the boat" attitude and when I saw it was OK with many of my TeAm mates, I made my decision final and will continue using this client until some compelling reason comes to change.

Like the rest of you, I hope for a system where this kind of question doesn't even arise.. it will be hard with open source code to keep anything secure... I won't obsess and I won't worry. I'll just keep doing what seems right and let the chips fall where they may.

last word:

In MY OPINION (yes, that is only my opinion... I do not claim to be omnipotent.. or even correct) It isn't cheating to use the 5.3.1 boinc client and I will respect no Rosey any more or any less as a result of their personal choice in this matter.

-Sid

 

[mrwizer]

Originally posted by: Rattledagger



As already mentioned, if only 2 results used for validation, lowest claimed credit is given to both.

This system would stop anyone artificially increasing their benchmark, but the problem is on some computers the claimed credit is very low...

I guess it would also matter on the unlikely chance of two WU being crunched by two optimized clients. I think...

 

[mrwizer]

Originally posted by: Insidious

In MY OPINION (yes, that is only my opinion... I do not claim to be omnipotent.. or even correct) It isn't cheating to use the 5.3.1 boinc client and I will respect no Rosey any more or any less as a result of their personal choice in this matter.

-Sid

Is it just the standard build client, or an optimized build that has different files for P4's, AMD's, etc.? And 5.3, is that a development version?