WCG - 16th Birthday Challenge

[pututu]

I only utilized to about 1/10 of what I could have in this contest, so maybe next time I can come up to bat.

The next big contest will be BOINC pentathlon 2021 in May and likely that WCG will be included since it wasn't in this year event.

 

[cellarnoise]

I do not mean any malice! Not at all.

I am on here and elsewhere to learn what others can be brought to bear...

There is a lot to learn about all of DC'ing. It is really a deep rabbit hole I fear! Just like anything that humans do for a long time and it now is evident that uses a lot of energy.

 

[crashtech]

I expected Stefan to have some bunkers at the end, but what happened was as much a surprise to me as I think it was to everyone! Amazing performance xii5ku, and great job to all those who provided the leg up, it is a team effort after all!

 

[VietOZ]

I would too, but Stephan is a little "private" so I just read what he posts. My 1000 cores are in my sig, no surprise there. But I can't touch him. I should say, he has taught me a few things, that I only utilized to about 1/10 of what I could have in this contest, so maybe next time I can come up to bat.

is it weird that a teammate doesn't know his teammate's gears and an opponent kinda know?
Stefan has 2x of 7452 and a few (i'll let you guys guess) xeon V4s.
More Gears is good, but if you don't know how to maximize them (or if you know but not willing to) then it's going to be in this situation.
While I may sound like a long time cruncher, actually I was, but i was off Grid for about 7 years before the come back in 2017. or maybe 16' ... beat me i can't remember sheit now lolz. Anyways, all i'm trying to say is thru out the crunching experiences, I've enjoyed going against xii5ku the most. We're not friends, we're not enemies, we don't even talk to each other. But every time a competition came up, I've always seek out to see what's he's doing. He may doesn't have much gears, but his timing and knowledges can do some real damages. So it's not always about the gears, it's about sharing your knowlegdes. PPL take it or not, that's on them ... we did what we can. I can't tell you how many times I've tried to shared something I learned and it turned out my team already knew about it. Embarrassing? Sure, but wtf. Just treat it like a double post... np.

 

[StefanR5R]

Stefan has 2x of 7452 and a few (i'll let you guys guess) xeon V4s.

This is true in a literal sense, but does not give the whole picture. :-P

--------

I am exclusively using computers which I built myself, and am operating them at home at German household electricity rate. As noted in this thread, I ran WCG for 17 days, not just 7. I'll write some more later today. Edit, or on another day, when I'm not tired.

 

[pipin]

Congrats guys, that was an impressive run in the last hours.

Only me can predict what xii5ku will do and still got beat
P3D just got a taste of it ... lolz
I don't mind to get the credit though ... haha
mehhh, that ain't right. TaaT deserved this win. Now let's see how P3D can cope with it

We never expected to win the Thor Challenge, so that made our year already.

And after some years with a lot 3rd places, this competition was fun und a success, too.

We will come back, with more cores, if those availability issues are gone and prepare for the next time.

Thanks to SG too for this competition and the stats.

 

[VietOZ]

This is true in a literal sense, but does not give the whole picture. :-P

--------

I am exclusively using computers which I built myself, and am operating at home at German household electricity rate. As noted in this thread, I ran WCG for 17 days, not just 7. I'll write some more later today.

And I have NOT said anything about you didn't built or owned the computers. I've just said you had 2x Epycs and some Xeon.
Just like you, and as many others, I built my own stuffs. I've got accused of renting or using free credit shiet, but i just laughed. "Yah, try to pay 800/month electric bill then talk to me about cloud." just my thoughts ... wasn't any quote or anything.

The estimate above was from an opponent's perspective. I could be wrong. Hell, i'd be a millionaire if I can predict everything right.

 

[Markfw]

is it weird that a teammate doesn't know his teammate's gears and an opponent kinda know?
Stefan has 2x of 7452 and a few (i'll let you guys guess) xeon V4s.
More Gears is good, but if you don't know how to maximize them (or if you know but not willing to) then it's going to be in this situation.
While I may sound like a long time cruncher, actually I was, but i was off Grid for about 7 years before the come back in 2017. or maybe 16' ... beat me i can't remember sheit now lolz. Anyways, all i'm trying to say is thru out the crunching experiences, I've enjoyed going against xii5ku the most. We're not friends, we're not enemies, we don't even talk to each other. But every time a competition came up, I've always seek out to see what's he's doing. He may doesn't have much gears, but his timing and knowledges can do some real damages. So it's not always about the gears, it's about sharing your knowlegdes. PPL take it or not, that's on them ... we did what we can. I can't tell you how many times I've tried to shared something I learned and it turned out my team already knew about it. Embarrassing? Sure, but wtf. Just treat it like a double post... np.

OI knew all of that... What I don't know is what he may not have told us. And, as I said, he taught me things, but too late to perform like he did. He is a master at this.

 

[Markfw]

And I have NOT said anything about you didn't built or owned the computers. I've just said you had 2x Epycs and some Xeon.
Just like you, and as many others, I built my own stuffs. I've got accused of renting or using free credit shiet, but i just laughed. "Yah, try to pay 800/month electric bill then talk to me about cloud." just my thoughts ... wasn't any quote or anything.

The estimate above was from an opponent's perspective. I could be wrong. Hell, i'd be a millionaire if I can predict everything right.

Oh, $800 a month electric bill ? How did you know thats what I pay ? And forget paying a cloud, I am my own cloud....

 

[StefanR5R]

Hi @VietOZ, my mentioning of using self-built computers at home was a general remark, directed to readers who potentially wonder, not in response to your post.

--------

Congrats to Planet 3DNow! for a tremendous WCG performance at Thor and Birthday in a row. We couldn't quite match yesterday what you did on the last Thor day... but weren't far off. :-) See you!

 

[StefanR5R]

Uhm... the medals have sort of a splatter theme this year:
link to the SG site — not for the faint of heart ;-)

 

[Endgame124]

I know scc noticed the extra processing power recently - I wonder if any of the other projects noticed an uptick in power?

 

[StefanR5R]

I am doing some forum research... Can we get a hit at the overall "Stephan/xii5ku " hardware? It looks a bit crazy if it is not elsewhere used for commercial production of some kind!

I would just like a peak behind the curtain!

Before I discovered Distributed Computing as a hobby, I had two dual-processor computers with Xeon E5-2690 v4. I bought and built them myself and used them occasionally for simulations in my job. Such simulations were ran on an Ethernet cluster of office PCs before. Consolidation of this workload onto the SMP machines made these simulations quite a bit faster.

I posted photos of one of these computers before this forum gained the feature of attachments.

When I got into DC, I was quite interested in comparing performances of a variety of CPUs. But as I participated in one DC contest after another, I got tired of controlling computers with different performance profiles. And so I have lately preferred to build monocultures of several computers of the same type. This has helped me to retain some last shreds of sanity during contests like Pentathlon or Birthday Challenge.

Another thing which changed but already during the time when I started to get into DC, was that I am no longer using my own computers for my job now. (I am an employee, not a business owner.)

At this time I have several dual-processor computers for CPU based DC projects, and some dual-GPU single-processor computers (one triple-GPU computer) for GPU-based DC projects.

The exact total core count is: The same as many of us here have; that is, not as many as we would like to have, but more than a sensible and responsible person would have at home.

Anyway: One key step in my preparations to Birthday Challenge was to have a somewhat homogeneous pool of hardware, rather than a harder to control historically grown variety.

The next step was to figure out which of the WCG subprojects gave best PPD on my computers. Sounds like whoring for credits, but OTOH this also ensures that the WCG project gets the most use out of my computers, since the most fitting application is selected to run on this hardware. Sometimes I would run more or less meticulous measurements to find out PPD of different subprojects or different settings. But in preparation of this challenge I relied on prior knowledge and some simple crosschecks. This approach proved to be completely sufficient to select the optimum application.

Then I had to find out how many boinc client instances were required on each of the computers in order to be able to bunker for 10 days. And I had to "warm up" these client instances by completing an amount of WCG work in each of them, in order to get them recognized as "reliable" hosts by the WCG servers, which is necessary to get a reasonable number of tasks in progress assigned to them later. I combined the steps of determining the required number of client instances and warming them up, and in addition this warm-up gave a little contribution to the TeAm's involvement in the Thor Pure week.

Then a 10 days long routine began of loading some client instances with tasks, completing the work on them, loading more instances as late as possible, but not too late before the former client instances are done, start these next instances when the former are done, and so on ad nauseam.

I loaded new instances as late as possible because I also wanted to be able to unload them as late as possible. I figured that my TeAm mates who bunkered in front of the challenge too would unload the bunkers rather early in the challenge, making for a sharply decelerating production. I have a manic desire for acceleration, especially near the end of a contest. Therefore I loaded late and was able to unload late.

About the loading of a set of bunkers: Putting the respective client instances into the correct state, requesting work, and suspending network transfers as soon as all file downloads were finished, is a process which I automated to a good degree by means of some scripts. I nevertheless performed the loading of bunkers only when I was at home and awake, such that I could intervene if something unforeseen happens, like the internet link being down. This much restricted at which times of day I would perform downloads. Therefore it required some basic planning in how many client instances to fill when.

Filling a set of instances with work took about 50 minutes each time. Therefore, and because I usually had better things to do than to watch the work slowly trickling in, and because I had to use times of days for this when I was not always capable of concentrating on such mechanical procedures, it was crucial that I had automated this mostly, such that it was hard to make mistakes.

Whenever I had loaded more instances, I noted them on a list together with the dates and times at which the results of such a set of bunkers were due to be reported.

Another mechanical step which had to be repeated many times during the multi-day bunkering routine was to start computation in one client instance as soon as a former client instance on the same computer was about to finish all of its queued work. This step is automated by scripts here too. This procedure works reliably enough that it can run unattended for many days.

Finally, the uploading and reporting of results — in waves of each set of client instances which had the same reporting deadline — was supported by some simple control scripts too. I was present to watch this process for the initial two waves, to be sure that it works right. But subsequently I would let this happen unattended, in order to stretch these bunker releases over several hours during which I was asleep or was not at home.

I stretched each of the Tuesday...Saturday uploads over multiple hours (rather than uploading an entire set of buffers in a sudden burst) in order to obfuscate that TeAm AnandTech had bunkers in reserve all the time through the challenge. [Of course, those who know my modus operandi from contests such as Formula Boinc would have seen that my daily output on Monday through Saturday remained in the same ballpark, might perhaps have seen my announcement of 10 days upfront bunkering, have seen Tony's estimation that Sunday could be interesting, and would have put two and two together.] — I unnecessarily stretched the Sunday uploads over several hours too, for effect. ;-) This went on until it was already very late night here, therefore it was good that I had a script doing this while I could watch a movie and later read a book to kill time until the challenge results were up.

BTW, neither was it the first time that I maintained this routine during the entire 17 days before and including a WCG Birthday Challenge. Nor am I the only one who is doing such things. There was once a guy on another team who bunkered for many weeks in front of a SETI Wow!-Event contest, on one or two computers.

Summary: The result which I contributed to this contest was facilitated by 1.) planning, 2.) good kit, 3.) some degree of automation.

PS, another word on my use of server CPUs, and comparably modern ones at that: A starving DCer would normally stick to less costly desktop CPUs or older server CPUs. However, electricity rates are high here, and I have cooling constraints, and limited room for computers. Hence my desire for good perf/W and good density.

 

[VietOZ]

I'll point whoever said "bunker is easy" to @StefanR5R 's post from now on It's easy when just shut the network down and turn it back on.
It takes efforts (setup and manage), planning and good understanding how things work to be able to maximize the damage of a bunker. It's an art, and it's beautiful when we hit everything right.

There are projects allows us to have big or unlimited # of WUs can be downloaded. But there are also project that limited us to a certain # of WUs we can have. Milkyway for example, 900 WUs per instance won't last a Radeon VII much but a couple of hours. I remember one time I had 40+ instances on 5 different machines for Milkyway. Yah, no way in hell i can managed that without some sort of automation. Plus, I'm working 12hrs a day, so I mostly had to remote home to check on my farm every now and then.
Even when we think we had most of everything covered, something else will pop up. Power outages for example. Then we'll have to set the machines to auto power up when power comes back. And then all the "active" instances had to auto start up to continue to crunch ... etc etc.

Anyways, the point is it's easy for one to say "oh it's easy". The fact is ... it takes quite some efforts to nail everything right without suffering much of down time. Also takes a bit of guessing/predicting what other people will do as well. Like most people will upload their bunkers at the first/second hour when the competition starts. Most likely will have a traffic jam. So how to avoid that? Pre-upload your finished instances.

 

[SystemVipers]

I have to agree about what Z said about bunkering, i can' just turn off my network anymore so i have to manually set every system. Plus i never paid much attention to wu's Just grab what i can and go, but i was left too many times with a lot left over. This time i bunkered WCK just because I am Pushing Hard On OP and want to get to at least 50m before years end and still help my team.

So Z showed me a few times how it's done, pretty simple but i undershot it and was just sitting with a lot of cores doing nothing. But i felt good that I did not overcommit. I think my old ways were not helpful to the projects, teams and our competition so that is why i am trying to be a better cruncher, isay more responsible, in every way.

You have built up respect Creds with me over time and I really just wanted to stop in and say Happy ThanksGiving to everyone in the US and to the global pop, Just have a great day or night.

Top Notch Guys.
Steve
SystemViper / XtremeSystems

 

[SystemVipers]

Hi @Icecold you had a great flat earthers shot of the universe.
I want to send it to my family, if you could direct me to it that would be great.

Looks like i got some catching up to do, lots of post from lots of interesting people,
SO let me step back and do some reading

thanks
SV

 

[Icecold]

Happy Thanksgiving Steve / @SystemVipers . I don't remember where I got it from, but this should be the flat earthers perspective of the universe picture you're looking for-

 

[crashtech]

I have to agree about what Z said about bunkering, i can' just turn off my network anymore so i have to manually set every system. Plus i never paid much attention to wu's Just grab what i can and go, but i was left too many times with a lot left over. This time i bunkered WCK just because I am Pushing Hard On OP and want to get to at least 50m before years end and still help my team.

So Z showed me a few times how it's done, pretty simple but i undershot it and was just sitting with a lot of cores doing nothing. But i felt good that I did not overcommit. I think my old ways were not helpful to the projects, teams and our competition so that is why i am trying to be a better cruncher, isay more responsible, in every way.

You have built up respect Creds with me over time and I really just wanted to stop in and say Happy ThanksGiving to everyone in the US and to the global pop, Just have a great day or night.

Top Notch Guys.
Steve
SystemViper / XtremeSystems

Even though your post may mainly be directed at Stefan, I hope you will not mind my reply.

I totally agree about trying to get the proper amount of WUs. I'm still trying to get that right, myself. Getting a little better all the time, but with a full time job it's sometimes hard to devote as much time as I'd like to learning how to better control my clients.

I really appreciate you coming in and writing this post, it's not often these days when adversaries can find common ground, and it's something worth noting and celebrating! We remain rivals, but hopefully in a better sense, with more of the joy of competition that encourages us all to bring our "A" game. Not all of us can win, but the science always does! All the best to you and yours!