Two different projects on GPU

[GLeeM]

Reached my DiRT Milestone (50M) and so started other projects on nvidia GPU

Set up Seti for the WoW event and also snagged a few Poem WUs yesterday.
I have them set to run two Seti WUs at a time, or three Poem WUs at a time. When I ran DiRT or Colatz (set to one WU at a time), if I snagged any Poem WUs it would always run three Poem or one of the others.

Yesterday, after a short while I noticed BOINC Manager was running one Seti and one Poem?!? It did that all day. I didn't think that was possible.

Today it is only Seti, unless I Suspend the "Ready to start" Seti WUs. Then a Poem will start.

Can I make it run one of each all the time?

 

[Sunny129]

i'm assuming that when you found your machine crunching only SETI tasks today, they were running two at a time, and not one at a time?

the only way to force BOINC to run one SETI task and one POEM task simultaneously all the time would be to use two physical GPUs. of course with 2 GPUs, you may as well run 4 tasks at a time, and limit POEM to 2 simultaneous tasks like you have w/ SETI, thus forcing BOINC to run 2 tasks from each project, and prevent it from running say 3 from one project and 1 from the other. intuition might also lead one to alternatively use GPU exclude statements in a cc_config.xml file in order to force different applications (from the same project or from different projects) to run on different (separate) GPUs...but i recommend against this unless your two GPUs are of different types (for instance one AMD card and one nVidia card) b/c there is a bug in BOINC that prevents multi-AMD and multi-nvidia GPU machines from maintaining work caches when running more than one project (or more than one application from the same project even).

...of course none of this matters if adding another GPU to your box isn't an option. unfortunately i don't believe there is a way to force BOINC to constantly run one task from both projects simultaneously on just a single GPU. sometimes it'll be 2 SETI tasks running, or 1 SETI and 1 POEM, or 3 POEMS, depending on your projects' assigned resource shares and the amount of work from each project remaining in the queue. on the plus side, it seems to prefer SETI tasks for the time being. besides, i'm sure there will be enough SETI work to go around during the WOW race that you'll be able to simply suspend POEM for the time being and not run out of SETI work for the duration of the race.

 

[GLeeM]

Thanks Sunny129 for the answer about forcing to run two different projects.

...but i recommend against this unless your two GPUs are of different types (for instance one AMD card and one nVidia card) b/c there is a bug in BOINC that prevents multi-AMD and multi-nvidia GPU machines from maintaining work caches when running more than one project (or more than one application from the same project even).

A few days ago I added a 6870 to the rig with a 7950 so I will have to be careful to watch for the quoted.

 

[Sunny129]

A few days ago I added a 6870 to the rig with a 7950 so I will have to be careful to watch for the quoted.

i should elaborate a bit, b/c the bug won't crop up in every scenario where multi-AMD or multi-nvidia GPU machines crunch multiple projects (or multiple applications from a single project). the bug is induced specifically by the use of GPU exclude statements in the cc_config.xml file. in other words, only if you force the first project/application to run on a specific GPU and force the other project/application to run on the other GPU (by way of GPU exclude statements in the cc_config.xml) will the bug show up. but if you don't try to assign certain projects/applications to specific GPUs (and thus allow different combinations of tasks from both projects/applications to run on either GPU at any given time), then the bug will stay away and you'll be able to maintain your work queues for both projects/applications.

to be honest though, the bug really only has a substantial negative affect if the two (or more) projects/applications being crunched don't create new work on a regular basis. in other words, it doesn't matter much that the bug will drain your queue completely before downloading [a minimal amount] of new work so long as the project(s) you're attached to have plenty of new work available every time your queue runs dry.

the bug was reported on the BOINC message boards almost a year ago now, and while the thread has since been locked (i believe due to the fact that more than 90 days have passed since the most recent post), its a good read and explains a number of scenarios in detail...you can find that thread HERE.

 

[salvorhardin]

I think the bug you're talking about has been fixed inthe 7.1.15 beta (It's now up to 7.2.11).

client: fix work-fetch bug that can cause idle GPUs when using exclusions.

Round-robin simulation, among other things, creates a bitmap "sim_excluded_instances" of instances that are idle because of CPU exclusions.
There was a problem in how this was computed; in the situation where there are fewer jobs than GPU instances it could fail to set any bits, so no work fetch would happen.

My solution is a bit of a kludge, but should work in most cases. The long-term solution is to treat GPU instances separately, eliminating the need for GPU exclusions. (David)

I'm not running dual gpu anymore so I can't check it.

 

[Rattledagger]

...of course none of this matters if adding another GPU to your box isn't an option. unfortunately i don't believe there is a way to force BOINC to constantly run one task from both projects simultaneously on just a single GPU. sometimes it'll be 2 SETI tasks running, or 1 SETI and 1 POEM, or 3 POEMS, depending on your projects' assigned resource shares and the amount of work from each project remaining in the queue.

In theory this can easily be done by using app_config.xml for both projects and just specifying <gpu_usage> to be 0.5 and <max_concurrent> to be 1.

But one possible problem is you'll only have work from one of the projects on hand and it would be more efficient to run 2 wu's at once.

 

[Sunny129]

I think the bug you're talking about has been fixed inthe 7.1.15 beta (It's now up to 7.2.11).

holy crap that's great news! i don't know how it never came to my attention given it became available months ago! at any rate, i've got some pretty important Einstein@Home stuff running on dual-nvidia GPU machine right now, but i could probably afford to pull one of the one of the GPUs in my dual-AMD machine away from Milkyway@Home long enough to update BOINC to beta v7.1.15 or later and see if the work fetch bug is indeed fixed. i've been a bit busy lately with wiring my condo for ethernet and relocating one of my machines to another room (which can't be done until the ethernet is in), but i'll test the BOINC beta for the work fetch bug when i can and report back.

 

[Sunny129]

sorry for the off-topic bump, but as an update to my last post, i can confirm that the work fetch bug that plagued machines running different projects on two or more GPUs of the same type (brand) has been resolved. i don't know if the bug was actually fixed in v7.1.15 beta as Salvorhardin suggested above, or if the fixed came in a later revision. one thing's for sure though - i recently upgraded to the latest official release, v7.2.28, and started running both Milkyway@Home and Einstein@Home on the same machine (one on the upper HD 7970, the other on the lower HD 7970), and both projects are maintaining a constant work buffer as they should.

 

[faxon]

Semi-on topic, but my rigs both do something similar when i run einstein@home and milkyway@home without the proper project flags set in einstein@home settings (i have it set not to use the GPU), and it starts running 1 milkyway@home unit and steals one or both of my other GPUs in each rig (either stealing 2 5870s or 2 7950s) and running in tandem with milkyway@home, but on separate GPUs. what i haven't been able to determine though is, does einstein@home have the ability to run on 2 GPUS concurrently with a single task the way that say a single milkyway@home work unit can run on 3 cpu cores simultaneously instead of just running 3 WUs? Everything's cranking away right now but just curious

 

[GLeeM]

I had forgotten about this thread. I too have some further information.

Some projects don't play well together. Einstein WUs seem to not let other project WUs finish.

 

[Sunny129]

Semi-on topic, but my rigs both do something similar when i run einstein@home and milkyway@home without the proper project flags set in einstein@home settings (i have it set not to use the GPU), and it starts running 1 milkyway@home unit and steals one or both of my other GPUs in each rig (either stealing 2 5870s or 2 7950s) and running in tandem with milkyway@home, but on separate GPUs.

i'm not sure i'm understanding the scenario you just described, but could you tell me ideally how would you like the two projects to run side by side? there are most likely ways to make it happen...

what i haven't been able to determine though is, does einstein@home have the ability to run on 2 GPUS concurrently with a single task the way that say a single milkyway@home work unit can run on 3 cpu cores simultaneously instead of just running 3 WUs? Everything's cranking away right now but just curious

i had no idea you could run a single MW@H CPU (n-body) task across multiple CPU cores! then again i only run MW@H GPU tasks, so i guess i wouldn't have known about it. anyways, to my knowledge, no - you cannot run a single Einstein@Home task across multiple GPUs. i don't think it would work like that even if your GPUs were deliberately SLI'd/X-fired. in my experience, crunching with multiple GPUs in an SLI/X-fire configuration is less efficient from a PPD/RAC standpoint than crunching on multiple GPUs not in SLI/X-fire. i understand that one of your machines is also a gaming rig, but i believe you can enable and disable X-fire quickly and easily from the Catalyst Control Center.

 

[faxon]

yea its super easy to switch on and off. as for the scenario going on, i figured it out. WCG was issuing ATI work units and i had it enabled to run them. Einstein@home is also refusing to not send me ATI work units which is getting really annoying because i'd like to run CPU work units for them while my cards crunch but despite the fact that use ATI GPU flag is disabled in my settings for all locations for einstein@home is really annoying, and I don't care enough right now about running it to fix the issue. Also, if you set your CPU core utilization to 100% and you're not running another CPU using project, when it starts the N-Body simulation unit on all your CPU cores at once it prevents you from running more than 1 GPU at a time because there's not enough CPU time to run all 3 cards i guess? i'm honestly not sure what's going on with it but whenever there's not enough CPU time the GPUs stop crunching. I'm finding it more effective to just run GPU only and let the CPU not run tasks because of this constant switching. The annoying thing to boot as well, is that setting it so that the % of CPU time BOINC is allowed to use doesn't help at all because it's a universal pool to the CPU and GPU total. If i could set a flag to prioritize GPU time over CPU time 100% of the time or set separate % for CPU resources dedicated to running tasks vs feeding GPU tasks either one by itself would fix this issue. as it is now the only way would be maybe through config file fixes but im not sure if the flags to fix this type of issue are even in the program

ed: So turns out Einstein@home is cooperating with my settings after all. WCG snuck in under the radar and was running clean energy project WUs (phase 2 6.40) on the CPU and GPU (2 on 2 cpu cores, 1 on each of 2 GPUs, and milkyway@home was only getting 1 GPU). Like e@h but not m@h, it doesn't show when the program is using the GPU for its units, and i hadn't realized i'd enabled it to run on them as well since WCG wasnt issuing GPU units when i started running it.

ed2: they're just switching off lol, einstein@home and wcg both aren't listening to my commands not to use the GPU

 

[GLeeM]

I believe WCG has no GPU WUs at present.

You can free up some CPU time by running one less than max CPUs. In BOINC Manager click "Tools" drop down menu and select "Computing preferences...". On the "processor usage" tab change "On multiprocessor systems, use at most _____ % of the processors." If you have a quad, fill in 75%, if you have a quad with hyperthreading, fill in 87.5%.

 

[Sunny129]

Einstein@home is also refusing to not send me ATI work units which is getting really annoying because i'd like to run CPU work units for them while my cards crunch but despite the fact that use ATI GPU flag is disabled in my settings for all locations for einstein@home is really annoying, and I don't care enough right now about running it to fix the issue.

i encountered the exact same problem, but in a slightly different form. back when one of my hosts was a dual GTX 580 machine, the Einstein@Home web preferences for that specific host were set to accept nVidia GPU tasks only (and not ATI or Intel GPU tasks). then i upgraded that machine's CPU to an intel 3770K (with the integrated HD 4000 IGP enabled so i could run the display on it). despite having the appropriate project flags set via the web preferences, my host started downloading Intel OpenCL tasks. it didn't take me long to remember that i was using a cc_config.xml file w/ the <use_all_gpus>1</use_all_gpus> statement in it, which apparently was overriding my Einstein@Home web preferences. fortunately all i had to do was add the <ignore_intel_dev>0</ignore_intel_dev> statement to my cc_config.xml file and all was good again. for you, the solution will be slightly different b/c you aren't looking for BOINC to ignore a specific GPU altogether, but rather you want a specific project to ignore a specific GPU (or GPUs). what you need to include in your cc_config.xml file is a <exclude_gpu> statement. the basic format is as follows:

<exclude_gpu>
<url>project_URL</url>
[<device_num>N</device_num>]
[<type>NVIDIA|ATI|intel_gpu</type>]
[<app>appname</app>]
</exclude_gpu>

• because your host in question is an ATI GPU-only host (i'm assuming you aren't running integrated graphics of any sort on this machine), you can omit the <type>_</type> line
• because you want Einstein@Home to ignore all ATI GPUs, you can also omit the <device_num>_</device_num> line as well
• you also don't have to specify a project application since there is only one Einstein@Home application for ATI/AMD GPUs, so you can omit the <app>appname</app> line as well

...so your <exclude_gpu> statement should appear as follows:

<exclude_gpu>
<url>http://einstein.phys.uwm.edu/</url>
</exclude_gpu>

that should be enough to force your host to run Einstein@Home on your CPU only.

Also, if you set your CPU core utilization to 100% and you're not running another CPU using project, when it starts the N-Body simulation unit on all your CPU cores at once it prevents you from running more than 1 GPU at a time because there's not enough CPU time to run all 3 cards i guess? i'm honestly not sure what's going on with it but whenever there's not enough CPU time the GPUs stop crunching. I'm finding it more effective to just run GPU only and let the CPU not run tasks because of this constant switching.

you may be onto something there...unfortunately i can't confirm it b/c i don't run Milkyway@Home n-body (CPU) tasks - i only run Separation (GPU) tasks for Milkyway@Home. if you can afford to dedicate an entire CPU and all its cores to servicing multiple GPUs, your GPUs will be better off. to a certain extent you can run CPU tasks as well, but the continued efficiency of your GPU crunching will depend not only on available CPU resources, but also the projects and/or specific project applications you choose to run on the CPU. i don't know how important it is to you to run both CPU and GPU tasks for MW@H, but you might want to try running only MW@H GPU tasks for a while to see if not running n-body tasks helps your situation any.

The annoying thing to boot as well, is that setting it so that the % of CPU time BOINC is allowed to use doesn't help at all because it's a universal pool to the CPU and GPU total. If i could set a flag to prioritize GPU time over CPU time 100% of the time or set separate % for CPU resources dedicated to running tasks vs feeding GPU tasks either one by itself would fix this issue. as it is now the only way would be maybe through config file fixes but im not sure if the flags to fix this type of issue are even in the program

i always have my "% of CPU time" parameter set to 100% b/c my machine is always crunching. but if i'm looking to allocate CPU and GPU resources, i use the parameter just above it - the "% of the processors" parameter. this setting only affects the CPU, and not the GPU(s). for instance, i can limit CPU tasks to use only 75% of the CPU (or 6 threads of my 4C/8T CPU) for the puropse of leaving some CPU available for the GPU(s). if i then look at CPU consumption via the Windows Task Manager, i'll see that in fact more than 75% of the CPU is being used (75% for CPU tasks, plus any extra consumption for the servicing of GPU tasks). so i think you might find the "% of the processors" parameter more useful than the "% of CPU time" parameter in that respect.

*EDIT* - just realized GLeeM hit on this point before i could finish my post...

 

[faxon]

yea i stopped them actually and now i'm rapidly gaining on your RAC. used to run things this way most of the time i believe, but with the cpu set to run seti work units and a % adjust so the GPU tasks always had enough CPU time to hit the benchmark time bracket (1 second error window on either side, I was anal). Right now I have it just running GPU units on both rigs especially since i think the x3 720 BE in my other rig isn't enough to feed those 3 cards and run tasks at the same time. pausing everything else on it is part of why i'm gaining so fast on your RAC now, because while that rig had mostly stabilized out now it's gaining fast again hehe. I am considering switching the entire farm to LTC mining to pay for some upgrades if i don't make the cash for them here soon (get an 8 core drop in for the board the x3 is in and a new case i need), should be enough to feed it and start running some CPU work units again me thinks. As for getting the paramaters correct for % of CPU available to use, I tried this but it didn't work, at least not like it used to. it's possible i didn't allocate enough but this thread won't go anywhere and the info i've collected will get put to use if need be in the future. Right now though it's looking like LTC mining might actually be the only way i can fund this hobby right now. Once I let my RAC cap out and i figure out whats the most optimized setup to run what i might commit part of the farm over to LTC and part to GPU tasks simply because i could exclude one GPU using flags from use entirely, and the LTC miner has to be set per GPU to run. I've already started running it in the background on my 3rd 7950 because I don't need the extra performance, and they just did a major update which improved performance across the board as well, actually making decent money for the time im letting it run (enough to pay for dinner out for two in the last two weeks of gaming while i remembered to run it). Would be a really interesting financial experiment though, trying to self sustain a project like that by committing part of it's resources and time to mining, for those of us who are looking for ways to save or make money while still being able to compute all the time as well in some capacity.

 

[Sunny129]

yeah, now wasn't a very good time to engage you in a Milkyway@Home race to 300 million. you already know about my devotion to the Einstein@Home project. now add to that the fact that i was already in the middle of some major DC experimentation with multiple GPUs in multiple machines, and you can probably imagine how the occasional down-times (while switching and moving around GPUs) and the constant experimenting with multiple configurations between the Milkyway@Home and Einstein@Home projects has brought down my RAC quite a bit.

speaking of which, i just noticed this morning that two of my 7970s are completing Einstein@Home BRP4G tasks in approx ~70 minutes, while my 3rd 7970 is taking ~145 minutes per task (i run these tasks 4 at a time on each GPU). i have no idea why i didn't notice it earlier...it turns out that all 3 GPUs are installed in PCI3 x16 slots, and all are operating at full PCIe bandwidth...but the two 7970s completing tasks in only ~70 minutes are installed in PCIe x16 3.0 slots, while the 3rd is installed in a PCIe x16 2.0 slot! this bothers me, and i may end up pulling the trigger on yet another i7 Ivy Bridge or Haswell CPU and a Z77/Z87 mobo w/ lots of PCIe 3.0 slots and bandwidth to rectify the problem.

 

[faxon]

i think im running into a similar issue just in games performance on my 7950s just because i'm running a sandy bridge board/cpu with PCI-E 2 x8/x4/x4 as the configuration but i can't be sure. seems fine right now but I have the option to upgrade to a Z87 board/cpu through frys because the board and chip i have now are on one of their service contracts and i used to work there so I just tell them what they want to hear and get a store credit for the value of my old parts, and go pick up whatever equivalent part is available now. I also don't want downtime right now so if I do this i'm gonna wait until i have the money for my SSD and HAF Stacker case since my old SSD is still dead and out of warranty and isn't gonna heal itself anytime soon lol.