Question Rosetta@home now requires virtual box on ubuntu ?? linux/virtualbox experts needed to get more than 16 tasks !

Markfw

CPU Moderator, VC&G Moderator, Elite Member
Super Moderator
May 16, 2002
22,784
11,194
136
I tried to start Rosetta back up after the WCG 17th Birthday event, but I get the error in red "virtualbox is not installed". So after my google research failed.... I typed:

sudo apt-get install virtualbox and rebooted. Now its working, but says "rosetta python projects (vbox64)"

So the question is, why is it now required, and is it faster or slower ? And what a pain to have to do this on every box that runs Rosetta now.
 

Markfw

CPU Moderator, VC&G Moderator, Elite Member
Super Moderator
May 16, 2002
22,784
11,194
136
But it now only runs 16 threads and my 7742's are almost idle !! WTF ????
 

Icecold

Golden Member
Nov 15, 2004
1,047
946
146
I just tried to load up Rosetta on a machine that doesn't have Virtualbox installed and it won't get any work and shows the 'virtualbox is not installed' in the log entry that you were seeing. I'm going to install Virtualbox and will post back with the results. This thread has some info, it seems like they're out of the previous tasks and only 'python' tasks are available now from what i can tell by skimming it - https://boinc.bakerlab.org/rosetta/forum_thread.php?id=14771#102927
 

biodoc

Diamond Member
Dec 29, 2005
6,043
1,913
136
Check RAM usage. I think the python tasks take something like 8 GB per task. If so, it's a non-starter for me.
 

Markfw

CPU Moderator, VC&G Moderator, Elite Member
Super Moderator
May 16, 2002
22,784
11,194
136
Check RAM usage. I think the python tasks take something like 8 GB per task. If so, it's a non-starter for me.
I have 128 gig, 16 tasks is 8 gig each ?? But its not doing anything, almost idle.
 

Icecold

Golden Member
Nov 15, 2004
1,047
946
146
I'm having the same issue you are Mark. I installed Virtualbox and it does download tasks now, but it's only running 16 tasks on a 256 thread EPYC with 256GB RAM.
 

Markfw

CPU Moderator, VC&G Moderator, Elite Member
Super Moderator
May 16, 2002
22,784
11,194
136
I'm having the same issue you are Mark. I installed Virtualbox and it does download tasks now, but it's only running 16 tasks on a 256 thread EPYC with 256GB RAM.
If its 8 gig per tasks, you should have 32, so something is not right. We need the virtualbox and linux experts... HELP !!!!Edit: on one box with only 110 gig of ram, its using 22.5 gig actual and only running 14 tasks, I still could get at least 64 more tasks and not run out.

Edit: On my dual 7601 box, it has a bunch of computational errors, and is now only running 5 tasks and 10.5 gig of 256 gig memory usage !

Unless some magic happens, Rosetta may be dead for me. I will not let all these CPUs be idle.
 
Last edited:

Icecold

Golden Member
Nov 15, 2004
1,047
946
146
It was set to use only 50% of RAM when computer is in use. I bumped it to 90%, and it's now running 30 threads.
 

Markfw

CPU Moderator, VC&G Moderator, Elite Member
Super Moderator
May 16, 2002
22,784
11,194
136
It was set to use only 50% of RAM when computer is in use. I bumped it to 90%, and it's now running 30 threads.
It should be 250 ! Look at the usage in system monitor, its idle.
 

Icecold

Golden Member
Nov 15, 2004
1,047
946
146
My memory usage is steadily increasing. It was at 41GB about a minute ago, and is up to 46.4GB.

I'm not sure how they could expect people to have 8GB RAM per thread though, I don't have any machines that with a RAM capacity higher than 2GB per thread.
 
Last edited:

Markfw

CPU Moderator, VC&G Moderator, Elite Member
Super Moderator
May 16, 2002
22,784
11,194
136
My memory usage is steadily increasing. It was at 41GB about a minute ago, and is up to 46.4GB.

I'm not sure how they could expect people to have 8GB RAM per thread though, I don't have any machines that with a RAM capacity higher than 2GB per thread.
They are not going to have a lot of contributors if this is the final result.
 

biodoc

Diamond Member
Dec 29, 2005
6,043
1,913
136
I'm running WCG and Rosetta on my 5950X and currently have 7 Rosetta python vbox tasks running. Each task uses 2 cpu threads for a total of 14 threads. The remainder of the available threads are being used by WCG. Highlighting one of the vbox tasks and choosing properties shows a "working set size" of 7.45 GB. I suspect this value is set by the server and is what limits the total number of running tasks but that's speculation on my part. So it looks like I'll need to run both projects to use all the CPU threads available.

Application: rosetta python projects 1.03 (vbox64)
Name: boinc_cages_IL_2727241_55108
Resources: 2 CPUs
Estimated computation size: 80,000 GFLOPs
Virtual memory size: 2.89 GB
Working set size: 7.45 GB
Progress rate: 14.040% per hour
Executable: vboxwrapper_26198_x86_64-pc-linux-gnu


EDIT: 7 rosetta python vbox tasks are using 28.81 GB of disk space! Check to make sure you have enough disk space allocated to boinc.
Screen shot of my 5950 running both WCG and Rosetta.
1637845193652.png
 
Last edited:

StefanR5R

Elite Member
Dec 10, 2016
4,615
5,526
136
The "Virtual memory size" and "Working set size" properties are live data of the running application instance, observed by the boinc client, AFAIK.
 
  • Like
Reactions: biodoc

biodoc

Diamond Member
Dec 29, 2005
6,043
1,913
136
So it looks like I'll need to run both projects to use all the CPU threads available.
This isn't working. My cache is overloaded with rosetta vbox tasks and no new WCG are being downloaded. I guess the solution would be to run 2 boinc instances on the 5950X with one for Rosetta (14 threads) and the other WCG (18 threads). Or maybe messing with the resource share for each project might help.
 

crashtech

Lifer
Jan 4, 2013
10,041
1,749
136
It can be tough to get two projects to play nice within one client, in my experience. I usually don't even bother unless I'm just clearing out some stragglers.
 

Icecold

Golden Member
Nov 15, 2004
1,047
946
146
The answer for now(based on Michael Goetz's post to the Rosetta forum) is to overprovision RAM by using extra client instances since the tasks don't actually need 8GB per task. I was having some major performance issues even running 30 of the Rosetta tasks, though, BOINC manager kept disconnecting from the BOINC client like the client was hung up, etc. so I'm not sure if that will work or not.
 

StefanR5R

Elite Member
Dec 10, 2016
4,615
5,526
136
The answer for now(based on Michael Goetz's post to the Rosetta forum) is to overprovision RAM by using extra client instances since the tasks don't actually need 8GB per task. I was having some major performance issues even running 30 of the Rosetta tasks, though, BOINC manager kept disconnecting from the BOINC client like the client was hung up, etc. so I'm not sure if that will work or not.
Overprovisioning sounds tempting. I learned from @Endgame124 that this is even possible in a single client instance, by configuring >100% allowed RAM usage via an edited global_prefs_override.xml. *However*, based on what I saw at other projects which use vboxwrapper (I haven't tested this new Rosetta application yet), you definitely want to maintain a very responsive system at all times. Otherwise you might frequently see tasks which get stuck with "Postponed: VM job unmanageable, restarting later". Such tasks can then only be aborted, IME, with the respective loss of the task's computation up until that point. (Edit: There may be workarounds to get such tasks back up and running, but they don't seem reliable and feasible to automate.) Therefore I have doubts that overprovisioning of RAM is the way to go with a vboxwrapper based application like this.
 
Last edited:
  • Like
Reactions: Endgame124

Skillz

Senior member
Feb 14, 2014
539
450
136
I'm curious to know how many people have left the project since they implemented this new task requirement.
 

voodoo5_6k

Senior member
Jan 14, 2021
385
429
106
I am leaving. WCG for now.
I'm with @Markfw. I was annoyed by the project official's lack of communication for quite some while anyhow (therefore, only my RPi4 8G was left running Rosetta, but it only occasionally got tasks for several weeks now). But reading through this thread, wow. So, yeah, all-in for WCG on my end too. Until something changes on their end, I don't see much benefit in having resources on that project.
 

crashtech

Lifer
Jan 4, 2013
10,041
1,749
136
Is only classic being tracked? I wonder about the project at large, whether this is having an overall impact.
 

Endgame124

Senior member
Feb 11, 2008
858
543
136
Overprovisioning sounds tempting. I learned from @Endgame124 that this is even possible in a single client instance, by configuring >100% allowed RAM usage via an edited global_prefs_override.xml. *However*, based on what I saw at other projects which use vboxwrapper (I haven't tested this new Rosetta application yet), you definitely want to maintain a very responsive system at all times. Otherwise you might frequently see tasks which get stuck with "Postponed: VM job unmanageable, restarting later". Such tasks can then only be aborted, IME, with the respective loss of the task's computation up until that point. (Edit: There may be workarounds to get such tasks back up and running, but they don't seem reliable and feasible to automate.) Therefore I have doubts that overprovisioning of RAM is the way to go with a vboxwrapper based application like this.
Note that over provisioning ram works best when combined with compressed swap via zram in Linux. It prevents tremendous slowdowns or system crashes by converting the ram dynamically to a compressed swap space in ram. It has a performance penalty, but only kicks in when your tasks exceed physical memory. It used to be the only way to reliable way to get 4 Rosetta tasks running on a 4gb pi.
 

ASK THE COMMUNITY