folding on skulltrail

[DerekWilson]

so i went to bed last night after a frustrating day of benchmarking ... and i decided to set up F@H to run on my test system which happens to be skulltrail with 2x x9775 ...

i don't have a lot of time to look around for answers, but a quick search didn't do me much good and i figured you guys might have the answers i need.

i didn't want to run the smp client as the faq at stanford seemed to indicate that 8 cores would slow it down, so i tried running multiple single core clients. the lonely little console i popped up only sucked down 13% and seemed lonely ...

i properly set 8 consoles with machine ids 1 through 8 pegging my system while i slept ... only i just got up and it seems my machine had hard locked in the middle of the night after about 2 hours of crunching. running the 5.04 clients by the way.

this system is not overclocked and well cooled. i haven't seen any instability from different CPU and stress tests we've done.

so here's my question: is there a problem with running as many clients as there are cores on a box: should i run n-1 consoles? are there any specific problems with 8 core boxes or skulltrail in general? is folding just that stressful (maybe we should bench with it), or did i do something wrong?

thanks.

 

[Insidious]

If your system is set up to save power such as: turning off the hard drives, or going to standby or hybernating, that might do it.

eight instances with eight cores shouldn't be a problem if you have ample RAM.

any time you run multiple instances, you do need to have each instance located in it's own folder and it's own machine ID set (sounds like you did this fine)

make sure when you start each instance, you use the -local switch so the files don't get confused betwen different instances.

-Sid

edit: F@H is VERY intolerant of bad RAM or identifying system instability when people OC. (I read that you are running at stock) Stanford's software will not recover from errors or give you much help in identifying what went wrong.

 

[biodoc]

8 FAH console clients should be fine for 8 cores though I believe 8 is the max.

I'm assuming you have 8 fah directories and the machine ids are ok by your post.

Are you running each with the -local switch? (important for multiple fah clients)

You might as well toss in the -advmethods switch too for those nice double gromacs WUs.

fahxxxx.exe -local -advmethods

When Sid & GLeeM log in they may have additional suggestions too.

 

[Insidious]

Originally posted by: biodoc

When Sid & GLeeM log in they may have additional suggestions too.

:laugh:

(don't look up! )

:laugh:

 

[biodoc]

I knew the master folder would be logging in soon!

 

[Fullmetal Chocobo]

Originally posted by: Insidious

Originally posted by: biodoc

When Sid & GLeeM log in they may have additional suggestions too.

:laugh:

(don't look up! )

:laugh:

Oh, that made me laugh so hard. I don't know if I'm just tired or what, but that was funny.

 

[DerekWilson]

heh ...

i have 4GB RAM and am running vista 64-bit ...

i was not using the -local switch, but i did have each console running from its own folder with its own machine id.

i think for now i'm gonna try running 7 clients with the -local switch and if that works for most of the day i'll fire up the 8th client.

 

[DerekWilson]

also, on some of my consoles i see a line that says :

"Extra SSE boost OK."

i don't see that line on others ...

the ones with this line complete 1% of the WU about every 4 minutes ... the others take considerably longer (maybe 11-12 minutes) for 1%

what's up with that?

 

[Insidious]

That's telling you that it is using the SSE (highly optimized assembly code) instruction set and will make WUs go MUCH faster. (They don't all take advantage of it, but you want it enabled so the ones that can... do)

If you have an improper shutdown, F@H will assume there is a stability problem and on the next client start it will disable the optimized assembly code in the hopes of giving your machine added stability.

(edit: stopping, then) re-starting the client yet again should take you back to the optimized code.

If you want to avoid this, you can use the -forceasm switch. This will make the optimized code run everytime you run the client irregardless of the prior shutdown status.

on ALL of my F@H installations I use the following switches:

-local -forceasm -advmethods -verbosity 9

while they do not always apply (depending on the client type etc..., NONE of them will do any harm so it's easier for me to just always use all of them.

Also, if you want the best work units, you need to be sure to answer YES to the configuration question about allowing WUs larger than 5MB.

-Sid

(The individual folders and -local switch are a MUST for multiple client installations)

 

[legoman666]

Originally posted by: DerekWilson
also, on some of my consoles i see a line that says :

"Extra SSE boost OK."

i don't see that line on others ...

the ones with this line complete 1% of the WU about every 4 minutes ... the others take considerably longer (maybe 11-12 minutes) for 1%

what's up with that?

different WU sizes probably. Download a F@H monitor like FAHmon to see what's going on.

Also, 4 smp clients would absolutely rape 8 single core clients in terms of PPD.

 

[DerekWilson]

how do you "properly" stop the console client? is just hitting the x ok?

i'll go ahead and add forceasm

 

[Foxery]

Originally posted by: legoman666
Also, 4 smp clients would absolutely rape 8 single core clients in terms of PPD.

It sounds like he's using one of Anadtech's test systems - Derek, for reference, the SMP client is buggy and easily misses deadlines if the machine is rebooted or shut off frequently. Unless the boss gave you the system as the coolest "bonus" I've ever seen, the way you're running is a good idea

Looks like everything else has been covered already... definitely stick -forceasm in there to make sure SSE units use it. Also, if you want to keep running F@H for a while, the client windows can also be hidden in various ways. The program is very well-behaved about yielding CPU resources, so the only time you'd need to fully shut it down is when you need to free up the RAM.

 

[Foxery]

Oh, on stopping it... The console client should also respond to Control-C, which will put you back at a DOS prompt rather than losing the window if you wish to change something and fire it back up.

Some of you guys get up entirely too early in the morning!

 

[petrusbroder]

Very seldom ... I am talking about getting up in the mornings ... we are just ahead in time or rather way east of you ...

On topic: Thanks for all the advice I have read - most of them apply to me too.

 

[DerekWilson]

so ....


it locked up again running only 7 clients ... after about 2 hours ...

this is the same amount of time i have my display power down. could that be the issue? otherwise i'm running in high performance mode with the hard drives set to never power down.

also, yeah, this is the test system ... it'll only not be testing things when i'm writing rather than benchmarking (which is what I'm doing for the rest of the day today) or at night. i don't think the smp clients are the right option at this point.

::EDIT::

i noticed that occasionally windows moves each process to another core -- has anyone noticed any performance difference by setting the affinity of each process to a specific core?

 

[Insidious]

Make sure high performance mode options are set for no standby or hybernate.... just in case. Does your motherboard BIOS have options for throttling the CPU or Video? If so, you might want to try disabling those power savers too.

We do mess with affinity when we are running more than one SMP client. (There is an application that does it automatically for us, but it is only for 2 SMP clients on a Quad)
For the CLI clients you are running setting affinity might help you some (reduces windows overhead of thread assignments, but I don't think it would be a big difference)

finally, CHECK your RAM sticks!!!! You wouldn't believe how many of us found we had failing ram by folding. (you have to run memtest a LONG time to be sure)

-Sid

 

[GLeeM]

Originally posted by: DerekWilson
it locked up again running only 7 clients ... after about 2 hours ...

this is the same amount of time i have my display power down. could that be the issue?

Can't be coincidence, can it?

Maybe you found a bug when running vista 64-bit and F@H?

Try setting monitor shutdown to something different and see if it still happens!

Originally posted by: DerekWilson
is folding just that stressful (maybe we should bench with it), or did i do something wrong?

Some Work Units (there are many different types) stress different parts of the system.

Many here know that a system tested with Prime95 or other stressing applications IS NOT Folding@Home stable. But because there are so many different WUs there is no guarentee you will get the type that really tests a computer. There is a GROMACS stress tester, I think.

 

[DerekWilson]

quick update -- still writing so i haven't had a lot of time to play with this.

setting monitor power down to never didn't help (using modified high perf mode with no hard drive power down or monitor power down now).

running 3 clients with affinity set worked for 6 hours ... trying 4 clients with affinity set ...

interestingly, before i set affinity on my 4 running clients (after the system was up for about 30 minutes), cpu usage was at ~65% rather than 50% ... and i run processidletasks quite a lot due to benchmarking, so i'm not really sure what's up with that. after i set affinity, cpu usage has held constant at 50% with half my cores pegged (which is different when i don't set affinity -- it seems like sometimes windows moves the threads faster so that they aren't on a single core for more than fractions of a second. sometimes it seems to work differently where they'll sit on one core for maybe 5 seconds each and then move over somewhere else.

it's kinda odd.

i'll keep adding clients tonight every couple hours and see if i can get a full load running if i set affinity.

 

[VirtualLarry]

SkullTrail unstable... news at 11.

 

[GLeeM]

Originally posted by: GLeeM
There is a GROMACS stress tester, I think.

Found it, StressCPU v2

 

[DerekWilson]

it ran fine for a while with 4 clients ... after about an hour of running 5 clients all with their affinity set to specific cores, the system died again.
i'll test and see if it'll run 4 clients all night tonight.

 

[caferace]

Derek,

Most of us (if not all) run 2x SMP clients with out Intel Quads with Affinity Changer. That being said, Affinity Changer hasn't helped in the past with AMD dual-core CPU's and has even shown a reduction in productivity.

And that being said, you're in totally uncharted territory with the Skulltrail box, as I've not seen anyone talking about it, even at FoldingForum.org.

On top of all that, FAH is certainly not for the weak-of-kneed on SMP machines for optimized crunching. Props to you for sticking with it, as I know my early frustrations occasionally made me re-think multicore FAH.

:thumbsup:

-jim

 

[caferace]

p.s.... I'm assuming you're crunching for TeAm Anandtech (#198) and if so, what's you FAH UID? We can follow along in the sTAtS!

-jim

edit: please delete. :laugh:

 

[petrusbroder]

No, Jim, we are not doing a rerun of the "please delete" - thread - it was way too much writing, work, stats and so on!

But it was a lot of fun ...

OnTopic: I am following this discussion because I am contemplating a similar rig.

 

[biodoc]

Hmmm...can't stop thinking about this!

In your config did you choose yes for "use internet explorer settings"?

If so, rerun config and choose no. IE has additional security that changes with updates and has caused problems with fah when communicating via port 8080.

Also I read somewhere that on Vista, the client should be run as administrator and do do so right click on fah executable, choose properties, click on "compatibility" and click the run as admin box.

I still don't understand why it runs 4 cores fine but 4+ is a problem.

You may want to post this issue over at the fah forum to see if they can help.

cheers!:beer:

PS: 4 fine, 4+ problem could be consistent with Sid's point about a bad stick of RAM.