- Feb 5, 2001
- 2,989
- 18
- 81
It's the Little Things... (Sep 17 2007)
This was a rough weekend - but all due to the collision of a lot of minor things which, by themselves, would have been relatively harmless. Of course, I was sick with a cold all weekend and had rehearsals and shows with three different bands on three different days, so I couldn't do much anyway except check in and point things out to Jeff who dealt with most of it.
Anyway, early in the weekend there were some lost mounts on bruno (our main BOINC administrative server). Why does autofs lose mounts so readily? And why is it unable to get them back? This happens from time to time, with varying effects. In this case it caused various cronjobs to hang, then fill up the process queue, which ultimately brought the machine to a standstill. I discovered this in the evening and told the gang. Dan actually came up to the lab to power cycle the machine which cleared some pipes, but the fallout from this was extensive. Various queues were backlogged and certain backened processes were not restarting.
Upon the reboot of bruno, its RAID volume (which contains all the uploaded results) needed to be resync'ed. Not sure why, but it ate up some CPU/disk I/O for a while and then was fine.
Anyway.. the bruno mishaps caused gowron (workunit file server) to start filling up. I deleted some excess stuff to buy us some time, but there wasn't much we could do except keep a close eye on the volume usage until the whole backend was working again. Meanwhile splitters were stopping prematurely and not restarting (continuing mount problems). And the old mod polarity issue reared its head when we were low on work to send out (you can read more about that in some older threads).
Then, of course, we ran out of work to split. I believe several of our multibeam raw data files are being marked as "done" prematurely due to various issues over the past couple of months. Plus we haven't really had a solid couple of "normal" weeks to get a good feel of our current burn rate. In any case, Jeff got some more raw data on line earlier this morning.
Oh yeah.. we lost a disk on our internal NAS which contains several important volumes, including a subset of our download directories, so that slowed down production for a while as one of thirteen spare drives was pulled in and sync'ed up.
That's basically the gist of it. Back to work.
- Matt
#____Total Work Done____Todays WD_______AWD________overtake________Team-name
01______332.827.300______617.570______587.140______impossible______SETI.USA
02______273.013.956______360.561______340.116______impossible______SETI.Germany
03______116.907.548______213.509______202.602______impossible______L'Alliance Francophone
04______109.930.932_______63.350_______69.713______impossible______BroadbandReports.com Team Starfire
05_______99.568.842______161.330______150.547______impossible______BOINC Synergy
06_______91.553.013_______78.968_______81.550______impossible______Czech National Team
07_______81.667.295______114.726______100.271______impossible______SETI@Netherlands
08_______66.990.085______147.443______147.239______impossible______The Knights Who Say Ni!
09_______44.140.996______-19.622______-12.328______3.581 days______OcUK - Overclockers UK
10_______37.217.780_______55.207_______46.427______impossible______Team Art Bell
11_______36.327.974_______18.786_______18.020______impossible______Overclockers.com
12_______32.716.939_______36.641_______36.549______impossible______Team 2ch
13_______31.559.836______-42.652______-32.579________969 days______BOINC.Italy
14_______27.830.498_______39.861_______41.792______impossible______The Planetary Society
15_______24.496.276_______48.596_______64.685______impossible______Team MacNN
16_______17.392.743_______-4.271_______-4.360______3.989 days______Ars Technica
17________4.704.865______-10.927_______-7.241________650 days______Universe Examiners
18__________225.369_______33.916_______35.276______impossible______SETI@Taiwan
19__________202.206________9.745_______17.457______impossible______Phoenix Rising
20_______58.416.860_______95.216_______84.180______notanoption_____TeAm AnandTech
21_______-5.023.848_______62.002_______60.738_________83 days______Team China
22_______-5.184.934_________-507________9.841________527 days______Team Starfire World BOINC
23_______-9.541.031______-13.814______-11.967______impossible______Hewlett-Packard
24______-10.822.746______-14.238______-10.273______impossible______Amateur Radio Operators
25______-10.907.842_______-1.786_______-1.098______impossible______Dutch Power Cows
26______-10.954.086________3.870________4.426______2.475 days______Canada
27______-11.128.394______-36.832______-29.165______impossible______PC Perspective Killer Frogs
28______-16.023.200______-48.574______-46.767______impossible______Planet 3DNow!
29______-16.082.079______-24.221______-20.780______impossible______Team MacAddict
30______-16.379.592______-42.261______-38.370______impossible______2CPU.com
31______-16.930.529______-31.214______-25.885______impossible______Team NIPPON
32______-18.943.071________3.722________8.812______2.150 days______UK BOINC Team
33______-19.142.129______-11.144______-13.163______impossible______BOINC SETI@home RUSSIA
34______-19.347.752______-17.270______-14.973______impossible______BOINC@Denmark
35______-19.751.321_______15.994_______22.201________890 days______BOINC@AUSTRALIA
36______-19.776.698______-25.635______-18.602______impossible______Hungary
37______-20.606.393______-18.221_______-3.032______impossible______US NAVY
38______-21.913.379______-54.320______-43.443______impossible______Portugal@Home
39______-23.528.344______-13.688_______-4.866______impossible______U.S.Air Force
40______-25.293.102______-40.468______-33.401______impossible______SETI.hr
41______-25.697.348______-79.469______-69.060______impossible______Picard
42______-26.316.868______-36.441______-33.817______impossible______SETI@klamm.de
43______-26.533.445______-75.016______-65.093______impossible______LittleWhiteDog
44______-26.634.175_______31.288_______44.523________598 days______AUSTRIA - NATIONAL - TEAM
45______-27.550.675______-48.714______-44.296______impossible______Team EDGE
46______-27.913.520______-51.290______-41.130______impossible______HispaSeti & BOINC
47______-29.850.369______-46.005______-40.662______impossible______SETI Sverige [Sweden]
48______-30.151.290______-19.826______-13.090______impossible______BOINC.SK
49______-31.771.123______-59.277______-54.043______impossible______BOINC UK
50______-32.427.608______-54.408______-44.602______impossible______PurePC.pl
Appart for Anandtech's stats, it shows how much more/less than Anandtech.
Also shows based on Average Work Done how many days for Anandtech to overtake the team, or be overtaken by a team behind