Rattledagger
Elite Member
Rags and Bones (Oct 27 2008)
Bit of a weird weekend. Towards the end of last week we had some science database issues - apparently informix "runs out of threads" and needs to be restarted every so often. Around this time there were continuing mount problems on various servers. The usual drill. Then I headed to San Diego for a gig (only gone 28 hours) and Jeff went on a backpacking trip.
Things were more or less working in our absence, but - as it happens sometimes - sendmail stopped working on bruno. This wouldn't be a tragedy except for the fact that bruno wasn't able to send us the usual complement of alerts. For example: "the mysql replica isn't running!" So we didn't realize the replica was clogged all weekend. The obvious effect of this is our stats pages have flatlined. It's catching up now, but we'll probably just reload it from scratch during the outage tomorrow.
We also had more air conditioning problems last night. At least the repairguy returned today with replacement parts in tow. So that's being addressed, but not after Jeff got the alarm at midnight last night and Dan trudged up to the lab to open the closet doors and let things cool off. And the httpd process on bruno, once again, crapped out at random - meaning uploads weren't happening for a short while there. Jeff gave that a swift kick, too.
On the bright side, we're discovering ways to tweak NFS which have been vastly improving efficiency/reliability here in the backend. This may help most of the chronic problems like the ones depicted above.
- Matt
Stocktaking (Oct 28 2008)
Today's outage took a little longer than usual. This had mostly to do with the replica mysql database needing to be reloaded from scratch (since it fell behind over the weekend and would take days to catch up otherwise). Plus there was some more index manipulation, en route to a (slightly) more streamlined mysql database. I also replaced the drive that failed on bambi a week ago. So you can stop worrying about that.
Jeff and I spent way too much time fighting with our current raw data pipeline. We get SATA drives up from Arecibo full of data. What happens to this data is a matter of priorities. Do we need to send empty drives back down to Arecibo as soon as possible? Is the splitter data queue low? Is the raw data storage full? Etc. etc. So at any given time we're been either (a) sending data to our offsite archival storage or (b) moving data over to the raw data storage, or (c) both of the above. We're not here 24/7 so to ensure continual data flow we have external SATA drive enclosure on a couple systems.
However, due to various annoying mechanical/form factor reasons, very few of our systems can host these enclosures. Also the drives should be swappable (otherwise what's the point?) but we're finding that very frequently a drive is pulled, another is put it to be read, and the OS can't see the new drive until we reboot the system. This has been a problem with the enclosure directly connected to a SATA card, or via a SATA to USB converter. We're trying to automate this whole process, but with the drives/enclosures constantly disappearing for no good reason we're up against a wall on this.
- Matt
#____Total Work Done____Todays WD_______AWD________overtake________Team-name
01______687.935.337______778.703______759.532______impossible______SETI.USA
02______492.431.426______630.107______605.043______impossible______SETI.Germany
03______216.510.146______221.073______230.781______impossible______L'Alliance Francophone
04______131.646.689______144.714______146.555______impossible______The Knights Who Say Ni!
05______129.373.477______-23.040______-12.700______10.187 days______BOINC Synergy
06______123.310.274______169.749______153.804______impossible______SETI@Netherlands
07______121.706.206_______80.902______103.065______impossible______Czech National Team
08______118.480.995_______15.636_______22.362______impossible______BroadbandReports.com Team Starfire
09_______71.958.119_______51.967_______55.208______impossible______Overclockers.com
10_______35.012.204_______11.270_______15.914______impossible______Team 2ch
11_______23.550.480_____-159.782_____-139.396________169 days______Team Art Bell
12_______23.497.009_____-112.630_____-103.454________227 days______Team MacNN
13_______20.513.488______-96.595______-70.670________290 days______The Planetary Society
14_______18.977.995_____-143.252_____-126.500________150 days______OcUK - Overclockers UK
15________4.529.713______-62.956______-47.438_________95 days______Team Starfire World BOINC
16________3.091.708______-20.537_______-9.044________342 days______SETI@Taiwan
17______149.219.180______339.933______313.859______notanoption_____TeAm AnandTech
18_________-168.859______-43.363______-45.229______impossible______Team China
19______-23.196.116_____-215.601_____-199.085______impossible______BOINC.Italy
20______-23.991.881_____-201.793_____-186.113______impossible______Ars Technica
21______-26.816.495______-49.812______-39.785______impossible______US NAVY
22______-31.698.965_____-211.318_____-189.531______impossible______Phoenix Rising
23______-34.024.879_____-116.312_____-102.898______impossible______Canada
24______-38.532.314_____-103.645______-83.372______impossible______Amateur Radio Operators
25______-38.991.999_____-109.095______-96.451______impossible______U.S.Air Force
26______-41.946.343_____-218.678_____-200.762______impossible______Universe Examiners
27______-45.043.538_____-171.081_____-152.136______impossible______UK BOINC Team
28______-45.608.887_____-167.609_____-150.798______impossible______Dutch Power Cows
29______-47.920.802_____-114.544_____-113.906______impossible______BOINC@AUSTRALIA
30______-49.445.550_____-192.107_____-173.822______impossible______AUSTRIA - NATIONAL - TEAM
31______-49.818.444_____-190.965_____-184.069______impossible______PC Perspective Killer Frogs
32______-55.518.723_____-181.018_____-146.164______impossible______BOINC SETI@home RUSSIA
33______-60.185.335_____-198.426_____-175.282______impossible______Team NIPPON
34______-63.330.312_____-226.068_____-180.001______impossible______BOINC@Denmark
35______-65.027.087_____-102.093______-95.724______impossible______BOINC@Poland
36______-67.855.322_____-224.714_____-200.203______impossible______Hungary
37______-71.491.239_____-275.140_____-251.032______impossible______Hewlett-Packard
38______-73.826.304_____-258.887_____-237.771______impossible______Team MacAddict
39______-74.218.436_____-150.827_____-131.172______impossible______Elite Games
40______-74.790.133_____-214.028_____-195.713______impossible______Planet 3DNow!
41______-83.721.245_____-266.734_____-248.520______impossible______2CPU.com
42______-84.507.134_____-236.466_____-215.711______impossible______Portugal@Home
43______-85.135.012_____-273.008_____-244.676______impossible______SETI@klamm.de
44______-85.257.206_____-266.064_____-237.134______impossible______SETI.hr
45______-90.069.318_____-117.975_____-120.563______impossible______Boone Community School District - Iowa
46______-90.851.692_____-244.065_____-225.670______impossible______BOINC.SK
47______-91.837.616_____-261.417_____-235.373______impossible______SETI Sverige [Sweden]
48______-92.457.723_____-278.716_____-249.832______impossible______BOINC.BE
49______-94.193.346_____-270.980_____-245.744______impossible______Team EDGE
50______-94.987.161_____-283.810_____-257.174______impossible______HispaSeti & BOINC
Appart for Anandtech's stats, it shows how much more/less than Anandtech.
Also shows based on Average Work Done how many days for Anandtech to overtake the team, or be overtaken by a team behind...