Technical News and SETI@home daily stats for 06.10.2007.

[Rattledagger]

upload issues and other news (Oct 5, 2007)

Matt is still away on his well deserved vacation so I will summarize the week.

Last weekend we had 3 servers go down, as Eric described in the previous tech note. Two of these were attached to a UPS that malfunctioned. Not good, but at least we understand what happened. The third machine, bruno, crashes every week or two and hangs on reboot for reasons we have yet to understand. Our best guess at this time is that the fiber connection to the disk array that holds the upload directory is sometimes throwing garbage onto the bus that the machine cannot gracefully handle. This is an old fiber array that we would like to phase out anyway, so we thought about different storage devices that we currently have that could hold the uploads. We came up with the underutilized disk space on the master science database machine, thumper. This could have the added benefit of hosting the assimilators on the same machine that hosts the back end science database. Eric ran a script that gradually migrated the uploads over to thumper.

This worked fine until the migration reach a critical point, at which time the loads on the two download machines shot up to the 80-100 range (they are usually at 5 or less). The high loads were because each instance of the file_upload_handler was taking a long time to write the uploaded results over to thumper. To make a long story short, it turns out that the volume on thumper that held the new upload directory was getting slammed by the uploads. It was running at nearly 100% utilization (local disk, not network, utilization). This was, and still is, a bit surprising. The volume on bruno is software RAID50 and on thumper the volume is software RAID5, the latter having 2 more spindles than each of the RAID50 mirrors on bruno. At any rate, we are migrating back to the fiber array on bruno and have already seen download performance normalize. We'll have to figure this one out...

The other systems news of the week involves database replication on both of our production databases. The seti_boinc database (users, hosts, teams, recent results) replica was lost to a machine crash. We restored from the master and the replica is once again running normally. We are getting very close to having a replica of the back end science database. The initial data load is nearly complete. We will turn on replication either over the weekend or early next week.

Over in science development we are getting the splitter ready to handle the radar blanking signal that will be embedded in all new data once Arecibo comes back on line later this month.

-- Jeff

#____Total Work Done____Todays WD_______AWD________overtake________Team-name
01______342.309.283______545.436______514.004______impossible______SETI.USA
02______279.455.176______362.421______343.887______impossible______SETI.Germany
03______120.676.663______212.052______201.367______impossible______L'Alliance Francophone
04______111.051.245_______59.851_______62.943______impossible______BroadbandReports.com Team Starfire
05______102.265.402______118.269______141.510______impossible______BOINC Synergy
06_______92.932.499_______81.529_______76.479______impossible______Czech National Team
07_______83.512.646_______92.403_______96.874______impossible______SETI@Netherlands
08_______69.675.809______137.388______143.281______impossible______The Knights Who Say Ni!
09_______43.826.913______-34.393______-16.708______2.623 days______OcUK - Overclockers UK
10_______38.119.971_______44.357_______47.081______impossible______Team Art Bell
11_______36.826.325_______32.177_______27.286______impossible______Overclockers.com
12_______33.362.033_______26.149_______34.245______impossible______Team 2ch
13_______30.901.904______-54.964______-36.176________854 days______BOINC.Italy
14_______28.619.157_______41.717_______42.554______impossible______The Planetary Society
15_______25.732.644_______49.883_______64.526______impossible______Team MacNN
16_______17.265.667______-15.316_______-7.439______2.321 days______Ars Technica
17________4.409.258______-34.324______-16.677________264 days______Universe Examiners
18__________800.160_______19.717_______31.093______impossible______SETI@Taiwan
19__________447.987_______-7.865_______12.268______impossible______Phoenix Rising
20_______60.135.671______117.415_______92.595______notanoption_____TeAm AnandTech
21_______-3.989.957_______38.820_______51.537_________77 days______Team China
22_______-5.140.731______-10.762________2.696______1.907 days______Team Starfire World BOINC
23_______-9.883.754______-32.119______-18.183______impossible______Hewlett-Packard
24______-10.916.918_______-8.245________1.691______6.456 days______Canada
25______-11.066.573______-31.452_______-9.251______impossible______Dutch Power Cows
26______-11.084.675______-22.788______-14.750______impossible______Amateur Radio Operators
27______-11.734.212______-47.242______-32.450______impossible______PC Perspective Killer Frogs
28______-16.571.285______-33.839______-26.353______impossible______Team MacAddict
29______-17.223.400______-91.232______-63.610______impossible______Planet 3DNow!
30______-17.262.065______-76.515______-48.700______impossible______2CPU.com
31______-17.497.579______-47.008______-30.987______impossible______Team NIPPON
32______-18.979.556______-21.730_______-2.784______impossible______UK BOINC Team
33______-19.372.628______-25.720______-13.049______impossible______BOINC SETI@home RUSSIA
34______-19.727.667______-35.657______-20.994______impossible______BOINC@Denmark
35______-19.775.241______-43.681_______-3.313______impossible______BOINC@AUSTRALIA
36______-20.206.466______-36.025______-22.947______impossible______Hungary
37______-20.785.618________6.036_______-6.601______impossible______US NAVY
38______-22.824.106______-65.042______-47.862______impossible______Portugal@Home
39______-23.678.032______-20.749_______-9.196______impossible______U.S.Air Force
40______-25.476.912_______52.931_______59.271________430 days______AUSTRIA - NATIONAL - TEAM
41______-25.967.112______-54.910______-37.101______impossible______SETI.hr
42______-27.065.721______-54.886______-40.294______impossible______SETI@klamm.de
43______-27.123.833_____-101.665______-76.881______impossible______Picard
44______-27.860.507______-94.564______-71.762______impossible______LittleWhiteDog
45______-28.486.540______-72.312______-50.807______impossible______Team EDGE
46______-28.835.428______-67.681______-49.134______impossible______HispaSeti & BOINC
47______-30.634.309______-56.715______-26.750______impossible______BOINC.SK
48______-30.695.119______-63.151______-45.519______impossible______SETI Sverige [Sweden]
49______-32.921.758______-78.622______-61.167______impossible______BOINC UK
50______-33.427.279______-72.969______-53.564______impossible______PurePC.pl

Appart for Anandtech's stats, it shows how much more/less than Anandtech.
Also shows based on Average Work Done how many days for Anandtech to overtake the team, or be overtaken by a team behind

 

[petrusbroder]

Thanks for the stats, Rattledagger! :beer: :sun:

The production is very nice. Could it be that the effort of TAS starts to show up? :Q
I'll do some TAS-specific stats later tonight and report here.

 

[Assimilator1]

Sweet output