• We’re currently investigating an issue related to the forum theme and styling that is impacting page layout and visual formatting. The problem has been identified, and we are actively working on a resolution. There is no impact to user data or functionality, this is strictly a front-end display issue. We’ll post an update once the fix has been deployed. Thanks for your patience while we get this sorted.

SETI@home power-outage due to blown breaker.

R

Rattledagger

Elite Member
From SETI@home for BOINC technical news:
February 23, 2005 - 23:30 UTC
A sudden, unexpected power outage due to a blown breaker shut the whole BOINC project down for several hours (along with all the other projects in the lab). The cause is still unknown, so there will be a scheduled power outage in the near future to hunt for electrical problems. We do know this: we just can't seem to catch a break around here.

We are checking all tables in the database before restarting the data servers. We were able to gracefully shut down many servers on battery backup (UPS) before the batteries drained, but not all of them.

Earlier this morning the project was off for some routine maintenance (tweaking the BIOS on the database server to get rid of spurious error messages and snapshotting for database backups). An hour after we brought everything back up the power went off.
Click to expand...
 
"The cause is still unknown"

I wonder if it is somebody (thing?) that doesn't want to be found that is sabotaging the project. JK 😉
 
An update when it comes to the BOINC-part, no idea if there's similar problems for "classic"...
We were able to gracefully shut down many servers on battery backup (UPS) before the batteries drained, but not all of them, including the new BOINC database server. So the data is scrambled, and mysql refuses to start. Our last backup to tape is a week old. This week's tape backup was about 60% finished when the power went out (Murphy's law in a nutshell).

The good news is we have a replica database which should be up to date. The bad news is that this had disk errors upon booting up and its drives are still resync'ing. After that, we'll have to check the table integrity on the replica - if we're lucky and mysql is able to start, we can then dump the data from the replica back onto the master and continue right where we left off.
Click to expand...
 
I run an internal SETIQ at work, as of this morning I still have enough WUs to last 8.5 days. (I've been through some lengthy Berkeley outages before. 😉)

And I point my home herd to a TeAm public SETIQ, although my Internet connection is down right now. 🙁

I do run SETIDRIVER on all but 1 PC, so the home herd should last a couple more days at least, hopefully by then my Internet connection will be back up. 🙂
 
FYI, the rain here has calmed down, at least for the next couple to few days, so that should help any other straining situations they might be having.
 
An update from seti "classic":
February 24, 2005
UPDATE: Yesterday afternoon a breaker blew and power for the entire lab went down. We came back on-line three hours later, but due to servers shutting down ungracefully we had to check every table in every database. This process took all evening and still continues as of now (17:30 UTC). When these checks complete we can start the data server, web server stats, etc.
Click to expand...
 
I have plenty of WU's on CD from a couple of months ago, feel free to use my queue 🙂
http://coolkid.no-ip.com:5517
1500/256 connection, so it should be able to handle a few people 😉
Just send me a PM if you are having trouble downloading from it and ill move some WU's into your queue.
 
Nooooooooooo, I still can't connect and I'm all outta WU's!!! Can't send or receive them.
I get:
Getting data - connecting to server
connect: No such file or directory

WTF is going on!
 
Another update from SETI for BOINC technical news:
February 24, 2005 - 23:30 UTC
Update on yesterday's outage: We are still dealing with some database fallout. Most of the classic SETI@home systems are up - enough that we can serve workunits to users. However, BOINC is dead in the water until we get at least one database server up and running.

With the master database corrupted beyond repair, we turned all our attention to the replica. Its disks finished sync'ing last night, and after some file system checks the machine booted and mysql started just fine. A battery of tests revealed no corruption.. until we got to the result table. Of course, that's by far the biggest and most important table in the database. We are attempting to repair it now.

Assuming we can repair it with little or no data loss, we will then dump all the data from the replica back onto the master. If we're lucky, this will be done by tomorrow morning and we can start revving all the engines back up.

Please note that since it was a slower machine than the master, the data on the replica database server was about 30 minutes behind real time. We did try to limp both systems along to sync the replica data up even further but no dice. So, when we do get back on line it will be as if there was a half-hour hole in time during which all uploaded results were lost (and any user profile updates, message board postings, etc.). We sincerely apologize to all our users for this loss.

Court brought in a UPS from his personal server collection. So the master database will be protected while we scramble to purchase another. The database server was unprotected yesterday because is was in our lab, not in the data closet where all of our UPS's are. We were/are just weeks away from a data closet reorganization designed to make room for the DB server.
Click to expand...
 
Originally posted by: amdxborg
Wow what an operation! Seems like classic is dead again... Hope they get everything up soooon!!!
Click to expand...


Yea I was good last night- but today I can't sign on. It keeps propmting me to eter my email address of create a new account.
 
Man they've had some bad luck there!🙁

Originally posted by: MDE
Seems like there's a new SETI outage every week...
Click to expand...

Not quite ,but their have been more lately ,mostly down to the migratiotion to S@H2

Originally posted by: DaFinn
Dayum, hurryy... running out of WUs real quick...
Click to expand...

Already!??😛😉

 
Latest from SETI@home for BOINC home-page and technical news:
February 25, 2005
The database has been restored and we are bringing up services. More info in Technical News
Click to expand...
February 25, 2005 - 20:00 UTC
The database has been restored with a loss of the most recent 1/2 hour of processing just before the crash. Credit gained during that short period is lost and some folks may see transient download problems.
Click to expand...
 
From SETI@home for BOINC:

February 25, 2005
The database has been restored and all services are back up. The data server is very busy right now. There will be upload and download problems until the load normalizes.
Click to expand...
 
You must log in or register to reply here.

TRENDING THREADS

Back
Top