Seti had another unexpected power-outage...

[Rattledagger]

February 28, 2005
Around 18:00 UTC we had another unexpected lab-wide power outage. Systems were able to shut down more gracefully than last time, but we are leaving many services off as we survey the damage. The cause of these outages is still unknown.

 

[Dalephi]

Ouch.

Thanks Rattledagger.

 

[amdxborg]

Thanx RD

 

[Fardringle]

I think its the aliens trying to shut down the system before we discover their soap opera broadcasts streaming across deep space. The licensing and royalty costs from that type of network coverage would be staggering! :evil: :laugh:

 

[Specabecca]

It's funny, each time I have difficulty connecting, I think 'maybe today is the day they make us migrate to SETI BOINC!'

You'd think I'd wise up.

 

[Dragonbate]

Again an unexpected outage to SETI due to an unknown cause.... Where's Scully when I need her?!

 

[Unforgiven]

failure and the SETI servers are 2 things that go perfectly together :heart:

 

[kamper]

Originally posted by: Fardringle
I think its the aliens trying to shut down the system before we discover their soap opera broadcasts streaming across deep space. The licensing and royalty costs from that type of network coverage would be staggering! :evil: :laugh:

That's almost exactly what I thought when I read the post (except for the soaps part) We must be getting too close!!

 

[AgaBoogaBoo]

I increased my cache size 2-3 times last time this happened

I'm set for a week or so

 

[Coolkid]

I still have plenty of WU left on my queue

 

[DaFinn]

Bleh... maybe its time to shutdown SETI alltogether. They can't even keep the servers running, how can one expect them to find INTELLIGENT life from mouter space???
Have they not heard of UPS? I've had my company servers running almost non-stop from 1998. Infact I just sold another of our old and trusty IBM netfinity thats been running 24/7 since then.

 

[Rattledagger]

February 28, 2005
Around 18:00 UTC we had another unexpected lab-wide power outage. The cause of these random failures is being investigated by campus, but all our systems/databases survived without any corruption. We will be down as we work to further protect ourselves. More info in Technical News.

February 28, 2005 - 22:30 UTC
So we had another unexpected lab-wide power outage again this morning. This time around we had the BOINC database on battery backup so we were able to shut it down safely. After the power returned we brought the database back up briefly to check it out - and it's in perfect health. You can all thank Court for bringing in his personal UPS (and leaving his own systems unprotected) to put on the BOINC database server until we were able to obtain a new one.

But we shut the BOINC database right back down, and will leave most of the BOINC back-end services off for the time being until we have all our important systems on smart UPS (the systems will shut themselves off once they realize they are on battery power). This has always been the future plan (and please note that our previous configuration allowed for zero or minimal loss in the event of a power failure), but now that frequent random outages are part of the scenario, it would make life easier not to have to do damage control every time.

We are actually going to take this time off to do additional maintenance. For example, the disk array holding the upload/download directories is 98% full - Jeff discovered a bug in the file_deleter code that left a lot of old workunits around. So we need to get rid of those stale files before anything else.

 

[BadThad]

Originally posted by: AgaBoogaBoo
I increased my cache size 2-3 times last time this happened

I'm set for a week or so

ME TOO! lmao

Is it back up yet?

 

[JWMiddleton]

I have a stack of WUs waiting to be sent as well. I am amazed that we are having these problems as this seems to be a new type of problem. Surely they have had UPS units before.

 

[Rattledagger]

They have ups in their server-closet, but the new BOINC db-server is currently in their lab till have shuffled things around in the server-closet to make room for it.

 

[Unforgiven]

2 suggestions for the seti folks....

1) invest in ups backups and rtfm.

2) go to a windows operating system because obviously they dont know what the hell they are doing with their equipment and need something a little more basic that they can get a handle on.

 

[Rattledagger]

March 1, 2005
Power Outage Update: Since the cause of the random power outages is still unknown, we are leaving the data server off during the evenings (users will not get workunits/send results at that time). We can handle power outages during the day while we're at the lab, and are working towards a better system to handle outages at night. Meanwhile, campus is trying to diagnose and fix the problem (which effects the entire building - not just us).

 

[Polo]

Thanks for the infos RD.

175 queued results... and my SETI queue is OK.

 

[Ken g6]

Last updated: 20:05:00 (UTC) on 2005-03-01

Data Server Status

The data server is up and running, but dropping connections

 

[Ken g6]

Getting data - connecting to server.
Receiving data: 10K
Receiving data: 20K
Receiving data: 30K
Receiving data: 40K
Receiving data: 50K
Receiving data: 60K
Receiving data: 70K
...

 

[amdxborg]

Same here! Just slooooooow...

 

[Ken g6]

News

March 1, 2005
Power Outage Update: Since the cause of the random power outages is still unknown, we are leaving the data server off during the evenings (users will not get workunits/send results at that time). We can handle power outages during the day while we're at the lab, and are working towards a better system to handle outages at night. Meanwhile, campus is trying to diagnose and fix the problem (which effects the entire building - not just us).

 

[BadThad]

Originally posted by: Ken_g6

News

March 1, 2005
Power Outage Update: Since the cause of the random power outages is still unknown, we are leaving the data server off during the evenings (users will not get workunits/send results at that time). We can handle power outages during the day while we're at the lab, and are working towards a better system to handle outages at night. Meanwhile, campus is trying to diagnose and fix the problem (which effects the entire building - not just us).

ARRRGGGGGGGG

 

[Soggysocks]

quote:
February 28, 2005 - 22:30 UTC
So we had another unexpected lab-wide power outage again this morning. This time around we had the BOINC database on battery backup so we were able to shut it down safely. After the power returned we brought the database back up briefly to check it out - and it's in perfect health. You can all thank Court for bringing in his personal UPS (and leaving his own systems unprotected) to put on the BOINC database server until we were able to obtain a new one.


If recall......didn't they have a recent shut down for a power upgrade on the building there?


I think I'd find a new Electrical Contractor.

 

[Rattledagger]

They had a power-outage 10. January due to nearby building constructions...