D2OL: Whoops!

[TAandy]

Read this on the D2OL forum here
Someones not going to be popular, and AnandTech are now 16th in the top teams.

"Yes, stats blew up completely and the only way I could resurrect it was to zero it. While I am certain I will be burned in effigy, can we contain it to a few threads?

Here's the scoop.

We are in a beautiful position on the waterfront. That, however, comes at a cost of significant power fluctuations. We have a generator. It is overloaded because man, do freezers take a lot of power and god help us if the lab samples thaw. However, the problem is PEAK load, the moment those compressors kick on is something fierce.

Much of the IT is not considered mission critical - as in, it MUST be up 24x7. The word MUST applies more to things like banking and the insurance industry, not my website. Yes, we really WANT it up all the time but if we lose power, I have many more issues on my hand. We are a single site with no co-located servers. Such is the infrastructure.

That is not to say that we don't have a UPS or seven here on site. Every server is hooked up to some monster batteries and some batteries are even on the generator for things like Mail. That just takes way too long to reboot so it is given special dispensation.

Now here's the unfortunate truth. I'm getting a new, bigger generator because I won the argument that these things ARE delicate and should NEVER shutdown unless I am there to hold its hand. I even got brand new UPS's that need to be hard wired into the electrical system. But all this happens in a special order. I first need to change the electrical panel to make room for the transfer switch and this will happen on the 28th since I need to shut off power to the whole building. In the mean time, project servers remain on UPS power with shut down commands as always.

Well, wouldn't you know it - 2 weeks before they are to be given brand spanking new batteries, the runtime on the current ones has degraded to the point that I can NOT perform a graceful shutdown.

We had a power outage. Shutdowns were less than graceful. Power was lost during the backup.

I believe completely in 2 laws of the universe. Gravity and Murphy's.

--------------------
Charles Beckius
D20L Forum Administrator"

 

[amdxborg]

Wow that sux! Well there has to be record of stats on 3rd party stats site that he can use to put all the teams back where they were..?

 

[GeoffS]

I have user and team daily and total data from the date I went live with D2OL, but only node data for people who had 'registered' their nodes on my stats site to track them... dunno what his stats drive off, so I don't know if they are of any use...

 

[TAandy]

Originally posted by: GeoffS
I have user and team daily and total data from the date I went live with D2OL, but only node data for people who had 'registered' their nodes on my stats site to track them... dunno what his stats drive off, so I don't know if they are of any use...

See you've already made the offer 🙂
I wonder how safe all the results they've had returned to them are?
I'm just glad I'm out of it :roll:

Edit:
"I am unfamiliar with exactly what people download - is it a scrape from the website or the actual binary files from the stat processor? (67.98.89.132) I have no way to repackage the data into a binary - but if that's what someone's got...

--------------------
Charles Beckius
D20L Forum Administrator"

Edit again
"every 15 min, the stats prcessor runs it's job (unless it's already running).
The output of this job is:
candidates.dat
currentDaily.dat
dailyStats.dat
general.dat
lastUpdated.dat
members.dat
teams.dat
nodes.dat
These live natively on the stat server and it is these binaries that are taken and used by another job to CREATE the jsp's on the website. At this time, there is no job that can go backwards ... these are serialized java objects, not delimited files.

--------------------
Charles Beckius
D20L Forum Administrator"

 

[networkman]

Unbelievable. :roll:

Someone needs to go back to Tape Backup School and re-take the 101 course.

 

[RaySun2Be]

Originally posted by: networkman
Unbelievable. :roll:

Someone needs to go back to Tape Backup School and re-take the 101 course.

No joke. I do tape & disk backups daily. Rotating tapes. How do you lose all stats when one backup fails unless you are reusing the one and only tape or something. 😕

 

[BlackMountainCow]

Ah well, I'm glad I left D2OL long time ago and have my selfmade cert saved on my HD.

But indeed, how can that happen to such an enterprise? I mean, it's not like there aren't things like tape backup, external HDs and RAID systems ... :roll:

 

[Coquito]

Hold on, let me get the giant hampser wheel out of the closet.

All better. 🙂

Sorry.

 

[GeoffS]

Originally posted by: Coquito
Hold on, let me get the giant hampser wheel out of the closet.

All better. 🙂

Sorry.

Whoa... you have a giant hampster? 😉

 

[Coquito]

Originally posted by: GeoffS

Originally posted by: Coquito
Hold on, let me get the giant hampser wheel out of the closet.

All better. 🙂

Sorry.

Whoa... you have a giant hampster? 😉

That's where Andy's genetically engineered lab rats/college interns come in. Don't ask why I had the wheel in the first place. 🙂

 

[Freewolf]

Originally posted by: Coquito

Originally posted by: GeoffS

Originally posted by: Coquito
Hold on, let me get the giant hampser wheel out of the closet.

All better. 🙂

Sorry.

Whoa... you have a giant hampster? 😉

That's where Andy's genetically engineered lab rats/college interns come in. Don't ask why I had the wheel in the first place. 🙂

Are you sure you aren't in a cage?
😉

 

[Wolfsraider]

LOL so thats where all the food I send you goes!!! :evil:

 

[amdxborg]

lol 😀

 

[Coquito]

Originally posted by: Wolfsraider
LOL so thats where all the food I send you goes!!! :evil:

I still have some icelandic dry rack aged shark hidden in my left cheek.

...

I don't have an overbite, don't even try. 😛

Originally posted by: amdxborg
lol 😀

Quiet you! After the gerbils, the penguins are going in next.