12-27-2000 Forum Outage

Toggle sidebar Toggle sidebar
A

Anand Lal Shimpi

Boss Emeritus
Staff member
Oct 9, 1999
663
1
0
As you may have realized we had a outage on the forums for a period today. This issue has been tracked down to be a fault with the database server that holds and serves all of the posts that are made in the forums. Unfortunately the error, which was hardware related that took down the forum's database array happened to corrupt the forums database. Luckily we back up the forums every morning at 3:00AM (EST) and thus were able to restore from our backup. Unfortunately any messages posted after then were lost, we apologize for the inconvenience.

What do we plan to do in order to prevent this from happening again? I expect that all of the parts for our new forums database server will be here in the next few days. I will personally build the server and make sure that it is up at our datacenter as soon as physically possible. This server will have twice as much memory, a much better disk I/O subsystem, and 50% faster CPUs than the current database server. Just like the current server, it will only be used for the AnandTech Forums.

I will then pull down the current AnandTech Forums Database server and rebuild it, diagnosing the hardware issues and send it back up as a third webserver for the AnandTech Forums. This will make the Forums even faster and even more reliable for you all.

Again, I do sincerely apologize for this. I will do everything in my power to make sure that the new servers get up and running as soon as possible. You all are very important to us, and I do enjoy browsing the forums as much as everyone else. I'll make sure this works out perfectly so that we don't have to go through those withdrawl symptoms ;)

Take care,
Anand
 
N

networkman

Lifer
Apr 23, 2000
10,436
1
0
Actually, I may be able to help nail down the exact time of the backup as I have a post on my For Sale thread at 3:37am.

 
J

Jason Clark

Diamond Member
Oct 9, 1999
5,497
1
0
There is no exact time the process starts at 3 am.. and it could end at 4 it could end at 3:30.. depending on usage etc. So as anand said the backup started at 3am this morning.
 
V

Viper GTS

Lifer
Oct 13, 1999
38,107
433
136
Does that mean Brandon lost his title?

Or can I re-create my one thread for today?

:D

Viper GTS
 
U

urbantechie

Banned
Jun 28, 2000
5,082
1
0


<< I will personally build the server and make sure that it is up at our datacenter as soon as physically possible >>



Ahh, the boss building the server himself. It better be good then!! :D :p
 
A

Anand Lal Shimpi

Boss Emeritus
Staff member
Oct 9, 1999
663
1
0
ltk007

Hey hey, where's the faith guys? ;) The current server may have been damaged in the move to the new datacenter, once I get it back in my hands I'll be able to figure out exactly what happened to it. It is definitely a hardware issue though.

This new box is gonna rock though, I'll do an article on it too if I have some time.

Take care,
Anand
 
A

Anand Lal Shimpi

Boss Emeritus
Staff member
Oct 9, 1999
663
1
0
Viper GTS

Nope, he still keeps his title, and yep, I think you'll have to repost that topic. Hope your teeth are feeling better bud, I'm almost 100% myself. I had my first burger since the surgery yesterday, mmm... ;)

Take care,
Anand
 
V

Viper GTS

Lifer
Oct 13, 1999
38,107
433
136
You see the problem is my post got locked...

I would have thought it an honor to have my title misinterpreted as &quot;Ass Reporter.&quot;

:confused:

Every forum full of horny geeks needs an official &quot;Ass Reporter&quot; to provide them with their chick pics.

Viper GTS
 
L

ltk007

Banned
Feb 24, 2000
6,209
1
0
Oh me of little faith :p

I'm sure Anand will do a great job, otherwise he'll have almost 40,000 geeks chasing him and throwing various pieces of hardware. You don't wanna get a harddrive in the head, believe me it hurts!
 
H

Hawkeye_(BEL)

Banned
Dec 24, 1999
364
0
0
This is oh so sweet. :cool:

I sure hope you have time to make another article about this new database server, Anand, since the last article about the new webservers was really fun to read. :)



<< This server will have twice as much memory, a much better disk I/O subsystem, and 50% faster CPUs than the current database server. >>



Hmm, let's see... The current database server is a Dual XeonIII 500/1MB, so the new server will have a Dual XeonIII 750/?MB. That's ... impressive. I really love this place. Everything is done to make us happy, thanks again Anand for all your efforts !
 
A

AMD4ME2

Senior member
Jul 25, 2000
664
0
0
We forgive you Anand! and I personally thank you for working so hard for all of us!
I trust your reviews over all others.
(please draw my name next time you give away some hardware!) :D

Oh! BTW... how bout some more case reviews?!?!
 
S

scrubman

Senior member
Jul 6, 2000
696
1
81
thanks Anand!

im glad to hear you are still willing to get your hands dirty! ;)

you da man!
 
A

Anand Lal Shimpi

Boss Emeritus
Staff member
Oct 9, 1999
663
1
0
Viper GTS

Ass Reporter? I must've missed that one. I'll letcha know if there are any openings for an Ass Reporter on the AT staff ;)

ltk007

I'll do my best, hard drives don't hurt as much as taking a rackmount to the head though.

Hawkeye_(BEL)

I'll actually be experimenting with using Coppermines this time around. Judging by the nature of the Forum's DB server I'm going to try going with dual P3-800's. They have less cache however they have higher clocks and I will be using them on a Serverworks board with a higher memory bus clock so that should help as well.

Thanks for the kind words guys, we really appreciate your support. Oh and
AMD4ME2, we'd really like to do more case reviews, and in the next few months, if all goes well, we should be bringing on quite a few more editors. One of them will definitely be assigned the task of reviewing cases ;)

Take care,
Anand
 
S

Soybomb

Diamond Member
Jun 30, 2000
9,506
2
81
Wow, talk about service for us forum users :)
Now just tell us if you lurk on the forums under a different nick ;)
 
D

DABANSHEE

Banned
Dec 8, 1999
2,355
0
0
Not that I know what I'm talking about but would some sort of real time mirroring work better than a daily backup.

Or would mirroring mean that the backup Databass would have the same problem as the 1st one &amp; they'l crash together?
 
A

Anand Lal Shimpi

Boss Emeritus
Staff member
Oct 9, 1999
663
1
0
scrubman

I always get my hands dirty for AT, that's why Jason and I flew up to the datacenter not too long ago to setup the cluster properly. When you're doing something you love, you don't mind getting your hands dirty.

Valhalla1

Not really, it would take a pretty big outage (probably 12+ hours) in order to make a noticeable difference in forum traffic. Even then it would take much more than that to put a dent in traffic for more than a day. This is quite an active board, at any given time we have 800 - 1200 individual users browsing the forums (while only 300 - 400 usually stay logged in).

Soybomb

Now what fun would that be if I told you all about my other... Naw, I wish I had time to post under my real name much less another account. I do read quite a few threads though, I'm on the forums at least once a day.

DABANSHEE

Zuni would be the one to explain the nitty gritty about this, but the ideal situation would be a clustered database setup much like how our webservers are clustered. However, clustering our database backend in addition to our web front end would be a pretty big task. Looking at the future, yes, this will probably happen, but right now a six figure server upgrade investment isn't necessary ;)

Take care,
Anand

 
U

UnixFreak

Platinum Member
Nov 27, 2000
2,008
0
76
Thats cool how you are solving problems like this, no B.S., and a good effort, you should be commended, you run fantastic site, and if all other webmasters thought and acted like you do, the internet would be a better place. keep up the good work.


P.S. I can deal with a problem like this, as well as this site has ran since I got here, this is no big deal. These things happen
 
J

Jason Clark

Diamond Member
Oct 9, 1999
5,497
1
0
DABANSHEE, to do a proper DB Server cluster you are looking at a minimum of $140,000 or so if we build the servers, software licensing is the biggest hit. Micro$oft sure knows how to gouge the pockets with SQL licenses. If you were referring to RAID, the SQL server is currently configured with RAID 5, however with the new box we are moving to 0+1, as RAID 5 is quite a bit slower for the SQL box, as SQL has to wait for both writes to complete as well as the parity check. We are also going to add another SCSI channel for the full text files, to offload that I/O off of the main db channel. Anyway, have a great new year fellas/gals.

Overall the restore went very well, I received notification of the issue at approximately 5:00PM, and the database was back around 7:00PM. About 75% of that time was just the time it takes to restore a 2GB database :). Some people asked if they lost post counts, basically anything after 3:00AM was lost.
 
S

Sukhoi

Elite Member
Dec 5, 1999
15,342
104
106
Ah, I was wondering where today's SETI stats had gone. ;) Thanks for the info Anand. :)
 
I

IBhacknU

Diamond Member
Oct 9, 1999
6,855
0
0
Thanks for the attention to this Anand. These forums, and their database have helped myself out of a jam on countless occasions. It's nice to know you care :)



<< to provide them with their chick pics >>

What's that Viper GTS... you don't like my signature pics?
 
You must log in or register to reply here.

TRENDING THREADS

COMPANY
RESOURCES
FOLLOW
Top Bottom