Unexpected Forum Downtime

[jpishgar]

Hey there guys,

We're bumping into some unexpected downtime and sporadic outages today on the forum with AWS. Our Operations team is working with the network center guys to get this sorted, and hopefully it won't be too long before things have re-stabilized and are steady again.

We apologize for any temporary interruption of your usual forum activities. Will keep you updated as we work towards resolving the issues.

Yours,
Joe Pishgar
Senior Community Manager, Purch

 

[cabri]

You may wish to post this as a headline banner

 

[KeithTalent]

You may wish to post this as a headline banner

Done.

KT

 

[balloonshark]

And for the record I was getting those white screens mentioned in the other topic. It was the first time I've seen the issue.

 

[Ken g6]

I saw weird things going on yesterday and today too. Today I'd click a link and nothing would happen.

WGet returned this error message, if it helps:

2015-04-29 08:04:31 ERROR 503: Service Unavailable: Back-end server is at capacity.

 

[jpishgar]

Things appear to be stabilizing, but we're not totally certain about the fix, and the forums could potentially go down again. The teams are still working to find a solution to the issue.

-JP

 

[JFP]

Turns out that we've accidentally discovered a bug in Amazon RDS. We're working with Amazon support to get the issue fixed.

 

[cabri]

Very slow response at times now
4-10 seconds to shift forums and/or load a thread.

at times only a section of the page loads; then 2-3 seconds later, the rest

 

[jpishgar]

Hey there all,

The issue from this morning has been mitigated and everything should be working fine once again. Our teams are still working with the network guys to figure out what took place to prompt the unexpected downtime, but so far it's looking like it was a minor problem with the database getting hung up on a specific query.

We've made some fine-tuning adjustments to the database and on the web server side, and also added a bit of extra monitoring so we'll be able to better diagnose the issue if it should crop up again in the future.

Again, our apologies for the unexpected unavailability of the forums during this. You may resume your regularly scheduled flame wars at this time. 😉

-JP

 

[MagnusTheBrewer]

Hey there all,

The issue from this morning has been mitigated and everything should be working fine once again. Our teams are still working with the network guys to figure out what took place to prompt the unexpected downtime, but so far it's looking like it was a minor problem with the database getting hung up on a specific query.

We've made some fine-tuning adjustments to the database and on the web server side, and also added a bit of extra monitoring so we'll be able to better diagnose the issue if it should crop up again in the future.

Again, our apologies for the unexpected unavailability of the forums during this. You may resume your regularly scheduled flame wars at this time. 😉

-JP

Ya know JP, you're totally going to blow the whole IT infallibility thing if you go interacting with users like this. Sorry, I forgot it wasn't ATOT.

 

[jpishgar]

Ya know JP, you're totally going to blow the whole IT infallibility thing if you go interacting with users like this.

I keep telling them to do the Scotty thing, but noooo. Something about "Integrity" as one of our core values or some such. 😛

-JP

 

[Ken g6]

Hm, I'm still having problems with the RSS feed, which was the first place I saw trouble.

http://forums.anandtech.com/external.php?type=RSS2&forumids=39

 

[jpishgar]

What seems to be the issue you are having there, Ken?