"Offline for maintenance" - what are websites doing during maintenance?

[Jeff7]

ATOT used to go offline weekly for maintenance.
I can't access my bank account on weekend nights. I can't check my credit card statement on weekend nights. Some online stores won't allow checkout at some times because they're doing maintenance. My email server is evidently offline right now for maintenance.

Why do sites need to disable key features for this? Can't this stuff be done without disabling functionality? What all is being done during this maintenance?
Is it done because it's just cheaper to offline the system for this work? Evidently it's not necessary, as the AT forums don't seem to go offline anymore, and I'm assuming that they're not quietly tearing themselves apart right now as a result.





</FWP>

 

[BoomerD]

When else are they going to read your PM's?

(remember, PM does NOT really mean "private message," but rather, "personal message."

 

[DaveSimmons]

There isn't one answer.

AT forums was probably cleaning up databases and rebooting the server.

Sometimes it is a server code update. It's very tricky to do that "live" instead of with the site hidden from view.

Sometimes a system is not set up to do full backups while customers are busy entering new data. I can see that being risky for banks and credit card processing.

In some cases they might just be messing with you, personally. The rest of us can use the site but they've decided not to let you do it.

That last one is probably not true.

 

[who?]

Maybe they're defragging or doing scans.

 

[lamedude]

Removing a 0 from the speedup loop.

 

[sandorski]

To keep the Unpaid Mods happy, they are allotted time to download all the Porn they need.

 

[A Casual Fitz]

Just out of curiosity, why doesn't anandtech need to do the maintenance anymore?

 

[lxskllr]

Just out of curiosity, why doesn't anandtech need to do the maintenance anymore?

They didn't need to do it in the first place. It started as an experiment, and became ritual; kind of like religion.

 

[sportage]

Totally unnecessary with todays technology & software.
Unless... one is still stuck in the 1990's?

 

[disappoint]

They didn't need to do it in the first place. It started as an experiment, and became ritual; kind of like religion.

That's just what they want you to think. The real reason is far more sinister. They sacrifice virgin microprocessors that have never run Windows.:sneaky:

 

[lxskllr]

That's just what the want you to think. The real reason is far more sinister. They sacrifice virgin microprocessors that have never run Windows.:sneaky:

:^O

Won't someone think of the children!?

 

[cubby1223]

ATOT used to go offline weekly for maintenance.
I can't access my bank account on weekend nights. I can't check my credit card statement on weekend nights. Some online stores won't allow checkout at some times because they're doing maintenance. My email server is evidently offline right now for maintenance.

Why do sites need to disable key features for this? Can't this stuff be done without disabling functionality? What all is being done during this maintenance?
Is it done because it's just cheaper to offline the system for this work? Evidently it's not necessary, as the AT forums don't seem to go offline anymore, and I'm assuming that they're not quietly tearing themselves apart right now as a result.

Your bank is likely still operating off of a mainframe system installed in the 80's. Sometimes they have to shut down some functionality to catch up on other processing and reporting that needs to occur.

 

[brianmanahan]

Your bank is likely still operating off of a mainframe system installed in the 80's. Sometimes they have to shut down some functionality to catch up on other processing and reporting that needs to occur.

yep, nightly batch jobs

i worked at a place that needed about 6 hours every night to finish all the mainframe batch jobs. our website relied on some mainframe services, so when the mainframe was batch processing the site lost a lot of functionality.

it was kind of embarrassing to be a developer for a website that is partially down %25 of every day. but enough old timey people were in management and mainframe development that they didn't think it was necessary to move the mainframe to real time processing.

 

[IronWing]

I thought Saturday was movie night at ATOT when the gang turned off the servers and used all of AnandTech's bandwidth to do nothing but pull in porn.

 

[PottedMeat]

Official Bank of America IT Dude Benefits
...
Weekly Full Bandwidth Allocation for Personal Use, Customer Network Shutdown
Pornography Pipeline Access ( Fetish Only )
...

its a standard offering

 

[cubby1223]

yep, nightly batch jobs

i worked at a place that needed about 6 hours every night to finish all the mainframe batch jobs. our website relied on some mainframe services, so when the mainframe was batch processing the site lost a lot of functionality.

it was kind of embarrassing to be a developer for a website that is partially down %25 of every day. but enough old timey people were in management and mainframe development that they didn't think it was necessary to move the mainframe to real time processing.

There is a good reason why companies, especially banks, are still on mainframes. It's because they work.

How many dollars pass through a bank's system in a single day? At the top banks, billions if not tens of billions. How many new software systems are 100% bug free on day 1 of production? None. Banks stay in business because of confidence in the systems. How quickly do you believe the confidence will erode when $50 billion in transactions one day gets all messed up due to a single software bug? What manager, what software development team, wants to be on the project to replace this system knowing that's the low-end liability for a single software bug?

 

[olds]

Two words: office party

 

[TwiceOver]

It usually means that I took a couple days off that week or the following week (depending on maintenance day) and I don't want to use vacation time.

 

[DrPizza]

There is a good reason why companies, especially banks, are still on mainframes. It's because they work.

How many dollars pass through a bank's system in a single day? At the top banks, billions if not tens of billions. How many new software systems are 100% bug free on day 1 of production? None. Banks stay in business because of confidence in the systems. How quickly do you believe the confidence will erode when $50 billion in transactions one day gets all messed up due to a single software bug? What manager, what software development team, wants to be on the project to replace this system knowing that's the low-end liability for a single software bug?

Along the same train of thought - who knows for certain that there isn't some sort of vulnerability during the maintenance? If you only have 4 or 5 customers at the most during that time frame, for many businesses and banks, tis better to be safe than sorry.

 

[Markbnj]

Just out of curiosity, why doesn't anandtech need to do the maintenance anymore?

Self-replicating code optimizers paired with in-channel Schangelbeck arrays.

 

[ultimatebob]

Serious Answer: A lot of older (non-clustered, non-failover) data centers can't be upgraded without taking the servers down first.

Some data centers also take down their servers for things like offline database backups, which usually aren't necessary (depending on the database they used) but are still done out of an abundance of caution.

You also have issues like UPS and generator upgrades where the entire data center might have to go down for electrical work. If you aren't geo redundant (fancy way of saying multiple datacenters), you'll need downtime for that as well.

Basically, you can pull off near zero downtime if you're willing to pay for a ton of redundancy. Most companies aren't willing to pay for it.

 

[CZroe]

Anandtech would famously run benchmarks on the forum database.

 

[ultimatebob]

Anandtech would famously run benchmarks on the forum database.

I would hope that they would use a backup copy for that... but that might explain why we used to have so much downtime.

 

[XavierMace]

There is a good reason why companies, especially banks, are still on mainframes. It's because they work.

How many dollars pass through a bank's system in a single day? At the top banks, billions if not tens of billions. How many new software systems are 100% bug free on day 1 of production? None. Banks stay in business because of confidence in the systems. How quickly do you believe the confidence will erode when $50 billion in transactions one day gets all messed up due to a single software bug? What manager, what software development team, wants to be on the project to replace this system knowing that's the low-end liability for a single software bug?

Yeah, banks get really cranky really fast when you tell them their teller system is down due to a failed software upgrade over the weekend.

But due to the age of their software (and usually hardware), they observe a lot of practices that may seem archaic these days. Such as rebooting after updates.

 

[Red Squirrel]

Never understood the idea of it either. Other than out of the ordinary stuff, like imaging the server to backup the OS/settings in an image (usually a one time thing), All my stuff is designed in such a way that maintenance (backups, usually) can be done live. In the case of my Ultima Online game server, updates can be pushed live, and the system only needs to restart to apply them (couple minutes max). I've toyed with the idea of making it seamless, where there is a "connector proxy" that handles the network side and it just freezes the user's connection while the shard restarts, but I don't have enough of a big population to justify it.

Heck, I've practically "live" migrated my game server into another country. It involved 15 minutes of down time, that was the time it took to move the database between the data centres. A restart is needed to apply any kind of patch, this was basically equivalant to such a restart. None of this multi hour downtime business. Even the official servers for this game go down for hours to apply updates, it's crazy they can't design this better.

For websites there is really absolutely no reason for them to actually go offline. I could MAYBE see having a mode where everything is read only, if you want to ensure an atomic DB backup but even then there are ways for large sites to deal with that if they code it properly.

Of course there are other out of the ordinary maintenance like physical stuff like relocating servers in a rack, redoing networking etc... but that's different. Routine multi hour downtime for stuff is ridiculous.

At work our ticketting system goes down nightly for like 8 hours. On weekends it's down for almost the whole day Saturday and Sunday. Completely insane. That's what happens when you outsource programming.