"Offline for maintenance" - what are websites doing during maintenance?

[saratoga172]

For the ERP system I manage at work I've got the downtime down to about 30 minutes every two months.

Individual components get backed up such that they can be restored without having to restore others (would incur some downtime though if say the prod database had to be restored) and we've got other backup backups if all else fails.

Downtime for me is really just applying some patches, rebooting and clearing some connections. Can do some of that without having to reboot but I've noticed if I let it go too long processes and things like to hang (Java you suck).

Much nicer than the other guy who used to manage the system. It would go down randomly during the day and I'd see him scrambling to reboot the prod environment in the middle of the day because the system was unresponsive or some component would break. I inherited that and set about fixing it after it went down once. Since then (3 years ago) it's had an issue 1 time during business hours and that happened because I was (stupidly) seeing how long I could go without rebooting the environment. Made it almost 3 months.

 

[Jeff7]

These mostly seem like reasonable answers. I guess I'll put off the murderous fit of rage for another time.

Thank you for the insights.

To keep the Unpaid Mods happy, they are allotted time to download all the Porn they need.

Waaahhhh, we can't multitask. I thought they got off on banning people.

Well, I know some of them do. :whiste:

Just out of curiosity, why doesn't anandtech need to do the maintenance anymore?

They didn't need to do it in the first place. It started as an experiment, and became ritual; kind of like religion.

"Why do we take the servers offline every Saturday night?"

"Kill the heretic!"

There is a good reason why companies, especially banks, are still on mainframes. It's because they work.

How many dollars pass through a bank's system in a single day? At the top banks, billions if not tens of billions. How many new software systems are 100% bug free on day 1 of production? None. Banks stay in business because of confidence in the systems. How quickly do you believe the confidence will erode when $50 billion in transactions one day gets all messed up due to a single software bug? What manager, what software development team, wants to be on the project to replace this system knowing that's the low-end liability for a single software bug?

Various old things can stick around "because they work." But they can also impede progress or improvements.

 

[Red Squirrel]

Some older stuff is interestingly more reliable too. For example most telephony is on DMS or PRX or similar systems these days. Those systems are from the 70's, but they still work today, and there is not really any comparable replacements for them to even switch to. It's interesting to walk between the bays and hear all the relays clicking. I wonder what the mileage of some of those relays must be, probably millions of operations easily. Cards get changed out all the time mind you, so I'm not sure how many parts are actually original, but the way it's designed you can change almost any part without downtime. THAT is how you design a system that is offering a service to people.

 

[XavierMace]

Never understood the idea of it either. Other than out of the ordinary stuff, like imaging the server to backup the OS/settings in an image (usually a one time thing), All my stuff is designed in such a way that maintenance (backups, usually) can be done live. In the case of my Ultima Online game server, updates can be pushed live, and the system only needs to restart to apply them (couple minutes max).

A game server isn't exactly a good reference point for what's normal when it comes to server maintenance windows. That said most MMO's have hour plus maintenance windows too.

That said, most planned maintenance windows also allow themselves a fair bit of wiggle room. Just because they give an hour window doesn't mean it necessarily takes an hour.

 

[John Connor]

When else are they going to read your PM's?

(remember, PM does NOT really mean "private message," but rather, "personal message."

LOL. All administrators on forums can read "private" messages. And even if they don't run a script, mod or extension they could just go into the database.


When I do maintenance the site is offline no more than 30 minutes. A few hours and that's a god damn site upgrade!

Hell. My crappy bank upgraded their site. Took two freaking days! That's just the beginning. After of which shit wasn't working right and then there was another feature they finally got working after a month!

I COULD RUN THAT IT SHIT HOLE BETTER! Never mind the fact they went from AES to RC4! RC Fing 4! I E-mailed the crap out of them about this! Now they went back to AES. LMFAO! Duh!

It was pretty pathetic my own website uses AES 128 and is more secure than the damn bank. Even ssllabs.com gave them a fat F! I got an A FFS!

 

[John Connor]

I let it go too long processes and things like to hang (Java you suck).

It's amazing. They don't even allow it on a damn smartphone. Just like Flash, DIE!

 

[slag]

Totally unnecessary with todays technology & software.
Unless... one is still stuck in the 1990's?

Or your work doesn't want to pay for redundant systems and would instead suffice with hardware failover/cold standby.

 

[slag]

yep, nightly batch jobs

i worked at a place that needed about 6 hours every night to finish all the mainframe batch jobs. our website relied on some mainframe services, so when the mainframe was batch processing the site lost a lot of functionality.

it was kind of embarrassing to be a developer for a website that is partially down %25 of every day. but enough old timey people were in management and mainframe development that they didn't think it was necessary to move the mainframe to real time processing.

Give it another lpar and let WLM spread out the load over the entire system. As you know, you can prioritize those nightly batch jobs for highest priority so they finish faster and if they still dont' finish in time, yeah, its time for another lpar.

 

[brianmanahan]

(remember, PM does NOT really mean "private message," but rather, "personal message."

um, nope

 

[lxskllr]

um, nope

Call it what you want, but a person with the right credentials can read them. That doesn't fit my definition of "private".