IT response time at work

[Mark R]

Just wondering about this after some systems failures at work recently.

I work in a hospital, and the main medical results server failed over the weekend. This server is essentially the main link between the laboratories and X-ray department and the wards. It was not fully operational until this morning - causing quite a lot of stress amongst laboratory, medical and secretarial staff. I suspect that part of the delay was that a replacement piece of hardware had to be ordered in, with some more delay coming from the server having to resync from 10 day old backups.

Other things too have surprised me - after a recent power outage and on a seperate occasion, a routine emergency generator test, the entire IT system shutdown and was unavailable for the rest of the day, presumably while the key servers rebooted. I would have thought that the network infrastructure would be on the emergency supply - evidently not.

 

[amdskip]

Those servers aren't on APC's or backup generators? What hardware failed on the server?

 

[AgaBoogaBoo]

IT departments have lots of work to improve on across many companies.

 

[rh71]

could be bureaucracy getting in the way of expediting the new hardware. That's always a b!tch to deal with.

 

[LethalWolfe]

IT response at my last job was measured in weeks. Partially IT being over worked and partially IT being lazy.


Lethal

 

[brtspears2]

I say the biggest problem with IT depts now is that they are not funded well and under staffed.

 

[VTEC01EX]

We're underfunded, and rather lazy as a result. Gotta love government work!

 

[Goosemaster]

Originally posted by: VTEC01EX
We're underfunded, and rather lazy as a result. Gotta love government work!

but this is a fvking hospital...such delays should not be tolerated...yet they are...


<---knows nothing about a hospital, but realizes that every line of work has problems....

 

[phonemonkey]

Originally posted by: amdskip
Those servers aren't on APC's or backup generators? What hardware failed on the server?

I've seen server get knocked offline by a bad APC that sent a surge through to the computers (it happened about 2 months ago). It literally knocked everything in the company out, since it took down the backbone routers as well, and took about 6 hours to get everything back up and some semblance of normality back.

Then again, I know several people in my dept that intentionally will go slower if people start riding their backs when they're already working as hard as possible.

 

[rh71]

APC is a brand... was it an APC or some other UPS ? Aren't they supposed to be the most reliable ?

 

[her209]

Down time should be measure in minutes.

 

[Anubis]

at my work the responce time is instantinous cause i am IT

 

[rh71]

Originally posted by: Anubis
at my work the responce time is instantinous cause i am IT

Wow you have the ability to be in more than 1 place at once ?!

 

[BornStar]

Originally posted by: her209
Down time should be measure in minutes.

Ideally, yes, but that's not always possible. Our main server takes over half an hour to reboot at work sometimes. It isn't terribly rare for that one to crash either.

 

[DT4K]

We have at least one backup server for every server we have.
Our db server and our web server take less than 2 minutes to reboot.
Last week, we had a hardware failure on our main SQL Server box.
We did a db restore to our backup box and switched all our applications to point to this backup server in about 15 minutes.
I guess it all depends on how critical the system is. Our production lines can run for about 20-30 minutes without the db before they have to stop production. And it costs us about 50k per hour of production downtime.

 

[Batti]

The other alarming thing in the OP was 10 day old backups! I'd be swinging from a rope if that ever happened. Sheesh, my employer doesn't know how good they have it.

Almost everything we deploy these days is done on parallel servers with automated failover. We can run for days on generator. And yet we still have areas (networking) where we know the redundancy needs to be better, but the budget is the issue.

Hospitals typically have to endure IT audits. I know we do at Universities. I'd think those problems would be brought to light real quick.