Do you guys think this will be a reasonable request of the IT department?

brainhulk

Diamond Member
Sep 14, 2007
9,376
454
126
Background: Last night an update was applied to our hospital software. The update was FUBAR'D and there was a severe disruption of care.

I want them to download every update to a local test computer first before they push it hospital wide. That way they can iron out any bugs without affecting patient care.

Reasonable right? Is this standard of practice for most IT departments?
 

KMFJD

Lifer
Aug 11, 2005
30,031
45,261
136
I remember when our counterparts down south would randomly reboot production servers and make changes during prime hours without informing anyone, fun times fun times.
 

Ajay

Lifer
Jan 8, 2001
16,094
8,109
136
Background: Last night an update was applied to our hospital software. The update was FUBAR'D and there was a severe disruption of care.

I want them to download every update to a local test computer first before they push it hospital wide. That way they can iron out any bugs without affecting patient care.

Reasonable right? Is this standard of practice for most IT departments?

It is standard practice in professional IT departments. That said, if this is a client/server update, things can go wrong in unpredictable ways despite advance testing. Suggesting this to IT, will, at best, result in a polite brush off.
 
  • Like
Reactions: ctbaars and DigDog

lxskllr

No Lifer
Nov 30, 2004
57,957
8,204
126
xresized-the-most-interesting-man-in-the-world-meme-generator-i.jpg
 

KB

Diamond Member
Nov 8, 1999
5,402
386
126
Its standard practice to have a TEST environment with a copy of production servers and clients and to test on those first. But as Ajay said: keeping TEST and PRODUCTION in perfect sync is difficult and unexpected differences can cause issues.
 
  • Like
Reactions: ctbaars

Exterous

Super Moderator
Jun 20, 2006
20,479
3,597
126
I'd be surprised if they don't do that already and, if they don't, they really really should.

But test\pilot deployment groups won't get everything. Two major roadblocks that I can see. One is scale and recently hit one of our clients. Certain HP desktops had a keyboard driver that caused a BSOD with a new MS patch (A fricken keyboard driver...). While this client has a pilot test group there is no good way (space, money, time spent constantly buying new models) to cover all the numerous models of computers they have in their 7,000 PC environment. So their pilot group didn't include the affected models. I think around 300 computers were affected. Percentage wise it is a small number but it felt like the sky was falling for the Help Desk.

The other one is with software testing. No way your IT group is going to be able to test everything the software does after an update. They should validate that the software runs after a patch but beyond that you'll need numerous actual users to go through their daily routines on the software. Finding volunteers for that (Who will ACTUALLY test it*) is tough and almost always needs to be driven from the top down and by the departments and not IT (Because IT bosses usually don't get to mandate finance, HR, (insert department here) spend X number of hours on software validation).

* I worked with a client on a web modernization project that resulted in significant (and necessary) visual updates. We asked every owner to test their site in the dev environment before updating their sites. So very many people clearly didn't actually test it despite telling us they did. We know because when we did the updates those same people would reach out and say "Something big must have happened because my site looks completely different from Friday." making it abundantly clear they never even looked at the dev site let alone actually tested anything like they told us they did
 
  • Like
Reactions: ctbaars and KMFJD

[DHT]Osiris

Lifer
Dec 15, 2015
15,258
13,555
146
The other one is with software testing. No way your IT group is going to be able to test everything the software does after an update. They should validate that the software runs after a patch but beyond that you'll need numerous actual users to go through their daily routines on the software. Finding volunteers for that (Who will ACTUALLY test it*) is tough and almost always needs to be driven from the top down and by the departments and not IT (Because IT bosses usually don't get to mandate finance, HR, (insert department here) spend X number of hours on software validation).
This is probably the biggest part of this. I work in IT, I have test systems (both client and server) that I deploy patches and software updates to prior to pushing to my greater organization. There's virtually always something that breaks, somewhere, regardless of what and how I test. It's gotten worse over the years as well, you can have two otherwise identical VMs side-by-side, and one will completely crap itself on an update while the other is fine, due to $reasons.

There's too many permutations of OS + software installations to test every scenario. Now, if your 'we use this every day' software wasn't tested, and it bricked on every workstation, yeah, someone dropped the ball there.
 
  • Like
Reactions: Homerboy

Homerboy

Lifer
Mar 1, 2000
30,859
4,976
126
Certainly updates should be applied in a test environment first. But it's virtually impossible to test every single re-world case scenario before rolling out to production. Some issues may be mitigated and avoided, but it's not going be perfect.
 

spacejamz

Lifer
Mar 31, 2003
10,866
1,515
126
Certainly updates should be applied in a test environment first. But it's virtually impossible to test every single re-world case scenario before rolling out to production. Some issues may be mitigated and avoided, but it's not going be perfect.

But catching obvious and big errors in test makes life much easier. Having a documented test plan with the scenarios that were tested also helps and then performing a lessons learned after the production installation to add any additional scenarios that were missed will make future installations smoother...at least in an ideal world....
 

spacejamz

Lifer
Mar 31, 2003
10,866
1,515
126
also, at our company, we deploy network related updates (security patches, etc) to a small group (usually one or two power users in each group/department) about 4 days before being sent the rest of the company....hopefully they will catch anything major before it gets rolled out company wide....
 

Homerboy

Lifer
Mar 1, 2000
30,859
4,976
126
But catching obvious and big errors in test makes life much easier. Having a documented test plan with the scenarios that were tested also helps and then performing a lessons learned after the production installation to add any additional scenarios that were missed will make future installations smoother...at least in an ideal world....

Oh yeah - entirely. The updates should be applied in a test environment, and within that test environment, the mission critical apps and programs should be tested as thoroughly as possible. My point was that not EVERY scenario can be tested and accounted for and with any OS update, it's likely to affect somebody adversely.
 

spacejamz

Lifer
Mar 31, 2003
10,866
1,515
126
Oh yeah - entirely. The updates should be applied in a test environment, and within that test environment, the mission critical apps and programs should be tested as thoroughly as possible. My point was that not EVERY scenario can be tested and accounted for and with any OS update, it's likely to affect somebody adversely.

Hopefully good companies realize that but usually those companies have good testing procedures in place and they learn which scenarios they missed might cause the biggest headaches in the future so it doesn't happen again....but if a different PM or team handles it, alot of time all of the prior experiences is all for naught....
 

rh71

No Lifer
Aug 28, 2001
52,844
1,049
126
If for no other reason, they should be doing testing to cover their asses. Because that's the first question management will ask after a catastrophe of an update. You can then at least say "yes we tested and it didn't happen there" and show appropriate documentation. Say "no we didn't test" and heads will/should roll.

Bottom line, if anything has the potential to effect multiple clients, I would be sure to test. I guess some people just don't care about their jobs / livelihood.
 

dyna

Senior member
Oct 20, 2006
813
61
91
Background: Last night an update was applied to our hospital software. The update was FUBAR'D and there was a severe disruption of care.

I want them to download every update to a local test computer first before they push it hospital wide. That way they can iron out any bugs without affecting patient care.

Reasonable right? Is this standard of practice for most IT departments?

You can ask but if you take this path, you will likely be requested to be in the critical path, which means signing off the upgrade that all the features work. IT may test it but the full responsibility that it works will likely be you. It is called business sign off.

When something goes wrong after deployment, the question will be, "Why did brainhulk signoff on this upgrade when there are obvious issues?"
 

Brovane

Diamond Member
Dec 18, 2001
5,653
1,914
136
Background: Last night an update was applied to our hospital software. The update was FUBAR'D and there was a severe disruption of care.

I want them to download every update to a local test computer first before they push it hospital wide. That way they can iron out any bugs without affecting patient care.

Reasonable right? Is this standard of practice for most IT departments?

I would approach it in a different way. Don't try to tell them how to do their job. I assume you are in a leadership position. Go to IT leadership and explain the impact in lost hours etc. that this outage caused your department. Ask them what steps they are taking to prevent this from happening in the future. The fact that your department was impacted might not have even made it's way to IT leadership. Let them explain what steps they are taking, if any. If they don't mention a test environment, politely ask if they have a test environment. Going to them and telling to them how to fix the issue puts them immediately on the defensive, no matter how reasonable the question is.
 
  • Like
Reactions: Cuular

ultimatebob

Lifer
Jul 1, 2001
25,134
2,446
126
Background: Last night an update was applied to our hospital software. The update was FUBAR'D and there was a severe disruption of care.

I want them to download every update to a local test computer first before they push it hospital wide. That way they can iron out any bugs without affecting patient care.

Reasonable right? Is this standard of practice for most IT departments?

If they don't reply back and say that they are already doing this, you probably need new IT people.

Most places I work have separate Development, QA, UAT, and Production environments. Even if the updates make it through that gauntlet, they still need get QA approval, customer approval, AND management approval before the update goes to production.
 

brainhulk

Diamond Member
Sep 14, 2007
9,376
454
126
How do you know they didn't deploy to a test environment first?

The conversation I had with the our department IT person:

IT: Was there an update to Cerner last night?

Me: Yes. I received a pop up telling me I had to log out so an update can be performed.

IT: So after the update is when you started receiving the errors.

Me: Yes. It was working flawlessly prior to the update.

IT: ok, because Cerner was denying the errors had anything to do with their end. Now I can go back to them it was their update.

Yeah...our IT dept doesn't test anything. They didn't even know an update was coming.

Im not management by the way. Just the user at night that has to find work arounds when shit hits the fan. I have a feeling management will agree with me since this directly impacts pt care. Not sure why the issue hasn't been raised before.

I checked reddit though and it looks like it not a common thing to test updates at health institutions. Maybe it's a cost issue. And hospitals being frugal.
 

Exterous

Super Moderator
Jun 20, 2006
20,479
3,597
126
Yeah...our IT dept doesn't test anything. They didn't even know an update was coming.

Im not management by the way. Just the user at night that has to find work arounds when shit hits the fan. I have a feeling management will agree with me since this directly impacts pt care. Not sure why the issue hasn't been raised before.

I checked reddit though and it looks like it not a common thing to test updates at health institutions. Maybe it's a cost issue. And hospitals being frugal.

A fair number of vendors don't tell people when they release updates to software (that isn't to say your IT couldn't be incompetent just that its hard to make that call from this case). We've had to scramble more than a few times because someone releases an update that requires changes to dependent software\OS settings and doesn't provide Dev channel QA routes or even an email notifying you of changes.

That said I am surprised if they don't do any testing and what you found on reddit. I'm familiar with two large hospital systems in MI and they are crazy sticklers for testing and change management. But they also have 30k+ computer environments so they kinda need to be and have the financial weight to mandate software vendor behaviors.
 

Stopsignhank

Platinum Member
Mar 1, 2014
2,440
1,766
136
Sounds to me that it was not your IT department that did an update but an outside company.

If Adobe does an update to Acrobat and completely messes up your system, you should get upset with Adobe, not your IT department.
 
Feb 25, 2011
16,898
1,547
126
I checked reddit though and it looks like it not a common thing to test updates at health institutions. Maybe it's a cost issue. And hospitals being frugal.

Yeah, wouldn't surprise me. For most companies, IT is a line item cost, not an investment.

But if you're using third party hosted services, THEY should be testing the upgrades...
 

brainhulk

Diamond Member
Sep 14, 2007
9,376
454
126
I was just thinking right now in the shower...

What if I'm the test hamster for updates. There's less stuff going on in the middle of the night. Management is not present to hear the complaints. I have a lot of experience dealing with downtime. I never complain and I have the whole night to log the errors. They can review in the morning.

It all makes sense now!

:(
 

Brovane

Diamond Member
Dec 18, 2001
5,653
1,914
136
Sounds to me that it was not your IT department that did an update but an outside company.

If Adobe does an update to Acrobat and completely messes up your system, you should get upset with Adobe, not your IT department.

Depends on the size of the IT network. A lot of larger environment's block vendors from updating end-user software directly. They don't want a outside vendor pushing mass updates to their end clients. In your example with Adobe. IT would be made aware of the update, package the update up to be pushed out, do a few tests on test machines, push out to a Beta group of end users, if there are no issues then push to all end users.
 

Brovane

Diamond Member
Dec 18, 2001
5,653
1,914
136
The conversation I had with the our department IT person:

IT: Was there an update to Cerner last night?

Me: Yes. I received a pop up telling me I had to log out so an update can be performed.

IT: So after the update is when you started receiving the errors.

Me: Yes. It was working flawlessly prior to the update.

IT: ok, because Cerner was denying the errors had anything to do with their end. Now I can go back to them it was their update.

Yeah...our IT dept doesn't test anything. They didn't even know an update was coming.

Im not management by the way. Just the user at night that has to find work arounds when shit hits the fan. I have a feeling management will agree with me since this directly impacts pt care. Not sure why the issue hasn't been raised before.

I checked reddit though and it looks like it not a common thing to test updates at health institutions. Maybe it's a cost issue. And hospitals being frugal.

Bring this up to your leader in a e-mail including the impact that this had on his team that was working overnight. If your leader is competent he will then take this to IT leadership.
 

monkeydelmagico

Diamond Member
Nov 16, 2011
3,961
145
106
Ask them what steps they are taking to prevent this from happening in the future. The fact that your department was impacted might not have even made it's way to IT leadership. .

+1. If it caused any significant disruption at our hospital the CIO would be the recipient of this news and would be explaining to the rest of the C-suite how to keep it from happening again.

Depends on the size of the IT network. A lot of larger environment's block vendors from updating end-user software directly.

+1 again. Everything runs though the IT department prior. They do run test environment on some application updates but I do not know what the logic is for how they decide which ones need testing and which ones can go straight out. I'm curious now and next time I have the CIO in my office I will ask him. I'm sure my eyes will glaze over after a couple minutes of him droning on about it.