News Major tech outage gripping the world

[Amused]

Also covered here - https://forums.anandtech.com/threads/major-tech-outage-gripping-the-world.2620768/unread

You've linked this same Shroudreich thread!

The 2 threads were merged.
admin allisolm

 

[RnR_au]

This is such a massive screw up. Gonna be tough for crowdstrike to recover.

They are too big to fail. Many very large companies with significant seat numbers use them, and for these companies its too expensive to switch.

 

[pete6032]

You've linked this same Shroudreich thread!

Maybe the threads were merged?

Friday is a good time for this for business travelers. Not so good for personal travel.

 

[Amused]

Maybe the threads were merged?

Doh!

 

[RnR_au]

You've linked this same Shroudreich thread!

This is turning into a Benedryl Cabbagepatch meme thread...

 

[Amused]

This is turning into a Benedryl Cabbagepatch meme thread...

You leave Benzedrine Cloacalsplotch out of this!

 

[IEC]

Falcon Sensor?

More like Blue Falcon...

 

[Pens1566]

They are too big to fail. Many very large companies with significant seat numbers use them, and for these companies its too expensive to switch.

I mean it won't be immediate, but this will tarnish them when it comes time to re-up.

As I'm currently employed by one of those impacted companies, there are already discussions about comparing our loss vs a change of vendor.

 

[skyking]

Darling wife is effected with the blue screen boot loop BS. She is calling IT for help. Oh those poor bastiges. I offered to go do a safe mode boot for her and was declined.
Not my circus, not my monkeys.

 

[RnR_au]

I mean it won't be immediate, but this will tarnish them when it comes time to re-up.

From my understanding this has happened a number of times in the past...

https://www.reddit.com/r/sysadmin/comments/1594t5s/there_appears_to_be_another_widespread

- and the first comment in that thread point to an earlier event.

As I'm currently employed by one of those impacted companies, there are already discussions about comparing our loss vs a change of vendor.

I hope more will do the same 👍

 

[skyking]

<posting from my Chromebook, connected to my Linux server and local services> Hmm no outage here, I wonder what is up?

 

[MrSquished]

So this is Y2K? Their predictions were 24 years too early?

 

[RnR_au]

So this is Y2K? Their predictions were 24 years too early?

Nah... fix is easy but perhaps tedious.

Earlier today here in Australia my brother posted me this;

I just got free groceries at Woolworths [major supermarket chain here] as they couldn’t accept my gift card payment.

They asked if I had another way to pay. I said yes, but you have sold me this gift card and I would like to use it.

Perhaps worth a try depending on how things are going in your neck of the world...

 

[5to1baby1in5]

Sh!t, my work is up.
I was hoping for a 3 day weekend.

 

[[DHT]Osiris]

For the IT uninitiated what does Crowdstrike do?

Modern-era antivirus, uses a lot of machine learning to understand what's actually happening on a system vs what's supposed to, and stop stuff that isn't supposed to.

It's also really good at bluescreening half the planet, incidentally.

Sysadmins later today when people ask them for an after-action report:

 

[[DHT]Osiris]

Nah... fix is easy but perhaps tedious.

I guess tedious is one way to describe 'get hands on every workstation, and if you were smart enough to have bitlocker, get recovery keys for every workstation' then yes it's tedious.

 

[pete6032]

The news from IT this morning is that our company is not current with software patches so we never installed the faulty update and are not affected. Well done IT, well done.

 

[fskimospy]

Pretty sure they cloud-i-fied our network drives a year or two back, which would explain why they are all down. About half the computers can't log in at all, the other half can. Can't figure out the rhyme or reason why some work and others don't.

 

[[DHT]Osiris]

Pretty sure they cloud-i-fied our network drives a year or two back, which would explain why they are all down. About half the computers can't log in at all, the other half can. Can't figure out the rhyme or reason why some work and others don't.

DCs are probably dead, the ones you can log into are likely cached credentials. Try pinging your domain name and see if something comes back.

 

[fskimospy]

DCs are probably dead, the ones you can log into are likely cached credentials. Try pinging your domain name and see if something comes back.

Sorry I misspoke, the reason the other half can't log on is they are blue screened. Seems bad!

 

[Roger Wilco]

I’m still staring at a blue screen. The madness is setting in.

 

[GodisanAtheist]

There really is a meme for everything:



Anyhow, over/under on Crowdstrike goes out of business or crowdstrike is too big to fail?

 

[brycejones]

There really is a meme for everything:

View attachment 103452

Anyhow, over/under on Crowdstrike goes out of business or crowdstrike is too big to fail?

It won't go out of business, but they may take a hit.

The sad truth is most tech organizations have issues at some point. We've seen massive Azure and AWS issues in the past and no one has fled those platforms. It will be more about what crowdstrike does to talk about what the problem was and how they are going to avoid it in the future. (AKA we fired that fucker Bob over there who screwed up)

 

[outriding]

As an application support person for an EMR at a hospital that was impacted by the crowdstrike issue. And been up since 2am… I feel this outage is nothing compared to its competitors…

Cisco amp … cpu hog and day to day care and feeding

Carbon black… just sucks all around.

McCafee care and feeding and just decides to random crap here and there

CrowdStrike…. No care and feeding and very low cpu…. Been in it for about a year now. No exclusions.

Kudos to the server to to get us back and running in 5 hours. ( core infra and emr). Note the emr has about 600 servers. But 450 of them are Citrix clones

 

[hal2kilo]

There was also an azure outage right before that.

Some people were confusing the 2

It's specific to Microsoft software being affected by the update.