Mass IT Outage. Is it affecting you?

Piing

Shredder
Messages
1,443
I just went to register a online medical claim on Cigna, but I cannot login. I check the news and I see that there is a worldwide outage affecting airlines, banks, media, trains, health services, 911 emergency service... Windows kaput with Blue Screen of Death due to a Crowdstrike security update.

I'm using a laptop with Windows, but no issues here. I have disabled automatic updates, though, because that is one of the recommendations when using a DAW.

How is it going there? Have you find yourself affected?
art design 90s GIF
Windows Error GIF


As far as TGF and the Axe-FX are alive, everything is business as usual.

This Is Fine GIF
 
Last edited:
I work as a consultant doing web development (user interfaces, APIs, cloud stuff, test automation, mobile development...). My employer is an agency providing consultants for our clients.

Our company started using Crowdstrike a few months ago and there was a lot of pushback from us employees. The primary concerns from us employees were privacy and performance. Unfortunately every service like this that I have ever reluctantly had to install has turned out to be a nightmare that causes more issues than it solves.

The premise behind adopting it is sound: security compliance with clients. As a company, we need to be able to show that we are doing everything to make sure sensitive client data is not leaked out.

As consultants we can choose which laptop we use. Most of us use Macbook Pros, where the problem doesn't occur. But there are some who prefer Windows and they have basically not been able to work today.

Some of our people have complained that even on Mac, Crowdstrike has started hogging a lot of CPU cycles for no good reason.

Crowdstrike released a guide to delete some files on Windows to get it working again, but to delete them you need to get to safe mode, which means you need to have a Bitlocker key to do that, which you most likely don't have since it's centrally managed, so you end up in this great loop of not having access to fix the issue yourself. Some of these machines are also stuck in a boot loop so they cannot get an automated update that fixes this. This is a total nightmare for IT support at companies when employees need to bring their systems in and each has to be manually updated.

I would not be surprised if this is bad enough to make Crowdstrike go under. I have no sympathy for a security company that seemingly does not understand what update rollouts are. The way these work, is that a new update to a software gets rolled out first week to say 10% of users, the next week to 20%, and increasingly more until everyone has the update. This would have avoided exactly this problem where a huge number of clients are affected, instead it would have been caught with the first 10%.
 
Last edited:
Yeah all these gigantic "system management" companies suck ass. SolarWinds, an endless list of others and now CrowdStrike.

Thankfully, we don't use them at my company!
 
The thing is is that the internet of things and code are getting so complicated most dont understand the underlying workings
so it's often a crapshoot
 
My business was affected by the Azure Central outage last night, which was unrelated to (but overlapped) the beginning of the Crowdstrike bug. We're only beginning to get a sense of how bad this thing is. It looks like every affected machine has to be manually updated (boot into safe mode, delete a file, reboot), which is going to be incredibly time consuming for large organizations. My bank is down. Delta, United and American are all grounded. No flights anywhere in NZ. Schiphol is closed. It's absolutely wild.
 
Crowdstrike released a guide to delete some files on Windows to get it working again, but to delete them you need to get to safe mode, which means you need to have a Bitlocker key to do that, which you most likely don't have since it's centrally managed, so you end up in this great loop of not having access to fix the issue yourself. Some of these machines are also stuck in a boot loop so they cannot get an automated update that fixes this. This is a total nightmare for IT support at companies when employees need to bring their systems in and each has to be manually updated.

This is where I am at the moment, needing a BitLocker key to get access to my hard drive.
 
I've never heard of BitLocker key before. I've looked at it, and I see that it is not activated at my hard drives. Is that a good thing?
 
I've never heard of BitLocker key before. I've looked at it, and I see that it is not activated at my hard drives. Is that a good thing?
It's the manual encryption key (secret password) for MS Windows' "BitLocker" drive encryption. Most people don't encrypt their drives.
 
I'm working remotely today, and trying to pull some loose ends together before leaving for a (very rare) vacation without e-mail, etc. Most of our servers/ services seem to be back up, but about half of our workstations are down... including my own, naturally.

The timing is spectacular. :facepalm
 
A couple of external company websites our employees access daily are down today but the core aspects of our environment are still rollin
 
Back
Top