How a single IT update caused global havoc

A single update pushed out from an anti-virus company in the US has managed to cause global havoc today.

It’s being described as the biggest outage ever, and while there have been a few lately, it’s certainly hard to recall something that has taken out as many services and companies across the world as this one has. I stopped updating my list of brands reporting issues within an hour of starting it because there were simply so many names to keep track of.

You may never have heard of the anti-virus firm Crowdstrike before but something it did to its virus scanner Falcon had a very adverse effect on computers running Windows software – in their millions.

Blue Screen of Death reported worldwide. You probably don’t need me to tell you what that is. Microsoft was quick to say it was a “third-party issue” – in other words, not its fault. Apple and Linux users, unaffected, rejoiced.

Crowdstrike says it has now issued a fix but several IT contacts have told me every single machine in their organisations will require a manual reboot in safe mode, and some of these devices are likely to be more physically accessible than others.

There is currently no suggestion that it was malicious, or that anybody’s data has been compromised, accessed or stolen. The cyber-security world still advises that it’s a good idea to keep on top of software updates – although perhaps today is not the day to bang that particular drum.

One thing that was missing from Crowdstrike’s statement, when it arrived, was any hint of an apology. Perhaps a sign of awareness that this could prove to be an extremely expensive mistake, as people missed flights, trains, hospital appointments on a summer Friday. Already there are rumblings of a backlash that the word “sorry” was notably absent from CEO George Kurtz’s words.

It is a poignant reminder of how reliant the world has become on devices managed remotely by huge companies, and how powerless it leaves us when they fail.

These enormous platforms are bombarded all the time with attempted cyber-attacks, and ill-thought out software updates, and most of them are caught by the tech giants’ robust systems. There will no doubt be a post-mortem at Microsoft as to why this one was not.

Timing is also everything. “Never push an update on a Friday,” sighed one computer scientist I spoke to, head in hands.

That's because if something goes wrong and it takes time to fix, firms typically have fewer people working at weekends so it will inevitably take even longer to resolve.

If you are a Crowdstrike customer, there are details on its support website explaining the fix. If you work for a company with an IT team, they may well be co-ordinating a company-wide response.

Often by the time you’ve noticed an outage, it has fixed itself. This is certainly not the case here. It is likely to be a few days at least before the world returns to normal.