Yesterday experienced what is being labelled as the 'Largest IT outage in history' Meanwhile one in ten Democrats call for action from US President Joe Biden.
Someone, somewhere within IT cyber security giant CrowdStrike is having the worst day at work in their life.
On Friday, people in the northern hemisphere awoke to find TV news channels silent, trains cancelled and websites crashing in a massive IT outage impacting tens of millions of computers.
It seemed almost any business running Microsoft’s Windows 10 software experienced a sudden, dramatic failure. Computers appeared to automatically shut themselves down and display the so-called “blue screen of death”.
As dawn broke, Sky News UK broadcast an error message: “We apologise for the interruption to this broadcast.” It was forced to run archive footage for several hours.
Self-service checkouts have been plagued by the "blue screen of death" at Woolworths Metro in Auckland's Victoria St West. Photo / Sylvie Whinray
While it is not the end of the world or a nuclear apocalypse – and systems are, slowly, coming back online – one cyber security researcher dubbed the catastrophic blackout “CrowdStrike Doomsday”.
CrowdStrike is a US technology giant. The business is hardly a household name – although Formula 1 fans may recognise it as the sponsor of the Mercedes team and British racing drivers Lewis Hamilton and George Russell.
CrowdStrike is valued at more than US$80 billion ($133b) and is responsible for developing critical cyber security software used by thousands of businesses.
Its programs are deeply embedded into critical IT systems and networks. CrowdStrike develops a so-called “endpoint detection and response” system, helping large networks to detect and shut down hacking attacks – much likea business-wide antivirus.
Founded in 2011 and headquartered in Texas, its technology has proved wildly popular. It reported revenues of US$3b and more than 23,000 corporate customers last year. The company has also become well-known in cyber security circles for its work investigating high-profile hacks, including on the Democratic National Committee in 2015.
As a technology that has to respond to constantly evolving cyber threats, CrowdStrike routinely pushes out updates to its customers with new tools and layers of protection.
However, an apparently rogue file in one of its latest patches appears to have caused a massive IT collapse, according to cyber security experts and CrowdStrike engineers.
The outage has hit customers including airlines, train companies, airports, payment systems, supermarkets, the British health system and the Houses of Parliament cafe.
Passengers sit and wait at Suvarnabhumi Airport in Bangkok, Thailand, after a global IT outage has disrupted travel operations worldwide. Photo / Getty Images
On social media, IT managers reported dealing with implosions impacting tens of thousands of machines in their businesses.
Troy Hunt, a well-respected cyber security guru behind the website HaveIBeenPwned, said: “I don’t think it’s too early to call it: this will be the largest IT outage in history.”
While it was largely Microsoft’s Windows 10 systems hit by the IT problem, the root cause of the bug has been attributed to CrowdStrike – specifically an apparently dodgy file in its Falcon Sensor tool.
Posting on X, Hunt said CrowdStrike is a “massive player [in] the security space” whose technology often has so-called “privileged” access to business networks. This means it has broad control to update and modify a customer’s systems, in theory, to remove malware.
However, he said: “This also means that if something goes wrong with an update, it’s able to catastrophically nuke your machine.”
Security experts were already directing their anger at CrowdStrike over its handling of the outage. For several hours, there were no public updates or official statements from the company about the issue – other than those buried within its customer service portal behind a log-in screen.
The company did alert customers, saying: “CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor. Symptoms include hosts experiencing a bug check/blue screen error related to the Falcon Server.”
In a post on X, Brody Nisbet, a CrowdStrike executive, said the problem related to a “faulty channel file”, but added that a workaround for customers was not yet working for everybody. He said: “It’s a mess.”
Finally, George Kurtz, CrowdStrike’s founder and chief executive, issued a public statement.
“CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts,” he said.
“This is not a security incident or cyber attack. The issue has been identified, isolated and a fix has been deployed.
“Our team is fully mobilised to ensure the security and stability of CrowdStrike customers.”
CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts. Mac and Linux hosts are not impacted. This is not a security incident or cyberattack. The issue has been identified, isolated and a fix has been deployed. We…
CrowdStrike shares plunge over 14%, wiping billions off value
Shares in CrowdStrike plunged more than 14% in opening trading in New York, wiping billions of dollars off its value.
The disastrous outage is likely to raise questions over the stability of global IT systems, which are increasingly reliant on a little-understood web of IT infrastructure run by a handful of major companies.
“This is a very, very uncomfortable illustration of the fragility of the world’s core internet infrastructure,” said Ciaran Martin, the former head of the National Cyber Security Centre and a professor at Oxford University’s Blavatnik School of Government.
Customers queue at Woolworths Hobsonsville amid a mass IT outage.
Martin told The Telegraph he expected the cost of the outage to be “very significant and likely run into the billions” of dollars.
Meanwhile, Professor Alan Woodward, a cyber security expert at the University of Surrey, questioned how such a damaging glitch could have been released by the company. “You just can’t imagine something this awful would happen with proper testing,” he said.
Hunt said on X the problem was similar to fears of the so-called “Y2K” bug in 2000. Engineers feared millions of computers would crash at the turn of the millennium after failing to keep track of the change of date.
Those fears proved false, as companies effectively updated their systems. However, Hunt said: “This is basically what we were all worried about with Y2K, except it’s actually happened this time.”
