CrowdStrike Strikes Out

CrowdStrike system down

THE interconnected nature of our increasingly digital world, for all its benefits, comes with the caveat of enabling cyber incidents to spread rapidly across the globe.

By now, most people would be aware of the incident on 19 July 2024, when a software update issued by cybersecurity company CrowdStrike caused issues with subscribers using the Windows operating system (OS). Mac and Linux hosts were not impacted.

Those who had installed the faulty update were confronted with the so-called ‘blue screen of death’ (BSoD), a critical error screen that indicates a system crash when the OS has reached a condition where it can no longer operate safely.

Blue Screen of death

Cyber Specialists

CrowdStrike, is an USD83 billion company with more than 20,000 subscribers around the world including Amazon and Microsoft. The Texas-based cybersecurity company, founded in 2011, offers real-time threat monitoring and cloud server data protection services to many of the world’s biggest companies like Amazon, Google, Microsoft, Target and the US Government.

In the hours following the incident, CrowdStrike CEO George Kurtz responded* with the assurance that it was not a cyberattack and the issue had been “identified, isolated and a fix has been deployed”.

However, the extent of the damage has yet to be revealed. As eloquently put by an anonymous Redditor, “A software that is supposed to be used for protection has done more damage in a few minutes than any malware can dream of doing in a lifetime!”

Fallout

The incident immediately impacted airlines globally, with over 1,700 flights cancelled in the US alone. Banks and payment systems experienced disruptions, many hospitals were unable to access their patient databases and several public transport services in the UK and US were temporarily halted.

On what was expected to be the busiest day** in UK aviation in the past 5 years prior to the cyber incident, around 50,000 passengers were stranded across local airports, with some travellers reported to be “absolutely despondent”.

At Singapore’s Changi airport 10 flights were cancelled*** and many more were delayed, with the staff resorting to manual check-in methods.

Compensation

Some of the disrupted were offered immediate compensation, such as traveller Yuka Ibara who was travelling in Fukuoka, Japan. She reported difficulty checking in at ANA Crowne Plaza Hotel and was compensated with free ice-cream and coffee. The hotel also waived charges on all drinks in her room’s refrigerator. The IHG hotel group reported a global systems outage.

But as airline and public transport disruptions persist, for the many stranded travellers, reimbursement will likely be an afterthought for the time being, as operators struggle to restore order.

CrowdStrike Microsoft

Expert Opinion

Kevin Reed, Chief Information Security Officer of cybersecurity company Acronis, details the cause of the incident. “It’s a bug in the CrowdStrike EDR (endpoint detection response) software. Because EDR agents run with highest privileges on end-user devices, a bug in the agent can bring the entire operating system down.”

EDR systems continuously monitor for threats and initiate automated responses, hence digital shields of those affected will be lowered until the incident has been fully resolved.

Reed, who frequently highlights the impact of cybersecurity incidents on small and medium enterprises (SMEs), believes that there is little businesses could have done individually to pre-empt this issue, “EDR agents are typically auto updated with no staged updates because they need to react to changes in attack tactics and new malware faster.”

He goes on to explain that this type of error will require manual intervention, but worries about the speed with which the response can be executed. The recommended remediation is to reboot the workstation or server in ‘safe mode’, and manually delete the faulty CrowdStrike driver file.

This manual intervention will definitely affect companies’ ability to react. Also, removing the driver file will mean the system will no longer be protected by EDR, exposing it to attacks.

“I would expect opportunistic attackers to take advantage of it. The ease with which their driver files can be deleted also raises questions about the self-protection mechanisms of CrowdStrike’s software.”

Cybersecurity firm CyberArk’s CIO, Omer Grossman, expects this manual recovery process will take days to execute. He eagerly awaits a detailed breakdown of the event from CrowdStrike.

“What caused the malfunction? Possibilities range from human error, for instance a developer who downloaded an update without sufficient quality control, to the complex and intriguing scenario of a deep cyberattack, prepared ahead of time and involving an attacker activating a ‘doomsday command’ or ‘kill switch’.”

All possible outcomes will surely lead to a loss of faith in the EDR service provider and a loss of confidence from investors in the publicly listed company.

Recommendations For The Afflicted

Reed opines on the potential fallout. “The implications are that many businesses around the world were affected and have come to a standstill. A lot of companies won’t be able to recover quickly and it is already a mess. It does not seem that there is a way to recover systems automatically from this issue.”

For now, cybersecurity experts such as Reed place the blame solely on the EDR service provider.

“This incident highlights the importance of rigorous testing and staged updates for EDR agents. Normally, testing is done with every release and can take days to weeks, depending on the size of the update or changes.”

Reed notes that Acronis’ customers have been able to restore the previous day’s Windows/CrowdStrike update, and experienced minimal downtime and exposure to the incident. Much of the affected infrastructure was only momentarily compromised, and many public transport and payment services are once again functional.

But the hands-on treatment that the many corporate computers require will take some time, and in this brief window the seeds of future exploits may be planted.

Reed’s advice: “We recommend all businesses ensure robust backup solutions and advocate for better testing protocols from their security vendors.”

Domestic Solutions

Notably, China’s onshore businesses have been largely unaffected**** by the incident, being less reliant on Microsoft than the rest of the world, relying predominantly on local companies and cloud service providers Alibaba, Tencent and Huawei. Geopolitical tensions have seen the country’s government organisations, businesses and infrastructure operators adopt homegrown IT solutions, with some referring to this parallel network as the ‘splinternet’.

Such events highlight the fragility of digital life and call into question the extent to which global cyberspace should be interconnected, providing strong tailwinds to deglobalisation narratives that have taken hold since the 2020 pandemic. Other nations will likely be encouraged to follow China’s example, or at least encourage local entities to diversify their digital service providers to shore up their own digital infrastructure, as a matter of national security and economic integrity.

See also  TEST TEST TEST, Or Things Will Break Down

LEAVE A REPLY

Please enter your comment!
Please enter your name here