CrowdStrike reveals details of incident that led to global PC outage

The cybersecurity company explains that a misconfigured Rapid Response Content update to its Falcon platform caused PC crashes worldwide.

Cybersecurity company CrowdStrike has released an update on the causes of the infamous Blue Screen of Death on a number of Windows PCs worldwide.

As far as we know, the issue was due to an update sent to the company’s Falcon endpoint detection and response platform – more specifically, a single, misconfigured Rapid Response Content update sent to the Falcon sensor on those platforms.

The patch affected sensor version 7.11 and higher and was released on July 19 at 04:09 UTC. All machines running this sensor version that were online until 05:27 UTC – when the update was rolled back – were affected.

Unlike Sensor Content publishing, where the customer has control over deployment across their fleet, Rapid Response Content is automatically deployed to effectively track and identify new threats.

“This capability is used by threat intelligence engineers to collect telemetry, identify indicators of adversary behavior, and conduct detection and prevention activities,” CrowdStrike said in an update to its Remediation and Guidance Hub for the July 24 incident.

“Rapid Response Content is a behavior-based heuristic that is different from CrowdStrike’s sensor-based AI prevention and detection capabilities.”

Rapid Response content is published as “template instances” that “depict specific behaviors that the sensor should observe, detect, or prevent. Template instances have a set of fields that can be configured to match the desired behavior.”

In this case, it was an InterProcessCommunication (or IPC) template type that was first tested and validated on March 5 and introduced the same day via Channel File 291.

“Subsequently, three additional IPC template instances were deployed between April 8 and April 24, 2024,” CrowdStrike said.

“These template instances worked as expected in production.”

However, on July 19, “two additional IPC template instances were deployed. Due to an error in the content validator, one of the two template instances passed validation even though it contained problematic content data.”

“Based on the testing performed prior to the first deployment of the template type (on March 5, 2024), confidence in the checks performed in the Content Validator, and previous successful deployments of IPC template instances, these instances have been moved to production,” CrowdStrike said.

When received and loaded by the Falcon sensor, the “problematic content in channel file 291 resulted in an out-of-bounds memory read, which triggered an exception.”

“This unexpected exception could not be handled properly, resulting in a Windows operating system crash (BSOD),” CrowdStrike said.

In response to the incident, which affected more than eight million Windows 10 PCs, CrowdStrike announced that it would improve its Rapid Response content testing and add additional validation checks to specifically prevent the spread of such content.

In addition, the company will improve the error handling it already has in place in its deployments and, perhaps most importantly, stagger the delivery of Rapid Response Content and give customers control over the process “by allowing granular choice over when and where these updates are deployed.”

“In addition to this preliminary post-incident review, CrowdStrike is committed to making the full root cause analysis public once the investigation is complete,” CrowdStrike added.

David Hollingworth

David Hollingworth has been writing about technology for over 20 years and has worked for a variety of print and online titles throughout his career. He enjoys covering cybersecurity, especially when he can talk about Lego.