Explanation of the Falcon Content Update for Windows hosts

Updated 5:39 a.m. ET, July 20, 2024

CrowdStrike is actively working with customers affected by a defect found in a single content update for Windows hosts. Mac and Linux hosts are not affected. This was not a cyberattack.

The issue has been identified, isolated, and a fix has been deployed. We direct our customers to the Support Portal for the latest updates and will continue to provide complete and continuous public updates on our blog.

We also recommend that organizations ensure they communicate with CrowdStrike representatives through official channels.

Our team is fully mobilized to ensure the security and stability of CrowdStrike customers.

We recognize the seriousness of the situation and deeply apologize for any inconvenience and disruption. We are working with all affected customers to ensure systems are back up and running so they can provide the services their customers count on.

We assure our customers that CrowdStrike is operating normally and this issue does not impact our Falcon platform systems. If your systems are operating normally, installing the Falcon sensor will not impact your protection.

Below is the latest CrowdStrike Tech Alert with more information about the issue and workarounds organizations can take. We will continue to provide our community and the industry with updates as they become available.

Summary

CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon sensor.

details

Symptoms include hosts experiencing a bugcheck/bluescreen error related to the Falcon sensor.
For unaffected Windows hosts, no action is required because the problematic channel file has been restored.
Windows hosts brought online after 05:27 UTC are also not affected.
This issue does not affect Mac or Linux based hosts
The channel file “C-00000291*.sys” with a timestamp of 0527 UTC or later is the reverted (good) version.
The channel file “C-00000291*.sys” with the timestamp 0409 UTC is the problematic version.

Note: It is normal for multiple “C-00000291*.sys files to be present in the CrowdStrike directory – as long as one of the files in the folder have a timestamp of 0527 UTC or later, which is the active content.

Current promotion

CrowdStrike Engineering has identified a content delivery issue related to this issue and has reverted these changes.
If hosts continue to crash and cannot stay online to receive the channel file changes, the following steps can be used as a workaround.
We assure our customers that CrowdStrike is operating normally and this issue does not impact our Falcon platform systems.If your systems are functioning normally, your protection will not be affected with the Falcon sensor installed. Falcon Complete and OverWatch services will not be interrupted by this incident.

Query to identify affected hosts using advanced event search

Please read this KB article: How to identify hosts potentially affected by Windows crashes (PDF) or sign in to view it on the Support Portal.

dashboard

Similar to the query above, a dashboard is now available showing affected channels and CIDs, as well as affected sensors. Depending on your subscriptions, it will be available in the console menu under one of the following options:

Next-GEN SIEM > Dashboard or;
Investigate > Dashboards
Named as: hosts_possibly_impacted_by_windows_crashes

Note: The Dashboard cannot be used with the Live button

Automatic recovery articles:

Please read this article: Automatically recover from a blue screen on Windows instances in GCP (pdf) or sign in to view it on the Support Portal.

Workaround steps for individual hosts:

Restart the host to give it a chance to download the restored channel file. We strongly recommend putting the host on a wired network (rather than WiFi) before rebooting, as Ethernet allows the host to connect to the Internet much faster.
If the host crashes again, then:

Start Windows in Safe Mode or Windows Recovery Environment

NOTE: Putting the host on a wired network (as opposed to WiFi) and using Safe Mode with Networking may help resolve the issue.

Navigate to the directory %WINDIR%\System32\drivers\CrowdStrike

By default, Windows recovery occurs via X:\windows\system32

First, navigate to the appropriate partition (default is C:\) and navigate to the Crowdstrike directory:

C:
CD Windows\System32\Drivers\Crowdstrike

Note: Under WinRE/WinPE, navigate to the Windows\System32\drivers\CrowdStrike directory of the operating system volume

Find the file “C-00000291*.sys” and delete it.

Not Delete or modify any other files or folders

Cold boot of the host

Shut down the host.
Restart the host from a powered off state.

Note: BitLocker encrypted hosts may require a recovery key.

Workaround steps for public cloud or similar environments, including virtual:

Option 1:

Detach the operating system disk volume from the affected virtual server.
Before proceeding, create a snapshot or backup of the disk volume to avoid unintentional changes.
Attach/mount the volume to a new virtual server
Navigate to the directory %WINDIR%\System32\drivers\CrowdStrike
Find the file “C-00000291*.sys” and delete it.
Disconnect the volume from the new virtual server
Reattach the fixed volume to the affected virtual server.

Option 2:

Back to a snapshot before 04:09 UTC.

AWS-specific documentation:

Azure environments:

User access recovery keys in the Workspace ONE portal

When this setting is enabled, users can retrieve the BitLocker recovery key from the Workspace ONE portal without contacting the help desk. To enable the recovery key in the Workspace ONE portal, follow the next steps. For more information, see this Omnissa article.