High Importance: Global IT Outages

Friday 19th July 2024

 
Ellen Hallam
Threat Intelligence Analyst
Author
 
Daniela Miccardi
Cyber Security Marketing Manager
Co-Author

Please Note: this page will be updated regularly with the latest news and information. Please continue to re-visit this site for updates, or reach out to your Bytes Account Manager.

[Wednesday, 24th July, 2024, 1:44PM GMT]

CrowdStrike Technical issues being exploited by Threat Actors: 

Overview:

On Friday 19th July, CrowdStrike Falcon Software Sensor issue impacted more than 8 million computers worldwide, across multiple industries. The issue was caused by a faulty update, which led to a malfunction in the Falcon Sensor, which is a component of the platform that runs locally on User's devices and Servers and scans it for malware. The malfunction resulted in a boot loop on CrowdStrike customers' Windows devices, also known as the blue screen of death (BSOD). This prevented users from starting their computers and running the usual boot-up.  

Although the problem was identified, isolated and a fix rapidly deployed, experts have warned that the process of fixing all the computers affected will continue this week due to the manual effort required to apply the fix to affected devices. Mac and Linux hosts were not impacted.

CrowdStrike handled the situation by following a structured incident response plan. and have since conducted a thorough review of the incident to learn from it and improve their response plan for future incidents.

Analyst Assessment:

Although not a security incident, the effects of the CrowdStrike outage have enabled Threat actors - not least by identifying key infrastructure of organisations (e.g., Gatwick Airport, Sky News and London Stock Exchange, among many others), which provides Threat actors with an advantage. This has effectively given Threat actors free and easy reconnaissance by covering the first phases of the Cyber Kill Chain and MITRE ATT&CK framework, which is how threat actors begin an attack.  Although not always a reputable/accurate source (and not always one to trust), Wikipedia has a page that identifies a significant number of Organisations who were impacted and who are therefore almost certain to be using CrowdStrike: 2024 CrowdStrike incident - Wikipedia. Although the use of CrowdStrike, a highly effective Security tool, could be seen as a deterrent to threat actors, it is highly likely Threat actors will target these businesses and exploit CrowdStrike vulnerabilities in the wild, as soon as a vulnerability is released/identified. 

It is almost certain Hackers will continue to exploit the CrowdStrike Technical issue, whilst Infrastructure continues to be reset due to the recent outage. Organisations with poorer Social Engineering knowledge and training will be more likely to fall victim to scams. A higher level of vigilance against phishing and social engineering attacks is likely to help mitigate attacks.  

Hackers have been exploiting issues with CrowdStrike Falcon in several ways:

1 - Fake CrowdStrike Fixes:
Cyber criminals are distributing fake recovery manuals to deliver malware, specifically the “Daolpu” infostealer, to affected companies. 

2 - Typosquatting Domains:
Threat actors have also created several typosquatting domains impersonating CrowdStrike to deceive users and distribute malicious files.

URLs:

  • Clownstrike[.]co[.]uk
  • Thecrowdstrike[.]com
  • www[.]thecrowdstrike[.]com
  • crowdstrike-hotfix[.]zip
  • crowdstrike-okta[.]quickintuits[.]top

Click here to access the complete list of URLs. (Mimecast)

3 - Phishing Attacks:
The UK’s National Cyber Security Centre (NCSC) has warned of increased phishing attacks related to the CrowdStrike outages. These campaigns use phishing emails to promote counterfeit fixes, which include a fake CrowdStrike Hotfix update.
 
4 - Malware Disguised as Updates:
Attackers are distributing data wiper malware disguised as CrowdStrike updates, which can destroy systems by overwriting files. A notable campaign targeted BBVA bank customers, directing them to a fake website mimicking the bank’s intranet. The promoted fake Hotfix installs HijackLoader, which subsequently deploys the Remcos RAT to compromise the systems further.
Since Friday's CrowdStrike incident, Mimecast has seen that threat actors are using remote access tools and data wipers to target organisations. In relation to these attacks, Mimecast is compiling and sharing a list of malicious or potentially dangerous domains. Mimecast will block all domains, and you should think about using this list to update security across all of your security solutions. Further details can be found here: https://www.mimecast.com/threat-intelligence-hub/crowdstrike-phishing-links/.

Recommendations: 

1 - Always refer to CrowdStrike’s official website, customer portal, or verified social media accounts for updates and guidance.

2 - Be wary of Direct approach contact, as this can be an indicator of phishing, and ensure you verify emails posing as CrowdStrike. Avoid clicking on unsolicited links and report suspicious activity to your IT or Security team.

3 - Develop a Comprehensive Plan: Ensure that your organisation has a well-defined plan for dealing with outages and investigations. This includes having backup systems in place and clearly defined roles and responsibilities, as well as a risk register, identifying key risks, especially third party software on which the Organisation relies.

4 - IT Staff Training: Ensure that your staff are well-trained to respond quickly in worst-case scenarios. This is especially critical if you contract with a third party for maintenance.

5 - User Awareness Training: Ensure that your staff are trained to spot social engineering attempts, especially in the event of a global incident as we have seen.

6 - Establish Communication Channels: Ensure you have the correct processes in place to manage internal and external communication of IT incidents.

7 - Emergency Response Plan: Develop an emergency response plan that includes steps to mitigate disruptions and keep your organisation moving during outages. 

______________________________________

[Wednesday, 24th July, 2024, 11:30AM GMT]

CrowdStrike have recently released the Preliminary Post Incident Review (PIR) of the Content Update impacting the Falcon sensor and causing Windows Operating Systems to BSOD.

Here is a summary of the PIR:

  • On Friday, July 19, 2024 at 04:09 UTC, as part of regular operations, CrowdStrike released a content configuration update for the Windows sensor to gather telemetry on possible novel threat techniques.
  • The defect in the content update was reverted on Friday, July 19, 2024 at 05:27 UTC. Systems coming online after this time, or that did not connect during the window, were not impacted.
  • On July 19, 2024, two additional IPC (InterProcessCommunication) Template Instances were deployed. Due to a bug in the Content Validator, one of the two Template Instances passed validation despite containing problematic content data.
    -> When received by the sensor and loaded into the Content Interpreter, problematic content in Channel File 291 resulted in an out-of-bounds memory read triggering an exception. This unexpected exception could not be gracefully handled, resulting in a Windows operating system crash (BSOD).
  • Out of an abundance of caution, and to prevent Windows systems from further disruption, the impacted version of the channel file was added to Falcon’s known-bad list in the CrowdStrike Cloud.
  • This PIR is preliminary, CrowdStrike is committed to publicly releasing the full Root Cause Analysis once the investigation is complete.

Full details of the PIR can be found here, along with a statement from CrowdStrike Founder and CEO.

Currently, CrowdStrike are continuing to work with impacted customers to ensure all systems are restored. Since the incident, they have provided both manual and automated means of remediation, and have posted content (Blogs, Articles & Videos) to further help customer with remediating this issue.  

______________________________________

[Tuesday, 23rd July, 2024, 3:02PM GMT]

CrowdStrike have released a video that guides through the steps of the manual remediation. 

This video outlines the steps required to self-remediate a remote Windows laptop experiencing a blue screen of death (BSOD) related to the recent defect in a CrowdStrike content update for Windows hosts.

https://www.youtube.com/watch?v=Bn5eRUaMZXk

______________________________________

[Monday, 22nd July, 2024, 11:33AM GMT]

CrowdStrike have recently published an update to further automate and address this issue from the Falcon platform itself - click here for more information.

______________________________________

[Monday, 22nd July, 2024, 10:15AM GMT]

On Friday, Bytes sent out information updating you on the CrowdStrike Falcon Software Sensor issue, which has since impacted more than 8 million computers worldwide, across multiple industries.

CrowdStrike Falcon Sensor Software is designed to prevent cyber-attacks on computer systems. The issue was caused by a faulty update, which led to a malfunction in the Falcon Sensor, which is a component of the platform that runs locally on User's devices and Servers and scans it for malware. The malfunction resulted in a boot loop on CrowdStrike customers' Windows devices, also known as the blue screen of death (BSOD). This prevented users from starting their computers and running the usual boot-up. Because of this, applying an automated fix proved difficult as the system will not be able to boot to receive the working copy of the Falcon Sensor.  

Although the problem was identified, isolated and a fix rapidly deployed, experts have warned that the process of fixing all the computers affected will continue this week due to the manual effort required of applying the fix to affected devices. Mac and Linux hosts were not impacted.

CrowdStrike handled the situation by following a structured incident response plan. and have since conducted a through review of the incident to learn from it and improve their response plan for future incidents.

The latest remediation details are below:

Currently, there is a manual method and automated method to fix affected devices. 

Workaround Steps for individual hosts (Manual):

  • Reboot the host to give it an opportunity to download the reverted channel file.  If the host crashes again, then:
    • Boot Windows into Safe Mode or the Windows Recovery Environment
      • Note:  Putting the host on a wired network (as opposed to WiFi) and using Safe Mode with Networking can help remediation.  
    • Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory
      • Note: On WinRE/WinPE, navigate to the Windows\System32\drivers\CrowdStrike directory of the OS volume
    • Locate the file matching “C-00000291*.sys”, and delete it.
    • Boot the host normally.

Note: Bitlocker-encrypted hosts may require a recovery key.

Full details can be found at this Microsoft Article: KB5042421: CrowdStrike issue impacting Windows endpoints causing an 0x50 or 0x7E error message on a blue screen - Microsoft Support

Using the Microsoft Recovery Tool for Host Remediation (Automated):

Microsoft, in partnership with CrowdStrike, have released a utility to assist with recovering hosts impacted by the issue. This Microsoft signed utility enables IT Admins to create a bootable USB Drive for automated remediation including BitLocker Recovery Key support.

Requirements:

  • A Windows 64-bit client with at least 8GB of free space from which the tool can be run to create the bootable USB drive.
  • Administrative privileges on the Windows client from prerequisite #1.
  • USB drive with (1GB). All existing data on this USB will be wiped.
  • A BitLocker recovery key for each BitLocker-enabled impacted device on which the generated USB device will be used.

Procedure:

  1. Download the utility from Microsoft here
  2. Extract the utility to a working directory
  3. Open PowerShell as an administrator and navigate to your working directory
  4. Run the MsftRecoveryToolForCS.ps1 file
  5. Please wait a few minutes as the utility begins initial Windows PE image creation, including downloading required files from Microsoft
  6. The utility will prompt you to select an option for adding drivers to the WinPE image. Press Y only if you need to add drivers to support specific host hardware. Otherwise, select N to just use the base WinPE drivers.
    -> If you selected Y, enter the path where you have stored INF based drivers and the utility will add them to the WinPE image.
  7. When prompted, insert a USB Drive and provide the drive letter it is assigned by Windows
  8. Once creation is completed, you can remove the USB Drive from the device.
  9. Insert the USB Flash Drive into the target system
  10. Reboot the target system and enter the UEFI boot Menu (usually F1, F2, F8, F11, or F12).
  11. Select the USB Drive from the Boot Menu. If given both a MBR and UEFI option, select UEFI
  12. Wait for Windows PE to load
  13. If prompted, enter your BitLocker Recovery Key to unlock the volume
  14. Let the utility find and remove the impacted Channel File 291 sys file
  15. The utility will report once it's completed and exit Windows PE, which should then reboot the targeted system.
  16. The targeted system should now load Windows successfully.

Public Blog Link for issue details from CrowdStrike: Technical Details: Falcon Update for Windows Hosts | CrowdStrike

______________________________________

[Friday, 19th July, 2024, 3:07PM GMT]

Microsoft Update

"We're continuing to resolve the residual impact and we're monitoring the Microsoft 365 Apps and Services while they fully recover. Customers should experience incremental recovery as we recover the remaining impact".

______________________________________

[Friday, 19th July, 2024, 2:32PM GMT]

CrowdStrike Update

Summary:

  • CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor.

Details:

  • Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor.
  • Windows hosts which have not been impacted do not require any action as the problematic channel file has been reverted.
  • Windows hosts which are brought online after 0527 UTC will also not be impacted
  • Hosts running Windows7/2008 R2 are not impacted.
  • This issue is not impacting Mac- or Linux-based hosts
  • Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version
  • Channel file "C-00000291*.sys" with timestamp of 0409 UTC is the problematic version

Query to identify impacted hosts via Advanced event search: Click Here

Current Action:

  • CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes.
  • If hosts are still crashing and unable to stay online to receive the Channel File Changes, the following steps can be used to workaround this issue:

Workaround Steps for individual hosts:

  • Reboot the host to give it an opportunity to download the reverted channel file.  If the host crashes again, then:
    -> Bootndows into Safe Mode or the Windows Recovery Environment
        ->Note:  Putting the host on a wired network (as opposed to WiFi) and using Safe Mode with Networking can help remediation
  • Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory
    -> Note: On WinRE/WinPE, navigate to the Windows\System32\drivers\CrowdStrike directory of the OS volume
  • Locate the file matching “C-00000291*.sys”, and delete it
  • Boot the host normally

Please Note: Bitlocker-encrypted hosts may require a recovery key.

Workaround Steps for public cloud or similar environment including virtual:

Option 1:

  • Detach the operating system disk volume from the impacted virtual server
  • Create a snapshot or backup of the disk volume before proceeding further as a precaution against unintended changes
  • Attach/mount the volume to to a new virtual server
  • Navigate to the %WINDIR%\\System32\drivers\CrowdStrike directory
  • Locate the file matching “C-00000291*.sys”, and delete it.
  • Detach the volume from the new virtual server

Reattach the fixed volume to the impacted virtual server

Option 2:

Roll back to a snapshot before 0409 UTC. 

AWS-specific documentation:

Azure environments:

Bitlocker recovery-related KBs:

Latest Updates:

  • 2024-07-19 05:30 AM UTC | Tech Alert Published.
  • 2024-07-19 06:30 AM UTC | Updated and added workaround details.
  • 2024-07-19 08:08 AM UTC | Updated
  • 2024-07-19 09:45 AM UTC | Updated
  • 2024-07-19 11:49 AM UTC | Updated
  • 2024-07-19 11:55 AM UTC | Updated
  • 2024-07-19 12:40 PM UTC | Updated, added query

_______________________________________________

[Friday, 19th July, 2024, 11:24AM GMT]

As you are aware, there is a global outage affecting Microsoft Services, which is having an impact on industries from GP surgeries in the UK, to planes, trains and other industries who rely on Microsoft Cloud services. 

There is also another current issue related to Windows Operating Systems caused by CrowdStrike. 

At present, we do not have enough intelligence to suggest that the two are linked, so are treating them as different events.

Bytes will continue to keep you updated as further information emerges and have set up a triage system to support you if required:

Bytes Support Contact Details:

Situation Summary:

  • The outage was caused by an issue with the latest update from CrowdStrike's Falcon Sensor. This update led to "Blue Screen of Death" (BSOD) outages on many Windows devices. 
  • The outage has caused significant disruptions worldwide, affecting banks, airlines, broadcasters and even 911 systems. Many Windows computers have crashed.
  • Both CrowdStrike and Microsoft are working on the issues and in the process of removing the global update. Microsoft is also taking mitigations to repair its Azure servers and remedy the problem for Global Windows Users.

This is a developing story, and more updates are expected as the situation unfolds. Bytes will send out further detail, once more information is available.

In the meantime, please find below suggested remediations from CrowdStrike and Microsoft.

Microsoft

Microsoft are continuing to see an improvement in service capability across multiple M365 apps and services. They are closely monitoring their telemetry data to ensure the upward trend continues, as their mitigation actions continue to progress.

Summary from Microsoft [please note: the below article has been taken directly from a Microsoft source]:

Everything is up and running.

  • Title: Users may be unable to access various Microsoft 365 Apps and Services
  • User Impact: Users may be unable to access various Microsoft 365 Apps and Services
  • More Info:
    -> OneDrive for Consumer: Users may have been unable to access OneDrive for Business content
    -> OneNote: Users may have been unable to sync content, experienced delays syncing notebooks, or may have been unable to open notebooks
  • Final Status: We've confirmed that impact has been resolved following our mitigation efforts.
  • Start Time: Thursday, July 18, at 10:59PM UTC
  • End Time: Friday, July 19, 2024, at 4:21PM UTC

CrowdStrike

Summary:

  • CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor

Details:

  • Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor
  • Windows hosts which have not been impacted do not require any action as the problematic channel file has been reverted
  • Windows hosts which are brought online after 0527 UTC will also not be impacted
  • This issue is not impacting Mac- or Linux-based hosts
  • Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version
  • Channel file "C-00000291*.sys" with timestamp of 0409 UTC is the problematic version

Current Action:

  • CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes
  • If hosts are still crashing and unable to stay online to receive the Channel File Changes, the following steps can be used to workaround this issue:

Workaround Steps for individual hosts:

  • Reboot the host to give it an opportunity to download the reverted channel file.  If the host crashes again, then: 
    -> Boot Windows into Safe Mode or the Windows Recovery Environment
    -> Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory
    -> Locate the file matching “C-00000291*.sys”, and delete it
    -> Boot the host normally

Please Note: Bitlocker-encrypted hosts may require a recovery key.

Workaround Steps for public cloud or similar environment including virtual:

Option 1:

  • Detach the operating system disk volume from the impacted virtual server
  • Create a snapshot or backup of the disk volume before proceeding further as a precaution against unintended changes
  • Attach/mount the volume to to a new virtual server
  • Navigate to the %WINDIR%\\System32\drivers\CrowdStrike directory
  • Locate the file matching “C-00000291*.sys”, and delete it
  • Detach the volume from the new virtual server
  • Reattach the fixed volume to the impacted virtual server

 Option 2:

  • Roll back to a snapshot before 0409 UTC

 Workaround Steps for Azure via serial:

1. Login to Azure console --> Go to Virtual Machines  --> Select the VM
2. Upper left on console --> Click : "Connect" --> Click --> Connect --> Click "More ways to Connect"  --> Click : "Serial Console"
3. Once SAC has loaded, type in 'cmd' and press enter. 
-> type in 'cmd' command
-> type in : ch -si 1
4. Press any key (space bar).  Enter Administrator credentials
5. Type the following: 
-> bcdedit /set {current} safeboot minimal
-> bcdedit /set {current} safeboot network
6. Restart VM
7. Optional: How to confirm the boot state? Run command: 
-> wmic COMPUTERSYSTEM GET BootupState

Latest Updates:

  • 2024-07-19 05:30 AM UTC | Tech Alert Published
  • 2024-07-19 06:30 AM UTC | Updated and added workaround details
  • 2024-07-19 08:08 AM UTC | Updated
  • 2024-07-19 XXXX AM UTC | Updated

AWS has also released a fix for CrowdStrike Customers:


Want to keep informed? Sign up to our Newsletter

Connect