From Vulnerability to Victory: An Ivanti Vulnerability Case Study

[dennisy]

From Vulnerability to Victory: An Ivanti Vulnerability Case Study

In January of 2024, a US government agency initiated an investigation with CPIRT following suspicious RDP (Remote Desktop Protocol) connections to one of their devices. This event occurred in the wake of two critical CVEs (CVE-2023-46805 and CVE-2024-21887) potentially impacting their Ivanti VPN. The customer’s team, along with the Check Point Incident Response Team (CPIRT), embarked on a forensic analysis to assess the extent of the breach.

In this CPIRT blog post, we will dive into the vulnerabilities exploited, the actions of the threat actor, the investigation logs, and the lessons learned.

Understanding the CVE

On January 10, 2024, Ivanti publicly announced CVE-2023-46805 and CVE-2024-21887 affecting Ivanti VPN Connect Secure. When exploited together, they enable threat actors to execute commands and bypass authentication controls. Threat Intelligence reports document threat actors using these capabilities to drop web shells, backdoors, and credential harvesters onto the Ivanti appliance. Because there is a publicly available proof of concept (POC) for both vulnerabilities, there has been an increase in the activity of automated exploits by threat actors. With that said, there have been no reports that the exploits are used in targeted attacks.

Going on a hunt

In this incident, the customer was contacted via email by a security researcher informing them that their Ivanti appliance was vulnerable to CVE-2023-46805 and CVE-2024-21887. In the email, the researcher attached a screenshot from the vulnerability indexing search engine LeakIX, showing the appliance was vulnerable to the public POC for the exploit. This notification arrived after the customer had already applied the mitigation file provided by Ivanti. However, when the customer subsequently upgraded the appliance, the mitigation file did not persist. Following this notice, the customer’s IT admins examined the firewall logs and noticed suspicious Remote Desktop Protocol (RDP) activity happening in the very early hours of the morning.

CPIRT found in our forensics process, that starting from the compromised Ivanti appliance, the threat actor was able to establish unauthorized RDP sessions, gaining access to multiple devices within the customer’s network such as file servers, admin workstations, and the domain controllers. When CPIRT followed the trail of RDP logins, we observed that the RDP sessions were originating from a device assigned an IP belonging to the VPN’s DHCP pool. Because the customer was using Domain Administrator level credentials to authenticate through the VPN, the threat actor was able to obtain Domain Admin creds and gain unfettered access to the customer’s network.

 The forensic investigation conducted by CPIRT revealed a timeline of events, detailing the threat actor's activities across various devices within the customer’s network. This included devices accessed, folders and files accessed, as well as programs launched.

Ivanti VPN logs were a key indicator in helping to pinpoint the Threat Actor's IP address, as well as correlate attacker activity via account authentication preferences. Windows Security logs were also relevant as authentication logs helped expose the threat actor’s computer name, which was verified as not legitimate by the customer. CPIRT deployed a forensic agent throughout the customer’s environment that allowed CPIRT to hunt for indicators of compromise such as the attacker’s computer name, as well as RDP events using known compromised accounts or during the attacker's working timeframe.

Thankfully for the customer, the forensic investigation concluded that while the threat actor accessed user folders, there was no evidence of exfiltration, encryption, or installed persistence. It is likely the threat actor wanted to return at a later time for actions on objectives. After the customer rolled credentials and correctly remediated the VPN vulnerabilities, CPIRT observed in the VPN logs that the Threat Actor attempted to log in with valid accounts but got themselves locked out. CPIRT monitored the EDR logs and ran several forensic scans that indicated the threat actor was no longer in the customer’s network, and concluded that their earlier attempts at logging into the VPN were their only other way of getting back in.

Incident Response Life Cycle

CPIRT operates as an incident response consulting service, as such, we follows the 6-step Incident Response pathway. Below is a summary of incident response steps taken, minus preparation, throughout the engagement:

Identification:

• Identifying compromised accounts and devices: In this incident, the customer had already identified suspicious Remote Desktop connections at off hours by observing the firewall logs. Using the forensic agent, CPIRT was able to grab a forensic image of the source and destination device to identify which account was being used to move laterally throughout the customer’s network. CPIRT could then hunt for devices utilizing this account, as well as for other RDP events that happened during off hours. These events were then verified with the user and confirmed illegitimate.

Containment:

• Containing the VPN: Bringing down the VPN was a calculated decision that was approved by the customer’s IT admins and management. As the RDP events were coming from a device assigned an IP from the VPN DHCP pool (along with the critical CVE being released), CPIRT had high confidence the threat actor was intruding from the VPN. 
• Quarantining devices affected by lateral movement: Using the forensic agent, CPIRT was able to quarantine devices remotely as forensics was underway.

Eradication:

• Patching the VPN: Rolling back the VPN to a known clean state and then applying the patch ensured that the device was not compromised when the patch was applied. This method would negate the ability of the threat actor to reuse their initial access vector.
• Rolling Credentials: Since the initial access vector was the VPN, CPIRT recommended proceeding under the assumption that all user credentials had been compromised. With that in mind, an organization-wide credential reset was issued. After the VPN was rolled back to a clean state and the patch was applied, CPIRT witnessed the threat actor attempting to authenticate with service accounts.
• Resetting KRBTGT twice: If a Domain Admin account is compromised, it can potentially be used to create forged Kerberos tickets, which can then be used to authenticate to any service within the domain. By resetting the KRBTGT account password, existing Kerberos tickets become invalid, thus mitigating the immediate risk. Resetting it twice ensures any potential lingering sessions or tickets are forcefully terminated.

Recovery:

• Bringing appliances and devices back online: Once the VPN was rolled back, the patch applied, and the Ivanti-issued External Integrity Checker tool came back clean, the VPN was set to come back online.

Lessons Learned:

• Continuous Vulnerability Management: Prompt patching and mitigation strategies are essential to address known vulnerabilities promptly. Regularly updating systems and applications can minimize the risk of exploitation by threat actors.
• Enhance Monitoring and Detection: Proactive threat hunting and real-time monitoring are critical for detecting and responding to suspicious activities effectively. Implementing robust logging mechanisms, including log retention policies, enables organizations to track and analyze events across the network for signs of compromise. In this case, the default size of the Windows Security Event Logs was a limiting factor. Had the Event Logs been centralized, a more sophisticated analysis of historic events could have been conducted.
• Access Control and Privilege Management: Implementing separation of duties (SoD) schema and enforcing strong password policies can mitigate the risk of unauthorized access and privilege escalation. Restricting the use of privileged accounts to specific tasks enhances security posture.
• Incident Response Readiness: Organizations must maintain comprehensive incident response plans and conduct regular exercises to ensure preparedness for cyber incidents. Effective incident response procedures facilitate timely containment, eradication, and recovery efforts. Creating a forensic image of a device before rolling it back to a clean state is crucial for preserving digital evidence and facilitating thorough forensic analysis.