Log4j Detection and Response Playbook

December 13, 2021

On December 09, 2021, a severe vulnerability for Apache Log4j was released (CVE-2021-44228). This vulnerability, also known as Log4Shell, allows remote code execution in many applications through web requests and without authentication. Almost immediately, many attackers on the Internet began to scan and exploit this vulnerability.

This is meant to provide guidelines and recommendations on how to prevent, detect, and mitigate this vulnerability. In general, TrustedSec recommends organizations perform the following:

  • Actively scan systems or use software inventories to identify vulnerable versions of Log4J
  • Update vulnerable versions of Log4j or apply mitigations
  • Search for exploitation and post-exploitation activities

TrustedSec has modified sections in this blog post to accurately depict the evolving information associated with the Log4j threat. For ongoing real-time updates, it is recommended to monitor critical updates by the Cybersecurity and Infrastructure Security Agency (CISA).

1.1      Affected Versions

Log4j version 1 does not appear to be directly vulnerable, but it is end-of-life (EOL) and has not received any updates since 2015 and is also affected by separate CVEs.

The following CVEs provide additional details for the vulnerabilities associated with specific versions of Log4j:

  • CVE-2021-44228 (RCE vulnerability in v2.0 through v2.14.1)
  • CVE-2021-45046 (RCE vulnerability in v2.0 through v2.15.0)
  • CVE-2021-4104 (RCE vulnerability in Apache Log4j 1.2)
  • CVE-2021-45105 (DoS vulnerability in 2.0-beta9 to 2.16.0)

1.2      Affected Software

While Log4j is maintained by Apache, it is utilized in many vendor applications and appliances as well as in custom built systems. The following reference lists the known affected vendors as of December 12, 2021 but should not be considered definitive. Organizations should contact vendors directly for additional information.

2    Vulnerable Software Detection

Proactive scanning for vulnerable software occurs in two (2) ways: searching for vulnerable code and active scanning of deployed code.

2.1      Searching for Vulnerable Code

Searching for vulnerable code is performed by examining all servers and applications for vulnerable instances of Log4j. Because Log4j could be buried in a Java class, a simple search for Log4j will not suffice and additional methods are needed.

If clients are not currently utilizing application scanning tools, there are two (2) open-source scanning programs available to search servers and list out code versions or vulnerable code:

These tools should be run on each server that is suspected of having vulnerable instances of Log4j installed.

2.1.1     Vendor Notifications

Many vendors are releasing bulletins that state if their code or appliances are vulnerable and how to apply upgrades. A list of known vulnerable vendors can be found at https://gist.github.com/SwitHak/b66db3a06c2955a9cb71a8718970c592 but should not be considered definitive. It is recommended to contact all applicable software vendors for insight into their remediation recommendations.

2.2      Active Scanning of Deployed Code

Vulnerability scanners, such as Nessus, have plugins that can actively scan a server and attempt to validate if the vulnerability exists. Public websites have also been set up to do minimal testing against an environment:

Note that vulnerability scanners typically do not check every input with a web application and may not perform authenticated scans. In-depth application assessments may be required to fully determine if a server is vulnerable.

The best project that we have seen so far to scan for the vulnerability is FullHunt’s log4j-scan.

3    Prevention and Mitigation

The best way to prevent exploitation of vulnerable code is to upgrade vulnerable versions of Log4j to version 2.17.0 or apply vendor patches. Although version 2.16.0 completely removed JNDI functionality from Log4j, it was identified that Apache Log4j2 versions 2.0-alpha1 through 2.16.0, excluding 2.12.3, did not protect from uncontrolled recursion from self-referential lookups (CVE-2021-45105).

Alternatively, this infinite recursion issue can be mitigated in configuration:

  • In PatternLayout in the logging configuration, replace Context Lookups like ${ctx:loginId} or $${ctx:loginId} with Thread Context Map patterns (%X, %mdc, or %MDC).
  • Otherwise, in the configuration, remove references to Context Lookups like ${ctx:loginId} or $${ctx:loginId} where they originate from sources external to the application such as HTTP headers or user input.

Note that only the log4j-core JAR file is impacted by this vulnerability. Applications using only the log4j-api JAR file without the log4j-core JAR file are not impacted by this vulnerability.

If it is not possible to upgrade, there are several mitigation tactics that can be performed. However, (CVE-2021-45046) identifies additional vulnerabilities that make the flag mitigations ineffective in some cases. Therefore, the best mitigation tactic is to upgrade to 2.17.0.

  • For Log4j versions >= 2.10, set the log4j2.formatMsgNoLookups system property to true on both client- and server-side components.

    This can be done in multiple ways:
    • Add -Dlog4j2.formatMsgNoLookups=true to the startup scripts of Java programs.
    • Set the following environment variable: LOG4J_FORMAT_MSG_NO_LOOKUPS=”true”.
  • For Log4j versions from 2.0-beta9 through 2.10.0, remove the JndiLookup class from the classpath. For example:
zip -q -d log4j-core-*.jar
  • Isolate systems into their own restricted DMZs or VLANs.
  • Block all outbound network connections from servers or limit outbound network connections to trusted hosts and network ports.
  • Turn on any endpoint or network security signatures for Log4j exploitation.
  • Monitor networks and servers closely for suspicious or unexpected behavior.

Testing should occur after ALL mitigation steps above have been completed to ensure that vulnerabilities do not persist.

4    Exploitation Detection

Scanning for vulnerable Log4j servers has increased significantly since the vulnerability was identified. Therefore, detection of exploitation attempts and post-exploitation activity is critical. Note that Log4j vulnerable software may be on back-end servers, so it is important for organizations to not only examine externally exposed systems but to also focus on internal systems for exploitation activity.

If any evidence of post-exploitation activity is found, Incident Response procedures should begin immediately.

4.1      Log Analysis

Exploitation attempts can be detected by searching through all log files on a server for JNDI-related exploit strings. While straight searches can be performed, attack strings can be easily obfuscated. The following script takes the obfuscation into account when searching:

Manual searches can also be performed, but the following do not consider any obfuscation attempts. (Reference – https://gist.github.com/Neo23x0/e4c8b03ff8cdf1fa63b7d15db6e3860b)

  • Utilize the following Linux commands to search uncompressed log files in /var/log. Note that /var/log should be changed to include any additional application-specific directories.
$ sudo find /var/log/ -type f -exec sh -c "cat {} | sed -e 's/\${lower://'g | tr -d '}' | egrep -i 'jndi:(ldap[s]?|rmi|dns):'" \;
$ sudo egrep -i -r '\$\{jndi:(ldap[s]?|rmi|dns):/[^\n]+' /var/log
  • Utilize the following Linux commands to search compressed log files in /var/log.
$ sudo find /var/log -name \*.gz -print0 | xargs -0 zgrep -E -i '\$\{jndi:(ldap[s]?|rmi|dns):/[^\n]+'
$ sudo find /var/log/ -name "*.gz" -type f -exec sh -c "zcat {} | sed -e 's/\${lower://'g | tr -d '}' | egrep -i 'jndi:(ldap[s]?|rmi|dns):'" \;

Additional regular expressions that can be used to detect attacks can be found on https://gist.github.com/karanlyons/8635587fd4fa5ddb4071cc44bb497ab6.

4.2      Endpoint Analysis

If an organization has visibility into the activity on their servers, endpoint post-exploitation activity can be discovered through the following methods. Note that many EDR and endpoint security systems have implemented signatures to detect this activity.

  • Suspicious execution of common command line tools used to download files such as: curl, wget, or powershell
  • The creation of suspicious or unexpected programs or services on an endpoint
  • An increase in CPU and memory usage on a server (This is due to many attackers placing cryptominers on exploited systems.)
  • Endpoint security software generating alerts for post-exploitation tool usage or activity

4.3      Network Analysis

Once an attack is successful, the victim system will often reach out to the attacker’s staging server to download malicious code. Netflow data can significantly assist with the identification of this communication at the network layer or by utilizing EDR telemetry at the endpoint layer. Therefore, the following network analysis can be performed to look for post-exploitation activity:

5    Additional Resources

The security community has quickly come together to gather information and provide guidance for this vulnerability. Following is a list of additional resources that may also be useful:

  • Browse by Category

  • Clear Form