I have taken some flak over the years for rolling my own basics like logging, telemetry, tracing, authentication, session management, etc...
Incidents like this remind me that it is sometimes OK to ignore those who constantly order you to "vendor it out" over some notion of principled development excellence wherein one never reinvents a single hypothetical wheel. Arguably, writing text to a log file on disk is a simple thing you dont want to keep implementing over and over. But, it is also a simple thing that doesn't take a whole lot to do correctly in many use cases.
DIY (in aggregate) would represent one of the largest barriers to any attacker producing a wormable/DDOS exploit of this log4j-style vulnerability. Imagine ten thousand plus unique implementations for how to get a string to disk and I can almost assure you that not one of them would contain an attack vector like this. Even if one of them did, the other 99.999% of implementations would be unaffected and nothing would have made the headlines.
Depending on the code paths that others have developed is the biggest liability in all of software engineering. We are going to have to get better at evaluating this cost relative to the benefits. This doesnt stop with Log4j. There are other things more ubiquitous and more complicated out there still. IMO, the best we can do is to stop producing new software that is exposed in these ways.
Well it sure does suck to find out that you have a RCE vulnerability in your application thanks to a dependency, but I’m not sure that means it was wrong to decide to use Log4j based on the information available at the time. Having to rewrite a bunch of library code in-house is a significant burden, without a guarantee you’ll get it any more right than the Log4j developers did.
I don’t think this event is sufficiently bad to overturn the principle that code reuse is a good thing.
The better lesson is to learn how to contain the behaviour of your dependencies so you don’t end up with ridiculous situations like backend database servers being permitted to make JNDI calls to the public internet.
On the other hand, you don't know what vulnerabilities are hidden in your own code and just haven't been discovered yet, and you're one fuzzing attack away from a vulnerability being discovered/exploited.
On the other hand, if you roll your own logging but import a dependency for something else (you generally can't roll your own everything) and that dependency uses log4j, isn't that strictly worse from a security perspective than just using log4j everywhere? Whatever the probability of any given logging utility exposing an RCE exploit, the probability that none of your logging utilities expose an RCE exploit decays exponentially to zero as the number of unique utilities you're using increases.
Given many of the engineers i have worked with and much more importantly the PMs and Managers that work over them, I would expect vulnerabilities like this to proliferate all over the place if people were to roll their own logging and telematry and especially authentication.
I mean SQL injection is such an easy known mitigation yet is still on the OWASP top 10 even after so many years.
I'm not sure that an argument from monoculture is the right way to go here. It's merely a form of security by obscurity, and probably shouldn't be the dominant factor in whether or not you take some amount of mythical man hours to roll your own code vs. adopting a piece of very well established OSS that has had countless eyes on it over the years.
To be fair, log4j offers more than to write a file on the disk. Logging is also quite relevant for performance as these I/O operations are a bottleneck in many applications.
I think we cannot prevent bugs like this, but mitigation has to become easier.
After reading about this Log4j thing I realize it's not a bug. The vulnerability is due to a feature being exploited by the attacker. I'm not even going to qualify that by saying "exploited in unexpected ways" because it's being used exactly as intended. It seems the people who added and use that feature never took the time to ask "how can this be used nefariously?" That's not a criticism, most developers focus on how something is going to do what they want and not really consider ways to abuse it. But this was documented and in plain sight if you think about it.
In cases like this I think it's usually a collection of changes that result in the vulnerability. Someone adds support for format strings in a config. Then later someone else says "hey, what if we expanded format string support so that they can be interpolated in log strings" and at that time this didn't present a risk since the strings didn't support commands. Then someone else says "wow, what if we added support for commands to our format strings", not realizing that they now are also interpolated in log strings.
I'm not saying this is exactly what happened, but I've seen it often enough.
That should be taken as a criticism. Security is critically missing from consideration in many designs.
In my experience, many teams have a "security" person that they rely on to make security judgements. That's broken. Security should be thought about by everyone, especially in a world with so many malicious actors (and growing)
Does this routing happen before or after log4j would be exploitable? Like if log4j is still running its parser and doing all the crazy stuff, but then routes to some other logger, it doesn't matter.
Not a Java programmer, I still don't really understand how the ability to inject a JNDI lookup to your sever results in an RCE.
OK, an attacker can inject a JNDI lookup, and can control what the JNDI lookup via LDAP returns... I would have thought that whatever it returned would just get inserted as a string into the log output, no big deal. But a remote JNDI lookup can return code that gets executed on the local site somehow? Can anyone explain this?
The main design mistake in log4j is that it performs ${…} substitution on the complete log message by default. There’s no way for application code using the log4j front-end API to have input data containing such strings be logged verbatim, without nonstandard log4j back-end configuration (someone correct me if there’s actually a way). In principle, each application has to sanitize all strings to be logged by stripping/escaping all log4j substitution syntax from them, to prevent unexpected substitutions.
Personally I'd like to see more runtimes with some concept of capabilities. As a user, I'd like to be able to say "this package cannot make outbound calls" rather than doing so at the program or host grain via layer 3 or layer 7 firewall rules. I think maybe Deno allows this, and I wouldn't be surprised if JVM or .Net did as well.
Failing that, it would be interesting to see this implemented as a linter--low level I/O and HTTP functions are annotated such that callers must have the appropriate capability annotation in order to call them. This wouldn't prevent a nefarious library publisher from making HTTP requests through an FFI or an alternate HTTP library unknown to the linter, but it would prevent attackers from exploiting bugs/misfeatures in libraries as in this log4j case.
> Ideally, log4j should not have the ability to make "outgoing calls" right?
What is the intended scope of log4j? I spent some time looking over the source code and cannot fathom why something that logs information would need to become this complicated.
Apparently the intention was to add the ability to parse random crap one might wish to put in config files, including things like pulling a string out of a JNDI reference, system variables (maybe environment variables?); the person making the change did not understand that the scope extended into actual log printing as well.
I probably spend more time grepping dependencies than googling for fixes. Scope kills.
That would help but it isn't enough. You can still load classes from your class path, and then those would make the calls instead. And log4j does have valid reasons to read/write to files, naturally.
But yes, I think a major issue here is a logging library packaging its own meta language that can trigger side effects.
> Ideally, log4j should not have the ability to make "outgoing calls" right?
If it's in the application execution context, it will have the same abilities.
If your middleware needs to make calls to third-parties, and it uses log4j, the log4j code has the same rights as the middleware app.
If you're using a registry like Nexus, and it uses log4j, then it also has the ability to make calls to Maven repos, pypi repos, and anything else that Nexus can reach.
And so on and so forth.
Essentially you're expecting there to be privilege separation, where there is none.
I'm not a security researcher, so I'd like to know if (probably "why") this is a bad idea:
It seems like there should be some intermediary stage between logging the vulnerability & full public disclosure. Something like a partial public disclosure that says "Hey, we have a CVSS 10 about to come down on Log4j, everyone carve out a chunk of time to get their systems fixed when we publish to full disclosure & patch."
And security & sysadmins around the world can get ready to work before exploits can get too much of a hold.
Site author here. Thanks for helping to get the word out - every bit may help someone.
While I consider my page to provide useful color, and I validate and summarize and cache info updates locally to add value ... it won't scale for long. It's really a stopgap - to buy defenders time until better efforts emerge.
A few efforts likely to become higher leverage than mine, because they can be driven by pull requests:
Last night, we kept getting disconnected from HyPixel on Minecraft 18.1 client. I wonder if they check for the vulnerability and boots people? Although 18.1 should be fixed. When we added the log4j JVM flag mitigation, we stopped getting booted.
Graylog was affected. Patched soon after the notice went public and not even a handful of hours later I saw some hits on the public web docker runner. Feel like we just barely dodged this one.
I was going to talk about software engineering being different because you have to build like you're in a nuclear war all the time. But, this sort of thing happens in other industries. Remember asbestos or leaded fuel, or CFC's.
I think doing a retro on the feature and asking why a logging systems needs to have so many features may be a question worth answering. Because there's tons of dependencies I use, where I want 15 to 10% of the features and the rest is just the library developers building hyper niche use cases or avoiding other work. So I expect there's a ton of other features waiting for this kind of CVE to emerge.
I'm curious about the disclosure of this vulnerability and why it seems that no prior notice appears to have been given by the researcher who published it. Is it because it was already being exploited in the wild?
My workplace uses a vendor potentially impacted by ransomware as the result of log4j. When asked, the vendor-- Kronos-- did not deny it as the issue [0], and we ourselves have not received any more substantive answer. Although they may not yet know the attack vector so I wouldn't read too much into it yet. However, regardless of the Kronos cause, I will not be surprised if the next year or so brings a strong uptick in ransomware.
(As a side note, this isn't a great time-- right before Christmas-- to have issues that could impact paying employees on time. Fortunately my workplace doesn't have many of our employees in in this system and Business Continuity can be achieved more easily by temporarily using plain old paper & manual processing than spending time on a temporary technical work around)
It seems to have worm like capabilities. Right now it seems to spread. Who knows what will happen after? It's a bit of race against the clock. I found this library to be used somehow.
https://github.com/projectdiscovery/interactsh
They seem to exfiltrate data. If you see these files hosted in your projects, then you are probably part of it now.
You read it right. Just a simple curl can infect your system. Your api can be behind multiple layer of security and still it will reach log4j and infect your system.
You should never be logging user input non-escaped anyway. At the very least, that creates a hazard where someone might be fooled as to the correct behavior/false-internal-systems-corroboration by logging something misleading that masquerades as a legitimate log message. It is a _huge_ hazard.
Log4J version 1.x met end-of-life in 2015 [1]. Of course, that's only 6 years ago, so there are still projects out there running it. It would look for -Dlog4j.format... and would not know about -Dlog4j2.format... or, hypothetically, -Dlog4j3LTS.format...
Environment variables targeting Log4J version 2.x should get the prefix "log4j2.*" [2] so the latter is correct.
In the end it doesn't really matter if you declare an environment variable that's never read, so you could define both if you're not sure which version of Log4J is used in your stack.
Strange the cheat-sheet does not list setting the environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS=true as a mitigation, which supposedly works for log4j versions >=2.10.
> protects against remote code execution by defaulting "com.sun.jndi.rmi.object.trustURLCodebase" and "com.sun.jndi.cosnaming.object.trustURLCodebase" to "false".
[+] [-] bob1029|4 years ago|reply
Incidents like this remind me that it is sometimes OK to ignore those who constantly order you to "vendor it out" over some notion of principled development excellence wherein one never reinvents a single hypothetical wheel. Arguably, writing text to a log file on disk is a simple thing you dont want to keep implementing over and over. But, it is also a simple thing that doesn't take a whole lot to do correctly in many use cases.
DIY (in aggregate) would represent one of the largest barriers to any attacker producing a wormable/DDOS exploit of this log4j-style vulnerability. Imagine ten thousand plus unique implementations for how to get a string to disk and I can almost assure you that not one of them would contain an attack vector like this. Even if one of them did, the other 99.999% of implementations would be unaffected and nothing would have made the headlines.
Depending on the code paths that others have developed is the biggest liability in all of software engineering. We are going to have to get better at evaluating this cost relative to the benefits. This doesnt stop with Log4j. There are other things more ubiquitous and more complicated out there still. IMO, the best we can do is to stop producing new software that is exposed in these ways.
[+] [-] jl6|4 years ago|reply
I don’t think this event is sufficiently bad to overturn the principle that code reuse is a good thing.
The better lesson is to learn how to contain the behaviour of your dependencies so you don’t end up with ridiculous situations like backend database servers being permitted to make JNDI calls to the public internet.
[+] [-] Johnny555|4 years ago|reply
[+] [-] tfehring|4 years ago|reply
[+] [-] hpoe|4 years ago|reply
I mean SQL injection is such an easy known mitigation yet is still on the OWASP top 10 even after so many years.
[+] [-] ineedasername|4 years ago|reply
[+] [-] neximo64|4 years ago|reply
[+] [-] unknown|4 years ago|reply
[deleted]
[+] [-] mro_name|4 years ago|reply
[+] [-] raxxorrax|4 years ago|reply
I think we cannot prevent bugs like this, but mitigation has to become easier.
[+] [-] phkahler|4 years ago|reply
[+] [-] bink|4 years ago|reply
I'm not saying this is exactly what happened, but I've seen it often enough.
[+] [-] beebmam|4 years ago|reply
In my experience, many teams have a "security" person that they rely on to make security judgements. That's broken. Security should be thought about by everyone, especially in a world with so many malicious actors (and growing)
[+] [-] taeric|4 years ago|reply
[+] [-] throw_m239339|4 years ago|reply
It's an exploit and shouldn't be a "feature" at first place.
[+] [-] layer8|4 years ago|reply
But it’s often still possible to use a different backend, via the Log4j to SLF4J adapter: https://logging.apache.org/log4j/2.x/log4j-to-slf4j/
[+] [-] shartte|4 years ago|reply
[+] [-] staticassertion|4 years ago|reply
[+] [-] jrochkind1|4 years ago|reply
OK, an attacker can inject a JNDI lookup, and can control what the JNDI lookup via LDAP returns... I would have thought that whatever it returned would just get inserted as a string into the log output, no big deal. But a remote JNDI lookup can return code that gets executed on the local site somehow? Can anyone explain this?
update this is the best explanation I've found: https://mbechler.github.io/2021/12/10/PSA_Log4Shell_JNDI_Inj...
[+] [-] ThinkBeat|4 years ago|reply
It has been years since I used it but I can't remember a valid reason why it should be allowed to do so.
Since it is running inside the stack it may not be easy to enforce it. If the "client" log4j does it before even logging that is a bother.
It seems like having a "central" "syslog" logging server.
Traffic goes from the stack -> logging server. Logging server not allowed to call anyone external.
[+] [-] layer8|4 years ago|reply
[+] [-] throwaway894345|4 years ago|reply
Failing that, it would be interesting to see this implemented as a linter--low level I/O and HTTP functions are annotated such that callers must have the appropriate capability annotation in order to call them. This wouldn't prevent a nefarious library publisher from making HTTP requests through an FFI or an alternate HTTP library unknown to the linter, but it would prevent attackers from exploiting bugs/misfeatures in libraries as in this log4j case.
[+] [-] bob1029|4 years ago|reply
What is the intended scope of log4j? I spent some time looking over the source code and cannot fathom why something that logs information would need to become this complicated.
[+] [-] kerblang|4 years ago|reply
I probably spend more time grepping dependencies than googling for fixes. Scope kills.
[+] [-] staticassertion|4 years ago|reply
But yes, I think a major issue here is a logging library packaging its own meta language that can trigger side effects.
[+] [-] rodgerd|4 years ago|reply
If it's in the application execution context, it will have the same abilities.
If your middleware needs to make calls to third-parties, and it uses log4j, the log4j code has the same rights as the middleware app.
If you're using a registry like Nexus, and it uses log4j, then it also has the ability to make calls to Maven repos, pypi repos, and anything else that Nexus can reach.
And so on and so forth.
Essentially you're expecting there to be privilege separation, where there is none.
[+] [-] jcadam|4 years ago|reply
[+] [-] ineedasername|4 years ago|reply
It seems like there should be some intermediary stage between logging the vulnerability & full public disclosure. Something like a partial public disclosure that says "Hey, we have a CVSS 10 about to come down on Log4j, everyone carve out a chunk of time to get their systems fixed when we publish to full disclosure & patch."
And security & sysadmins around the world can get ready to work before exploits can get too much of a hold.
[+] [-] royce|4 years ago|reply
While I consider my page to provide useful color, and I validate and summarize and cache info updates locally to add value ... it won't scale for long. It's really a stopgap - to buy defenders time until better efforts emerge.
A few efforts likely to become higher leverage than mine, because they can be driven by pull requests:
* https://github.com/NCSC-NL/log4shell - already quite comprehensive
* Whatever CISA may spin up - https://github.com/cisagov/log4j-affected-db
* Kevin Beaumont (@GossiTheDog) - turns out he worked with CISA on this
That being said, I'll keep working on mine as long as it still provides value; updates/corrections welcome.
[+] [-] jbverschoor|4 years ago|reply
[+] [-] srcreigh|4 years ago|reply
Biggest one is probably the Minecraft client
> Minecraft users were using it to execute programs on the computers of other users by pasting a short message in a chat box.
https://www.ctvnews.ca/sci-tech/the-internet-s-on-fire-as-te...
Last night, we kept getting disconnected from HyPixel on Minecraft 18.1 client. I wonder if they check for the vulnerability and boots people? Although 18.1 should be fixed. When we added the log4j JVM flag mitigation, we stopped getting booted.
[+] [-] wingmanjd|4 years ago|reply
[+] [-] lostmsu|4 years ago|reply
[+] [-] gonzo41|4 years ago|reply
I think doing a retro on the feature and asking why a logging systems needs to have so many features may be a question worth answering. Because there's tons of dependencies I use, where I want 15 to 10% of the features and the rest is just the library developers building hyper niche use cases or avoiding other work. So I expect there's a ton of other features waiting for this kind of CVE to emerge.
[+] [-] ridaj|4 years ago|reply
[+] [-] ineedasername|4 years ago|reply
(As a side note, this isn't a great time-- right before Christmas-- to have issues that could impact paying employees on time. Fortunately my workplace doesn't have many of our employees in in this system and Business Continuity can be achieved more easily by temporarily using plain old paper & manual processing than spending time on a temporary technical work around)
https://arstechnica.com/information-technology/2021/12/as-lo...
[+] [-] zdwolfe|4 years ago|reply
For example, I want to block all classes from opening sockets except those in my.company.domain.* packages.
[+] [-] john37386|4 years ago|reply
They seem to exfiltrate data. If you see these files hosted in your projects, then you are probably part of it now.
[+] [-] saberdancer|4 years ago|reply
[+] [-] john37386|4 years ago|reply
[+] [-] foobiekr|4 years ago|reply
[+] [-] Droobfest|4 years ago|reply
-Dlog4j.formatMsgNoLookups=true
or
-Dlog4j2.formatMsgNoLookups=true
? Every project seems to list one or the other, even this cheat-sheet seems to list both in a random way...
[+] [-] LeifCarrotson|4 years ago|reply
Environment variables targeting Log4J version 2.x should get the prefix "log4j2.*" [2] so the latter is correct.
In the end it doesn't really matter if you declare an environment variable that's never read, so you could define both if you're not sure which version of Log4J is used in your stack.
[1] https://logging.apache.org/log4j/1.2/
[2] https://logging.apache.org/log4j/2.x/
[+] [-] Droobfest|4 years ago|reply
https://github.com/apache/logging-log4j2/blob/04637dd9102175...
[+] [-] unknown|4 years ago|reply
[deleted]
[+] [-] unknown|4 years ago|reply
[deleted]
[+] [-] jbverschoor|4 years ago|reply
[+] [-] unknown|4 years ago|reply
[deleted]
[+] [-] crescentfresh|4 years ago|reply
[+] [-] throwaway_JiY4E|4 years ago|reply
that's what the CVE (https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-4422...) says:
> protects against remote code execution by defaulting "com.sun.jndi.rmi.object.trustURLCodebase" and "com.sun.jndi.cosnaming.object.trustURLCodebase" to "false".