Ironically, on a previous team I had switched our log4j2 log formats over to %m{nolookups} like 8 months ago... I didn't realize the whole jndi issue, what we ran into was the O(n^2) behavior of its string substitution.
While deploying an ancillary change, our jvms started locking up for minutes on end. What was happening was that we were logging customer input, and the change caused it to run certain things in parallel, which ended up logging the data multiple times. Normally the extra logging didn't matter but one customer had data like "${foo} ${bar} ${baz} ...". Even when the ${foo} portion is replaced wothout modification, this triggers quadratic behavior. So we were already potentially vulnerable to the DOS but it was rare enough that we never got locked up until logging the string multiple times, which then overflowed log4js internal buffer and blocked worker threads.
You can try this yourself by just logging a string like "${}${}${}..." And in fairly short order it starts taking forever. I'm very glad the fix in 2.15 is to disable lookups by default.
I hope that in the time after I left, the security org at the big tech company I worked at and reported this to (as I thought it was - a dos vector, not the complete pwnage it actually was) forced teams to switch to nolookups. Otherwise a lot of people had a bad week forcing updates through...
> So we were already potentially vulnerable to the DOS [...]
> the security org at the big tech company I worked at and reported this to
I'm confused about these two statements, because I did not find any recent CVEs for log4j in the DoS category, nor related to format lookup (other than CVE-2021-44228 of course).
Perhaps I misread it, but are you basically saying that (after you reported the issue to them internally) the security team at your previous company could not successfully report a DoS vulnerability in the default configuration of a widely used (by them, at least) Apache library and make sure a CVE got assigned to track it?
If so, it would be interesting to know where the CVE/vuln-reporting chain broke, possibly to reduce the blast radius for similar future cases.
Hypothetically speaking, a CVE in March for a DoS in a problematic design/feature could have resulted in flipping the default setting earlier. Instead of chasing live RCE in the wild in December.
There seems to be a misunderstanding here. We have on the one side a garbage feature that should never have been implemented - but if you want to keep it for backwards compatibility, sure. But then we have log4j scanning all values instead of only format strings - I think it can be argued that this behavior is a critical bug and was never intended to begin with. It seems to have only come about because whoever implemented the JNDI stuff lost their bearing in the absurd class hierarchies and abstractions in log4j.
Of course the last part holds the solution for our backwards compatibility issue. Remove the JNDI nonsense from the default package and move it into an extension package. Whoever wants to keep it can just add that to their dependencies and continue to enjoy logging functions that sometimes also make network connections and block your program.
Indeed - as evidence for this, I would submit that slf4j and logback were created to offer a drop in replacement for log4j (slf4j literally provides alternative implementations of the org.apache.log4j.Logger class), but I have never seen anybody complain that "I switched to logback and slf4j and my jndi substitutions stopped working."
Nobody thought this was how log4j worked; log4j's documentation for format syntax only covers {} placeholders - the same format that slf4j has grandfathered in from log4j.
I agree this feels like a case where they got confused about their internal terminology. Log4j refers to messages with {} placeholders as 'FormattedMessages'; it refers to the log pattern syntax as 'Patterns' in code - but it seems to refer to them as 'log formats' in documentation.
Somewhere in this mess, someone hooked up the pattern capabilities into the formatting system.
> But then we have log4j scanning all values instead of only format strings - I think it can be argued that this behavior is a critical bug and was never intended to begin with.
It was actually intended behavior, and this is what really boggles the mind!
Javadoc says explicitly that variable replacement is recursive, with cycle detection (which will throw! What happens to the log line in this case?) [0].
Right, I was also confused by the blame on backward compatibility. You can keep things backward compatible without necessarily making it on by default. There is no reason why `formatMsgNoLookups` should the default. If it is indeed an obscure and hacky feature for backward compatibility, just make it opt-in. People who really care about it will enable it, most people won't have to carry that baggage and we wouldn't be in a situation like this.
"lost their bearing in the absurd class hierarchies and abstractions" sounds familiar. Java app stack traces are like Neal Stephenson epics, but less entertaining.
And enabled by default. That's the most mind-blowing bit of this feature. The backcompat argument is a deflection for shipping a time bomb into people's codebases.
> for a feature we all dislike yet needed to keep due to backward compatibility concerns.
It's logging. While logging is extremely important, I think we could all tolerate removing a vulnerable feature. Or, just move the feature to a separate package.
I have made bad decisions, we have all made bad decisions. Own them, improve, and celebrate the opportunity to learn and improve. Keeping this around, as a default, was a bad decision. If your enterprise contracts don't want to turn a flag on, then they can always skip upgrading (they generally do regardless).
I can imagine the maintainers being scared of silently breaking the workflow and monitoring for some users. If you change this feature to opt-in, you may silently break the alerting system users built on top of this feature, and then you get the heat for breaking a somebody’s IT system(a hospital maybe), just because you hated that feature. That it had an RCE would not be known at the time.
In a perfect world, the feature would have been an option from the start, but in that same perfect world, the downstream users would be diligent and check release notes before upgrading. You might, but many of your colleagues don’t, they just upgrade, and complain when their system breaks.
One place I worked used syslog to ship important analytics data from services to Kafka. log4j is a reasonable choice for logging to syslog from Java (but let's be honest, you should be on Logback). Now, using jndi as part of this? That's getting a little too clever.
Keeping this around, as a default, was a bad decision.
Definitely. But really, they were screwed once it had shipped. They could and should have disabled it in an update long ago, but then anyone who read the release notes or the code would know how to exploit the millions of un-updated systems.
People and, many companies seem to forget that such software comes "AS IS" and it means, AS IS, I would be glad to see fortune 500 companies try to put together a team providing flawless logging capabilites. In reality I know they would not be able to get to be half as good as an open source library, fist of all drowning developers in unnecessary administrative tasks, imposing stupidly unreasonable deadlines and fully ignoring engineering advice from... well the engineering team. It's an insult that those companies profiting masively from so many open source projects still have the audacity to put blame on (again) software whose premise is "AS IS" specially when if you look at their projects (even the ones they sell to their customers) are basically bullshit put thogether with spit and boogers (and I've work in more than one FAANG to know this is truth by experience)
I'm still flabbergasted that the original maintainers are rushing around trying to patch these problems. Unless their specific personal/professional projects are at risk they have no responsibility to hurry and fix a thing.
You'd think, in the spirit of open source, these multi-billion dollar companies--like Apple and Google and Amazon--would recognize the danger and immediately divert the best engineers they had to help this team identify and mitigate the problems. They should have been buried in useful pull requests.
For that matter, they should have really picked them all up in private jets and flown them to neutral working space with those engineers for a one or two week hackathon/code sprint to clean up the outstanding issues and set the project on a sustainable path. To get those maintainers there they should offer a six figure consulting fee and negotiate with their current employers to secure their temporary help.
I can't believe these folks just get abandoned like this while CEOs/CTOs from rich companies wring their hands wailing about the problems and not offering solutions.
> I'm still flabbergasted that the original maintainers are rushing around trying to patch these problems. Unless their specific personal/professional projects are at risk they have no responsibility to hurry and fix a thing.
Sorry, but what's the hard part to understand? Open source maintainers end up in this position because they are nice, helpful people who like using computers to solve problems for others. People who spend years on a project and then see a bigger problem arise don't suddenly turn that off. With the bigger problem, they'll want to work harder, not just hoist a middle finger and go binge Netflix without a care in the world.
But I totally agree with you on the CTOs, etc. I don't expect random programmers who like working on logging to also be good at solving complicated sociotechological problems around paying for global infrastructure. But it boggles my mind that none of these richly rewarded, supposedly brilliant experts at organizing engineers has gotten out in front of this. If not out of community spirit or social responsibility, then out of pure self interest.
> You'd think, in the spirit of open source, these multi-billion dollar companies--like Apple and Google and Amazon--would (...) mitigate the problems.
FAANG engineer here, and one who had to work extra hours to redeploy services with the log4j vulnerability fix. I'm not sure you understand the scope and constraints of this sort of problem. Log4j's maintainers have a far more difficult and challenging job than FANGs or any other consumer of a FLOSS package, who only need to consider their own personal internal constraints, and if push comes to shove can even force backwards-incompatible changes. The priority of any company, FANG or not, is to plug their own security holes ASAP. Until that's addressed the thought of diverting resources to fix someone else's security issues doesn't even register on the radar. I mean, are you willing to spend your weekend working around the clock to fix my problems? Why do you expect others like me to do that, then? Instead I'm spending a relaxing weekend with my family with the confort of knowing my service is safe. Why wouldn't I?
>> I'm still flabbergasted that the original maintainers are rushing around trying to patch these problems.
Agreed, while reading it I also disagreed at this point:
>> the maintainers of log4j would have loved to remove this bad feature long ago, but could not because of the backwards compatibility promises they are held to.
Nobody is holding them to anything. If they want to remove an old feature, go right ahead. If those using it think it's that important they can fork the project and maintain it themselves. Oh right, that would take effort or money.
Are data breaches actually treated as all that seriously? For all the talk about cyber security, there seems to generally be little investment. It appears to be viewed as more of a reputational concern than an operational one.
A past organization of mine had a data breach (the kind that ended up making the news everywhere). A few people left (probably making it worse with all the turnover there), but I would be surprised if anything really changed in that organization.
> I'm still flabbergasted that the original maintainers are rushing around trying to patch these problems.
If the RCE had been responsibly disclosed instead of via tweets and PR comments, maybe there wouldn't have had to be so much scrambling. And indeed maybe ASF could have found corporate OSPOs to help with remediation.
There are lots pixels being spilled on how the users of open source software should be paying for it (?), but I haven't seen much criticism of the vulnerability not being responsibly disclosed.
I'm not sure about Amazon but Google project's zero and openfuzz teams seem to be doing a lot of good work when it comes to open-source security -- more would be nice always
Personally I'd like something like a security health card/metric on opensource libaries that we could tie into CI systems/pull requests or something
in the past there were so few libarries it wasn't as daunting
I'd be able reason about stuff like libpng, libttf ..etc and think about them or even support them but now some projects are massive hodgepodges of thousands upon thousnads of packages
> You'd think, in the spirit of open source, these multi-billion dollar companies--like Apple and Google and Amazon--would recognize the danger and immediately divert the best engineers they had to help this team identify and mitigate the problems.
Google doesn't even use log4j. What are you talking about? The spirit of open source does not dictate that the richest companies automatically shoulder the burden of maintenance of projects they do not even use. Google already has initiatives like Summer of Code that help open source projects it does not use, and I think it's perfectly fine to draw the line there.
> divert the best engineers they had
So the lessons from the mythical man-month are forgotten here. At this point I don't think adding more manpower helps.
What? It was already fixed. You just need to update. There's no need for the world's top fintech programmers to hack it out on a mountaintop somewhere.
Also, the reason the maintainers are rushing to fix it is: they're worried about losing "market share". Having been in open-source circles for a long time, maintainers care GREATLY about how many users they have. They just like watching their download stats go up every year. Even if it beings them no financial rewards. It's a sort of addiction.
An influx of pull requests is also equally difficult for open source projects.
Anything sufficiently at scale needs a set of maintainers that the commercial tech companies would then collaborate with to get the PRs going.
Otherwise if everyone's just panicking and rushing to submit PRs, they'll inundate the maintainer. There's also no guarantee that even the best engineers at these companies are intimately familiar with the project, and might introduce regressions or other vulnerabilities in the process.
Anyway I do agree companies should be working with OSS devs, but it shouldn't be rushed or knee jerk. It should be collaborative and measured.
> You'd think...these multi-billion dollar companies...would recognize the danger and immediately divert the best engineers they had to help this team identify and mitigate the problems.
For the general case, the problem is that a reporter might report the vulnerability to the open source project, then the project needs to keep it a secret while they make a fix. There isn't a great way to leverage these stakeholders. It's obviously different for something like Android that is open source, but clearly Google.
> I want to not spend much time upgrading a dependency
> Go compatibility promise:
>So whenever a change in behavior happens in an upstream library
You are comparing a promise from language designers to no promise from the library developers. Syntax from Oak (before Java was called Java) still compiles and works in Java 17 right now:
jshell
pub| Welcome to JShell -- Version 17
| For an introduction type: /help intro
jshell> public abstract interface I {}
| created interface I
You can still type (public abstract interface - all interfaces are abstract by default since Java 1) and it works. One of the reasons I gave up on writing desktop applications in Go was libraries were breaking compatibility with every commit. GTK+ binding was literary unusable as before gomod this would break literally, and I mean literally, every day.
Please tell me that none Go library had any breaking changes in the last 5 years and I'm using it as my default ecosystem from tomorrow.
To add some perspective, log4j has gone for 20 years with only two major versions. Assuming that they are following semantic versioning, that means they added new features/fixes in a backwards-compatible way and only broke compatibility _once_ in over two decades. That's both a testament to the stability of the library over time and a reminder that all the cruft accumulated over the years at most gets gated off through saner defaults.
This assumption isn't true, though. APIs routinely get changed in minor versions, which can make it non-trivial to upgrade large codebases that use lots of features.
I doubt anyone would have demanded retention of a feature where log _payload_ could cause the library to punch a connection to a server specified in the payload then _execute_ the response for any reason, never mind backwards compatibility.
By the way, what was the last time we experienced such catastrophic bugs in Python/Erlang/Ruby/Go... libraries? I think simplicity is deeply interconnected with security. Perfection comes from simplicity, and the choice of programming language can and will affect the security of your platform. Although I have to admit, bad engineering and over usage of libraries could happen in every environment, but Java technologies are unnecessarily complex compare to others tech stacks.
When a Javascript logging package has a vulnerability: "Why do you need a package for something so basic as logging? This should be part of JS core lib, or just roll your own."
When a Java logging package has a vulnerability: Sober introspection about the role of maintainers, dependencies, and backward compatibility in the OSS ecosystem.
> This is where software goes wrong the most for me. I want, year after year, to come back to a tool and be able to apply the knowledge I acquired the last time I used it, to new things I learn, and build on it. I want to hone my craft by growing a deep understanding of the tools I use.
This resonates with me deeply. Mastery of any subject needs a level of stability and permanence so that a skill base and a knowledge base doesn't erode away over time. Change is that erosive force, and we're so bad at knowing what changes to make in the software world.
> but complaints would have bubbled up to the Apache umbrella organization, which no doubt has plenty of Words written about "proper" versions, and the letter of rules would have been used to add heavily to their burden, while the spirit of compatibility is ignored.
Tailscale is wonderful software, but with all due respect I don't think David has much experience with Apache Foundation or Eclipse Foundation projects. I am a project lead of one (way smaller) Eclipse Foundation project and keep a close watch on another (bigger, but nowhere near the log4j scale) Apache Foundation project. We are free to make breaking changes (and make them we do). In both Foundations, there is a
Project Management Committee that signs off on releases. They would normally raise an eyebrow if you try to push a breaking change into a patch release, but their reaction would normally be "why don't you bump the major version?".
Edit: there is indeed a social contract between the project leadership and the users. We have a social contract of supporting the oldest possible JVM for our project. Now some of our dependencies (guess what, that ASF project!) dropped Java 8 support and published a few CVEs since and we are going to move to JDK 11 as a baseline soon. Yes, some users are grumpy but people understand that unless someone forks that ASF project and starts backporting CVE fixes, we got to make the move. Bottom line: it's a much more social process than a bureaucratic one.
Sure, people who are making money building on FOSS tools should understand that they bear the burden of fitness for their environment. Maybe they do. If so, they've done a good job of deflecting attention.
Let me tell you a story about something that happened to me this spring. I had trouble logging in to my customer account at a critical infrastructure provider (utility provider). When I called customer support one thing led to another and they ended up reading me my password from their internal systems. (Let that sink in.)
So I spent a week trying to get ahold of someone to let them know how fucked up that is, which they avoided, until ultimately they claimed I was "threatening" them. I reached out to the "hidden hand" of the internet, did anyone have contacts with their MSP (with "National" and [edit: changed from "Infrastructure"] "Information" in the name)? But nobody did, so someone whose name would be recognized offered to tweet it.
Didn't take long to hear from MSP's Chief Cloud Architect. The story that they tell is that they'd like to turn the feature off but the customer won't let them.
The last part advises to be feature conservative to avoid promises all together. For that look to enterprise software where backwards compatibility is a legal agreement. This carries a bunch of alternative issues.
I have used logging all my life (2 decades career) and even log4j at times, but I can’t remember ever logging any messages that were expanded by the logging runtime (rather than at compile time or by the language runtime). Have I been missing out on anything important? Of course the configured log pattern is expanded like log level and date, but if I understand correctly this expansion can take place in log messages?
Yes — simple pattern expansion is pretty handy. For example:
log.debug(“Added {} records”, added.size());
In this example, the cost of string concatenation / interpolation can be avoided when debug-level logging isn’t enabled, which is a lot cleaner to read than surrounding all the detailed logging with “if (log.debugEnabled())” checks.
I agree that the more complex expansions that log4j evolved to support are of pretty marginal use. The oft-cited examples — context information — are IMO better added as additional fields in the log record than developer-interpolated strings.
> Yet nothing is stopping people to bash us, for work we aren't paid for, for a feature we all dislike yet needed to keep due to backward compatibility concerns.
Does log4j not utilize a versioning scheme that has the notion of "breaking changes"? Anyone following semver, for example, could simply release a major to do away with an extremely troubling, and apparently well-known, back compat issue. Sounds like folks are right to criticize for poor decision making.
> I want to be able to build knowledge of the library over a long time, to hone my craft.
This is my problem not only with programing languages but also with desktop environments and other tools. It's actually why I see people preferring LibreOffice Writer over MS Word in some fields, and why some people stick with e.g. PHP when NodeJS or Python would, in a stable world, be the objectively better technology.
[+] [-] grogers|4 years ago|reply
While deploying an ancillary change, our jvms started locking up for minutes on end. What was happening was that we were logging customer input, and the change caused it to run certain things in parallel, which ended up logging the data multiple times. Normally the extra logging didn't matter but one customer had data like "${foo} ${bar} ${baz} ...". Even when the ${foo} portion is replaced wothout modification, this triggers quadratic behavior. So we were already potentially vulnerable to the DOS but it was rare enough that we never got locked up until logging the string multiple times, which then overflowed log4js internal buffer and blocked worker threads.
You can try this yourself by just logging a string like "${}${}${}..." And in fairly short order it starts taking forever. I'm very glad the fix in 2.15 is to disable lookups by default.
I hope that in the time after I left, the security org at the big tech company I worked at and reported this to (as I thought it was - a dos vector, not the complete pwnage it actually was) forced teams to switch to nolookups. Otherwise a lot of people had a bad week forcing updates through...
[+] [-] kaeso|4 years ago|reply
> the security org at the big tech company I worked at and reported this to
I'm confused about these two statements, because I did not find any recent CVEs for log4j in the DoS category, nor related to format lookup (other than CVE-2021-44228 of course).
Perhaps I misread it, but are you basically saying that (after you reported the issue to them internally) the security team at your previous company could not successfully report a DoS vulnerability in the default configuration of a widely used (by them, at least) Apache library and make sure a CVE got assigned to track it?
If so, it would be interesting to know where the CVE/vuln-reporting chain broke, possibly to reduce the blast radius for similar future cases.
Hypothetically speaking, a CVE in March for a DoS in a problematic design/feature could have resulted in flipping the default setting earlier. Instead of chasing live RCE in the wild in December.
[+] [-] stefan_|4 years ago|reply
Of course the last part holds the solution for our backwards compatibility issue. Remove the JNDI nonsense from the default package and move it into an extension package. Whoever wants to keep it can just add that to their dependencies and continue to enjoy logging functions that sometimes also make network connections and block your program.
[+] [-] jameshart|4 years ago|reply
Nobody thought this was how log4j worked; log4j's documentation for format syntax only covers {} placeholders - the same format that slf4j has grandfathered in from log4j.
I agree this feels like a case where they got confused about their internal terminology. Log4j refers to messages with {} placeholders as 'FormattedMessages'; it refers to the log pattern syntax as 'Patterns' in code - but it seems to refer to them as 'log formats' in documentation.
Somewhere in this mess, someone hooked up the pattern capabilities into the formatting system.
[+] [-] lultimouomo|4 years ago|reply
It was actually intended behavior, and this is what really boggles the mind! Javadoc says explicitly that variable replacement is recursive, with cycle detection (which will throw! What happens to the log line in this case?) [0].
[0] https://logging.apache.org/log4j/2.x/log4j-core/apidocs/org/...
[+] [-] ehsankia|4 years ago|reply
[+] [-] thanatos519|4 years ago|reply
[+] [-] x0x0|4 years ago|reply
[+] [-] zamalek|4 years ago|reply
It's logging. While logging is extremely important, I think we could all tolerate removing a vulnerable feature. Or, just move the feature to a separate package.
I have made bad decisions, we have all made bad decisions. Own them, improve, and celebrate the opportunity to learn and improve. Keeping this around, as a default, was a bad decision. If your enterprise contracts don't want to turn a flag on, then they can always skip upgrading (they generally do regardless).
[+] [-] diroussel|4 years ago|reply
Should maintainers of all core apache libs just remove or disable features they don’t like, when not known to be insecure?
That said, log4j2 isn’t that old. Not sure why this was added in the first place. At the very least it’s a performance issue.
[+] [-] superjan|4 years ago|reply
In a perfect world, the feature would have been an option from the start, but in that same perfect world, the downstream users would be diligent and check release notes before upgrading. You might, but many of your colleagues don’t, they just upgrade, and complain when their system breaks.
[+] [-] dehrmann|4 years ago|reply
[+] [-] orangecat|4 years ago|reply
Definitely. But really, they were screwed once it had shipped. They could and should have disabled it in an update long ago, but then anyone who read the release notes or the code would know how to exploit the millions of un-updated systems.
[+] [-] dang|4 years ago|reply
Log4j RCE Found - https://news.ycombinator.com/item?id=29504755 - Dec 2021 (457 comments)
Widespread exploitation of critical remote code execution in Apache Log4j - https://news.ycombinator.com/item?id=29520415 - Dec 2021 (80 comments)
[+] [-] ordiel|4 years ago|reply
[+] [-] awestroke|4 years ago|reply
[+] [-] unbanned|4 years ago|reply
Job done.
Logging libraries are unnecessarily complicated.
[+] [-] bshipp|4 years ago|reply
You'd think, in the spirit of open source, these multi-billion dollar companies--like Apple and Google and Amazon--would recognize the danger and immediately divert the best engineers they had to help this team identify and mitigate the problems. They should have been buried in useful pull requests.
For that matter, they should have really picked them all up in private jets and flown them to neutral working space with those engineers for a one or two week hackathon/code sprint to clean up the outstanding issues and set the project on a sustainable path. To get those maintainers there they should offer a six figure consulting fee and negotiate with their current employers to secure their temporary help.
I can't believe these folks just get abandoned like this while CEOs/CTOs from rich companies wring their hands wailing about the problems and not offering solutions.
[+] [-] wpietri|4 years ago|reply
Sorry, but what's the hard part to understand? Open source maintainers end up in this position because they are nice, helpful people who like using computers to solve problems for others. People who spend years on a project and then see a bigger problem arise don't suddenly turn that off. With the bigger problem, they'll want to work harder, not just hoist a middle finger and go binge Netflix without a care in the world.
But I totally agree with you on the CTOs, etc. I don't expect random programmers who like working on logging to also be good at solving complicated sociotechological problems around paying for global infrastructure. But it boggles my mind that none of these richly rewarded, supposedly brilliant experts at organizing engineers has gotten out in front of this. If not out of community spirit or social responsibility, then out of pure self interest.
[+] [-] throw_log4jfang|4 years ago|reply
FAANG engineer here, and one who had to work extra hours to redeploy services with the log4j vulnerability fix. I'm not sure you understand the scope and constraints of this sort of problem. Log4j's maintainers have a far more difficult and challenging job than FANGs or any other consumer of a FLOSS package, who only need to consider their own personal internal constraints, and if push comes to shove can even force backwards-incompatible changes. The priority of any company, FANG or not, is to plug their own security holes ASAP. Until that's addressed the thought of diverting resources to fix someone else's security issues doesn't even register on the radar. I mean, are you willing to spend your weekend working around the clock to fix my problems? Why do you expect others like me to do that, then? Instead I'm spending a relaxing weekend with my family with the confort of knowing my service is safe. Why wouldn't I?
[+] [-] phkahler|4 years ago|reply
Agreed, while reading it I also disagreed at this point:
>> the maintainers of log4j would have loved to remove this bad feature long ago, but could not because of the backwards compatibility promises they are held to.
Nobody is holding them to anything. If they want to remove an old feature, go right ahead. If those using it think it's that important they can fork the project and maintain it themselves. Oh right, that would take effort or money.
[+] [-] MattGaiser|4 years ago|reply
A past organization of mine had a data breach (the kind that ended up making the news everywhere). A few people left (probably making it worse with all the turnover there), but I would be surprised if anything really changed in that organization.
[+] [-] ralph84|4 years ago|reply
If the RCE had been responsibly disclosed instead of via tweets and PR comments, maybe there wouldn't have had to be so much scrambling. And indeed maybe ASF could have found corporate OSPOs to help with remediation.
There are lots pixels being spilled on how the users of open source software should be paying for it (?), but I haven't seen much criticism of the vulnerability not being responsibly disclosed.
[+] [-] sorry_outta_gas|4 years ago|reply
Personally I'd like something like a security health card/metric on opensource libaries that we could tie into CI systems/pull requests or something
in the past there were so few libarries it wasn't as daunting
I'd be able reason about stuff like libpng, libttf ..etc and think about them or even support them but now some projects are massive hodgepodges of thousands upon thousnads of packages
[+] [-] thrdbndndn|4 years ago|reply
That ("...private jets..") doesn't happen because the solution isn't exactly the hard part, and the unpaid original maintainers are doing them anyway.
[+] [-] kccqzy|4 years ago|reply
Google doesn't even use log4j. What are you talking about? The spirit of open source does not dictate that the richest companies automatically shoulder the burden of maintenance of projects they do not even use. Google already has initiatives like Summer of Code that help open source projects it does not use, and I think it's perfectly fine to draw the line there.
> divert the best engineers they had
So the lessons from the mythical man-month are forgotten here. At this point I don't think adding more manpower helps.
[+] [-] phendrenad2|4 years ago|reply
Also, the reason the maintainers are rushing to fix it is: they're worried about losing "market share". Having been in open-source circles for a long time, maintainers care GREATLY about how many users they have. They just like watching their download stats go up every year. Even if it beings them no financial rewards. It's a sort of addiction.
[+] [-] dagmx|4 years ago|reply
Anything sufficiently at scale needs a set of maintainers that the commercial tech companies would then collaborate with to get the PRs going.
Otherwise if everyone's just panicking and rushing to submit PRs, they'll inundate the maintainer. There's also no guarantee that even the best engineers at these companies are intimately familiar with the project, and might introduce regressions or other vulnerabilities in the process.
Anyway I do agree companies should be working with OSS devs, but it shouldn't be rushed or knee jerk. It should be collaborative and measured.
[+] [-] iJohnDoe|4 years ago|reply
[+] [-] dehrmann|4 years ago|reply
For the general case, the problem is that a reporter might report the vulnerability to the open source project, then the project needs to keep it a secret while they make a fix. There isn't a great way to leverage these stakeholders. It's obviously different for something like Android that is open source, but clearly Google.
[+] [-] agilob|4 years ago|reply
> I want to not spend much time upgrading a dependency
> Go compatibility promise:
>So whenever a change in behavior happens in an upstream library
You are comparing a promise from language designers to no promise from the library developers. Syntax from Oak (before Java was called Java) still compiles and works in Java 17 right now:
You can still type (public abstract interface - all interfaces are abstract by default since Java 1) and it works. One of the reasons I gave up on writing desktop applications in Go was libraries were breaking compatibility with every commit. GTK+ binding was literary unusable as before gomod this would break literally, and I mean literally, every day.Please tell me that none Go library had any breaking changes in the last 5 years and I'm using it as my default ecosystem from tomorrow.
[+] [-] azth|4 years ago|reply
[+] [-] boricj|4 years ago|reply
[+] [-] WJW|4 years ago|reply
[+] [-] shaldengeki|4 years ago|reply
[+] [-] ec109685|4 years ago|reply
Changing the major version number as long it's accompanied by a well written release note on what needs to change seems fine.
[+] [-] dboreham|4 years ago|reply
[+] [-] rodmena|4 years ago|reply
[+] [-] johnsolo1701|4 years ago|reply
When a Java logging package has a vulnerability: Sober introspection about the role of maintainers, dependencies, and backward compatibility in the OSS ecosystem.
[+] [-] titzer|4 years ago|reply
This resonates with me deeply. Mastery of any subject needs a level of stability and permanence so that a skill base and a knowledge base doesn't erode away over time. Change is that erosive force, and we're so bad at knowing what changes to make in the software world.
[+] [-] smarx007|4 years ago|reply
Tailscale is wonderful software, but with all due respect I don't think David has much experience with Apache Foundation or Eclipse Foundation projects. I am a project lead of one (way smaller) Eclipse Foundation project and keep a close watch on another (bigger, but nowhere near the log4j scale) Apache Foundation project. We are free to make breaking changes (and make them we do). In both Foundations, there is a Project Management Committee that signs off on releases. They would normally raise an eyebrow if you try to push a breaking change into a patch release, but their reaction would normally be "why don't you bump the major version?".
Edit: there is indeed a social contract between the project leadership and the users. We have a social contract of supporting the oldest possible JVM for our project. Now some of our dependencies (guess what, that ASF project!) dropped Java 8 support and published a few CVEs since and we are going to move to JDK 11 as a baseline soon. Yes, some users are grumpy but people understand that unless someone forks that ASF project and starts backporting CVE fixes, we got to make the move. Bottom line: it's a much more social process than a bureaucratic one.
[+] [-] m3047|4 years ago|reply
Let me tell you a story about something that happened to me this spring. I had trouble logging in to my customer account at a critical infrastructure provider (utility provider). When I called customer support one thing led to another and they ended up reading me my password from their internal systems. (Let that sink in.)
So I spent a week trying to get ahold of someone to let them know how fucked up that is, which they avoided, until ultimately they claimed I was "threatening" them. I reached out to the "hidden hand" of the internet, did anyone have contacts with their MSP (with "National" and [edit: changed from "Infrastructure"] "Information" in the name)? But nobody did, so someone whose name would be recognized offered to tweet it.
Didn't take long to hear from MSP's Chief Cloud Architect. The story that they tell is that they'd like to turn the feature off but the customer won't let them.
Discuss.
[+] [-] dexwiz|4 years ago|reply
[+] [-] alkonaut|4 years ago|reply
[+] [-] pcl|4 years ago|reply
I agree that the more complex expansions that log4j evolved to support are of pretty marginal use. The oft-cited examples — context information — are IMO better added as additional fields in the log record than developer-interpolated strings.
[+] [-] unknown|4 years ago|reply
[deleted]
[+] [-] unknown|4 years ago|reply
[deleted]
[+] [-] sandmandf137731|4 years ago|reply
https://deepfence.io/cve-2021-44228-log4j2-exploitability-an...
https://github.com/deepfence/ThreatMapper
[+] [-] andrew_|4 years ago|reply
Does log4j not utilize a versioning scheme that has the notion of "breaking changes"? Anyone following semver, for example, could simply release a major to do away with an extremely troubling, and apparently well-known, back compat issue. Sounds like folks are right to criticize for poor decision making.
[+] [-] dotancohen|4 years ago|reply