top | item 30059404

LogJ4 Security Inquiry – Response Required

323 points| elsombrero | 4 years ago |daniel.haxx.se

118 comments

order

1970-01-01|4 years ago

>>I answered the email very briefly and said I will be happy to answer with details as soon as we have a support contract signed.

This made my day. If a wealthy individual takes your tools and then calls for help while fixing-up their shed with said tools, do not move a muscle until you agree on the fee.

t0mas88|4 years ago

I expect they'll gladly sign a support contract with Daniel.

As a commercial SaaS vendor we received these same emails from all of the major banks / insurance companies. It's interesting to see that we got some on the Monday after the issue was discovered and some a few weeks later, with some showing a clear understanding of the risk in the context of our product and some looking like a standard copy/paste. Gives you a rare behind-the-scenes view of the information security practices of these companies.

zokier|4 years ago

I think it is pretty easy to see how this sort of thing happens:

1. Someone decides that we need inventory of all the libraries used (iirc requirement for some certifications and generally not a bad practice)

2. A system (/excel sheet) is enrolled where you have fields like $our_product, $library_used, $vendor_email

3. A dev, not quite understanding the point, dutifully fills in the data for the project they are working on

4. No-one reviews the data

5. Crisis strikes, so mass-send email to all vendors how they are handling it

Problem here is around point 4.; for the process to work, someone should have reviewed the data to check that the used libraries are from vendors with some sort of support arrangement.

I think the reply they provided is pretty promising, it makes it sound like they wanted to be a customer but are not only due an oversight.

djbusby|4 years ago

I've been responsible for parts 1,2,3,4 and hand off the work to owners/managers/investors.

There is a large time gap between 4 and 5 - and it seems everyone forgets who they hired for that supply chain "analysis" many moons ago.

verytrivial|4 years ago

For everyone boggling at the tone of the email, stop for a moment and have a guess at how many different sources of software they think the average large corp has on their books let alone on their infra. It can literally be hundreds or thousands of different sources. And each of those will have their own topology.

This is clearly a scatter-gun survey because they're realised they really have no idea of their exposure. (And before you re-boggle at that, there's a whole business ecosystem in just being able to answer that question let alone do anything about security issues.)

blihp|4 years ago

Also, for any FOSS author who gets one or more of these inquiries don't laugh it off or write blog posts mocking the sender. Take it as the business opportunity it is and send a professional response indicating your willingness to help them navigate through this, at least as it relates to your bit of code, for customers with paid support plans. You want money, they have money and you can trivially provide them something at least some of them are willing to pay up for with a potential opportunity for a non-trivial longer term relationship.

This is the best kind of sales call: they are coming to you.

devadvance|4 years ago

Generally this is an accurate take. I'd add two things:

> ...because they're realised they really have no idea of their exposure.

This is partially because it is often non-engineers being asked to figure this out. The "information security analysts" at F500s are asked to do a lot of unfair work, such as analyze risks related to decades-old software they didn't build.

> ...there's a whole business ecosystem in just being able to answer that question let alone do anything about security issues.

The first part (answering "what dependencies does my software have") isn't inherently bad. I'd emphasize the underinvestment in the second part more.

hotpotamus|4 years ago

Software eats the world just like a black hole.

softwarebeware|4 years ago

"...The level of ignorance and incompetence shown in this single email is mind-boggling...no code I’ve ever been involved with or have my copyright use log4j and any rookie or better engineer could easily verify that..."

Yeah, well, I've been quite shocked how rookie some F500 devs can be and how dysfunctional large corporations can also be. Probably what happened here is someone wrote a script that compiled the dependencies of all projects they have and they sent this same email to all of them (!) regardless of any actual or potential use of log4j.

dagw|4 years ago

When I worked at a large, but not F500, company I had to once every 6-12 month or so fill in a spreadsheet with all third-party dependencies, with their licenses and some other info, the project I was working on used. I then emailed this to a mystery person and never heard anything back ever. I can easily see someone pulling out these spreadsheets and just emailing away without any developer, rookie or otherwise, being aware of what was happening.

boring_twenties|4 years ago

Let's hope they apply a similar amount of due diligence when the author responds with an offer to look into it for $800/hr with a 20 hour minimum.

gkoberger|4 years ago

I don't want to defend this company, but my company (a dev tool used by many other companies) receives a handful of these a day. It's almost the exact same email, and they're just mass-sending them. It's not personal, and it's pretty standard.

The tone feels off if you assume a human wrote it. But that's only because it's a form letter their legal department wrote for them to send off. They probably collected "dependencies" from the entire company (and someone wrote "curl"), and sent a mass email.

If you just reply with a simple "We're unaffected!" (or ignore them), you'll never hear from them again.

throw8932894|4 years ago

Better to reply "yes, we are affected. Your support contract has expired, please renew at XYZ".

0x500x79|4 years ago

Many organizations document their 3rd party vendors and libraries and it doesn't surprise me that an automated email reached Daniel. Most likely someone mis-documented using one of Daniel's projects in a spreadsheet.

I am personally a bit surprised about the responses here. It is completely reasonable for this email to reach Daniel and is most likely an artifact of bad documentation by engineers in the company. At the scale this company is running the person/team sending out these emails do not have time to dig in and understand each dependency they are sending emails on.

The response is as simple as "What library/product does this email pertain to?", "Please see the licenses for the libraries or products in question.", and what Daniel responded with as well: "I would be willing the dig in further for specific questions with a support contract.".

matheusmoreira|4 years ago

> At the scale this company is running the person/team sending out these emails do not have time to dig in and understand each dependency they are sending emails on.

That alone is extremely disrespectful, it means they couldn't care less about the time of open source software maintainers. To say nothing of their "request" for review.

rmoriz|4 years ago

As far as I learned, a couple of big companies are sending this kind of mail to every provider, partner or copyright owner of code that they could find.

I assume some developer/supplier used curl and provided a list of third party code and licenses they use.

In the aftermath of the log4j incident, companies now target everyone about this issue partly to learn about potential exposure that they are not aware yet, eg exploited infrastructure of depending services like newsletter or analytics services.

Yes, it's annoying and pointless to spam this mails to open source projects. But at least someone is now behind auditing the supply chain.

cryptonector|4 years ago

It's a really dumb approach to vulnerability management for CYA. Spray and pray that the regulators are assuaged. It might even work as far as that goes.

But obviously, it's not a sound approach to actual vulnerability management.

ericcholis|4 years ago

I've read speculation that this is to cover their own asses with various regulations. Not sure if there's any weight behind this.

trevormcneal|4 years ago

The first email looks like someone who had zero idea of what they were doing, just did some dependency scanning and got your name/email there, probably these emails were sent to everything that they could find.

Quite well handled, not arrogant, not bending over and doing whatever they say, but being honest.

If curl is impacted or not, may not really matters for them, usually these companies go after compliance and someone who they can blame when things go wrong.

BiteCode_dev|4 years ago

It's actually fantastic to receive such email. You can answer:

"We are happy to provide you with support regarding this issue for $5000/day"

Then if they accept, proceed to do nothing for 10 days, then reply you find none of your code is impacted and they are safe then bill them $50k.

csdvrx|4 years ago

> proceed to do nothing for 10 days

That would be fraud. No, start grep on the source code and a few things like that, then provide the results: "a detailed audit found no reference to log4js, so another audit was started which found no reference to any java code in the C source; it was repeated 5 times to confirm these promising results. Another audit followed the Boltzman brain hypothesis to check if the affected log4js binary code could not be spontaneously generated during compilation, by following a Monte Carlo simulation to check for various length of binary data that would match the log4j binary code. (...)

Finally, to avoid this extremely remote risk, the code changed to switch to reproducible builts, which can guarantee this will not happen"

zitterbewegung|4 years ago

There was a HN post about selling to the Enterprise market. Doing it the way that was described there would be. Also, to not perform a scam as other posts here would be.

1. Insist that you need to talk to upper management until you get to the CEO.

2. Once there you need to sell them on a Fixed fee contract for five engineers so let’s say $1MM or more

3. Actually create a few scripts that run the log4j scanner from Google.

4. Have an extended support contract by doing this yearly at $1MM.

kube-system|4 years ago

It's fraud to bill someone T&M for time that wasn't actually spent. You're better off quoting it fixed-fee. :)

rdtsc|4 years ago

> Then if they accept, proceed to do nothing for 10 days, then reply you find none of your code is impacted and they are safe then bill them $50k.

Hopefully you don't do that or encourage others to. Just because F500 companies are big, stupid, slow and greedy, doesn't exactly make stealing right.

Grollicus|4 years ago

You'd probably need all of the 10 days to fight through all of their supplier management forms, answer pointless questions about security certifications, people involved and if you do business with iran.

perihelions|4 years ago

>"Thank you for your reply. Are you saying that we are not a customer of your organization?"

Isn't this the sort of question you'd ask your own side, first?

jaywalk|4 years ago

In a Fortune 500 company, I'd imagine it could be quite difficult to definitively prove that they are not a customer of any one organization.

The company I work for is not Fortune 500, but we have several Fortune 500 customers. The amount of inane bullshit we have to deal with as a result is mind-boggling.

Bedon292|4 years ago

I would bet this was sent out to a list that was put together that contained all of their "partners" which was in turn compiled from various other spreadsheets. Including one that had a 'support contact' column, or something like that and they assumed any that had that value was a partner. Over the course of the 6 years that the sheet has been cut and paste in various formats, they completely lost where any of it came from in the first place.

deltree7|4 years ago

OK, a large corporation legal team doesn't understand the nuance of ownership of open-source software.

Do we mock every single open source guy who displays the same amount of cluelessness about the inner workings of a business because I see plenty of that displayed here and everywhere else.

Mvandenbergh|4 years ago

Understanding the nuances of ownership and who is responsible for what is quite an important skill for corporate lawyers.

josephcsible|4 years ago

"Don't expect people to do work for you if you're not paying them" is obvious to anyone with a brain. I fail to see how nuance comes into this.

gautamdivgi|4 years ago

Ummm... I think the curl license is displayed pretty publicly. So, yes - this email deserves to be mocked roundly.

loriverkutya|4 years ago

Yeah, they get their pay check to understand the nuances :)

kube-system|4 years ago

I am sure their lawyers know exactly how software licenses work.

I also bet that the list of dependencies they used for this mass email was probably not generated by a lawyer.

0xbadcafebee|4 years ago

Welcome to Corporate Life. Somebody at the top says "Make sure we find out from all vendors what their log4j impact is", and that trickles down until some poor sap in InfoSec is told to do it. And of course "all vendors" includes "open source vendors", aka some dude named Carl in Uzbekistan who wrote a Node.js module. Since InfoSec sap shouldn't even have been tasked with this ridiculous ask, and he's got 10,000 of them to send, he sends a form letter.

stavros|4 years ago

I haven't seen anyone here comment on this, but I loved "Hi David" in response to Daniel's reply.

sandworm101|4 years ago

Not every open source project is run by the little guy. I want to see a a security vulnerability in something like AES. Then the complaint emails demanding answers in 24 hours would be going to nsa.gov addresses.

Anyone leading a shareholder action would love to see these emails. They are basic admissions that the company doesn't know how or from where it gets essential software.

bartread|4 years ago

This is golden and characterises a surprising amount of my experience of communications with large corporations:

>> Thank you for your reply. Are you saying that we are not a customer of your organization?

It's just so beautifully orthogonal. Oh, and they got his name wrong in the salutation.

cryptonector|4 years ago

> Are you saying that we are not a customer of your organization?

LOL.

gred|4 years ago

The cherry on top (if I'm reading the follow-up email correctly), is that he gets his name wrong (David vs Daniel).

tomjen3|4 years ago

I work for a much small company, but I can't say that we are not a customer of X organization, because somebody else may be buying from them.

notyourwork|4 years ago

I find it a bit sad that a tech literate group is bashing a non-literate group fo people. The entire reason your salary is much larger than many other career paths is because of your ability to deal with technology. The premise that when the less educated and informed try to question something they don't understand only to be left with pandering and jabs is disingenuous. The questions although perhaps better phrased by someone with a more tech focused background are fine questions for a business to ask. Stop being douchebags and grow up.

krisoft|4 years ago

> I find it a bit sad that a tech literate group is bashing a non-literate group fo people.

Creating a software bill of materials is a technical task. Managing software security risk is a technical task. These need to be performed by a technically literate person.

A Fortune-500 company has the resources to pay for such technical competence. They are not a mom-and-pop shop.

No Fortune-500 CEO would get their teeth done by a fly-by-night "dentist", nor would they hire "builders" who can't nail two planks together. They would pay for the expertise. If they don't know how to find the expert they would pay for the expertise of finding the expert first and then they would pay for the expertise.

But this is not what they did. They found someone who is both lacking the necessary technical common sense and is terribly arrogant. That is worthy of ridicule. And I'm not ridiculing the individual employee but the whole company.

> The premise that when the less educated and informed try to question something they don't understand only to be left with pandering and jabs is disingenuous.

That idea flies when a student is lost in the woods. When an economic juggernaut combines technical illiteracy with a lack of tack they can get the sharp ends of our tongues.

> The entire reason your salary is much larger than many other career paths is because of your ability to deal with technology.

Won't be for long if we silently support huge companies to employ muppets. Which is why asking for a support contract is the right answer here.

kahrl|4 years ago

I find it sad that the security department of a Fortune 500 company is sending out emails demanding OSS maintainers respond within 24 hours or else.

You can feel sorry for the poor sap that was forced to embarrass himself, but it doesn't change the fact that everyone here feels like that company can get bent.

commandlinefan|4 years ago

I had the same thought - the e-mail really wasn't that unreasonable, coming from the perspective of somebody who didn't realize there was no support contract in place (and maybe didn't even understand how that could happen). Haxx's response seems similarly reasonable - we don't have a support contract, let's get one in place and then move forward from there. This really seems to be an object lesson that if you're depending on somebody for business-critical infrastructure, make sure they have a reason to support your business.

Bedon292|4 years ago

It reads like this was a form letter that was to be sent out to all their actual paid partners, and after the massive game of telephone that is corporate hierarchy, it somehow became all dependencies they had contacts for. And somewhere someone filled out a form, probably years ago, with his email on it as the maintainer of a dependency they use (because leaving it blank isn't allowed). And he got caught up in that mass email, totally dumb, but also could easily see how it can happen.

slim|4 years ago

I think daniel's reaction is appropriate and well thought. One can suppose thousands of these emails asking for free work have been sent. There is close to zero chance his demand for a support contract would get past the first filter. Whereas making a blog post about it makes for a good story and also has more chances to get the attention of the right people at this company. Even if it's slightly aggressive.

devit|4 years ago

The document uses a monospace font, and the redacted name can be seen to be 10 characters long.

Based on the 2019 Fortune 500 list, that gives these possible candidates: Activision, Alaska Air, Albertsons, Altice USA, Amazon.com, Ameriprise, AutoNation, BB&T Corp., Bed Bath &, Blackstone, Booz Allen, BorgWarner, Burlington, CBRE Group, Chesapeake, CMS Energy, CVS Health, Dean Foods, DTE Energy, Enterprise, Eversource, Expeditors, Fannie Mae, First Data, Ford Motor, Home Depot, Huntington, JM Smucker, Jones Lang, Laboratory, Mastercard, McDonald's, Murphy USA, Nationwide, News Corp., NGL Energy, NRG Energy, Occidental, PBF Energy, Prudential, PulteGroup, S&P Global, State Farm, Unum Group, US Bancorp, WEC Energy, Windstream, World Fuel, WR Berkley, Yum Brands

aerovistae|4 years ago

"In the picture version of the email I padded the name fields to better anonymize the sender, and in the text below I replaced them with NNNN."

Miner49er|4 years ago

From the article: "The email comes from a fortune-500 multi-billion dollar company that apparently might be using a product that contains my code, or maybe they have customers who do. Who knows?"

The "or maybe they have customers who do" makes me think that this company must provide services to other companies, so probably not a Mcdonald's or Albertson's or something like that.

knob|4 years ago

This analysis is beautiful. Thank you for doing the math! :-)

specialist|4 years ago

Tangent:

Wasn't Java's SecurityManager stuff supposed to prevent these kinds of exploits?

I haven't used log4j for ages, so I didn't know offhand. Somewhat curious, I gleened that none of the enterprisey stacks use SecurityManager. I guess I kinda understand; SecurityManager was fashioned and pitched for an ecosystem of applets, agents, and sandboxes.

Further, I then gleened there's a JSR to outright remove SecurityManager. With no apparent replacement, just some vague advice to roll your own capabilities based system.

So, however we got here, what's then plan? Run JVMs on top of something like OpenBSD's pledge?

dudeinjapan|4 years ago

More accurately "a clueless IT lackey at a Fortune 500 company" sent the mail. I doubt the chairman was pounding the board table and barking "We demand answers from Haxx!"

daenney|4 years ago

It didn’t come from IT/engineering. This is legal/compliance.

kzrdude|4 years ago

I wonder what their reply is about. They probably have no idea what/who they are really talking to, and it's probably not some kind of legal trap.

travoltaj|4 years ago

It's a reply to David/Daniels email to the F500 org. The dev didn't post a screenshot of their reply, but they mentioned this - "I answered the email very briefly and said I will be happy to answer with details as soon as we have a support contract signed."

aero-glide2|4 years ago

Reads like an automated email they sent to many people.

aahortwwy|4 years ago

Is there a product out there that makes it easy for open source maintainers to offer enterprise support services?

I think support is probably the best way of making money from open source, but a lot of maintainers are unlikely to have everything set up to do so (business entities, contracts, ways to receive payment, probably a dozen other things that you'd never think of, etc.).

Like Stripe Atlas for open source consulting?

bastardoperator|4 years ago

Versus asking for a support contract because I don't really want to support anyone like this long term, I would have sent an invoice... If it gets paid, I answer the questions, if it doesn't everyone knows where everyone stands. I also think it's easier to get an invoice paid versus trying to negotiate a support contract.

daenney|4 years ago

He’s not trying to negotiate a support contract. It’s a polite “fuck off”.

rdiddly|4 years ago

A tangential point, and granted it's been around and rightly ridiculed since probably the 90s, but how 'bout that classic signature about CONFIDENTIALity? It's like the icing on the don't-know-how-computers-work cake!

phendrenad2|4 years ago

You can learn a lot from this. This is how efficient companies operate. No one who knows the difference between C and Java was involved in the sending of this letter. If they were, that would be a waste of resources.

alkonaut|4 years ago

If you want a company to change their behavior, give them a reason to do so. Daniel has quite a platform being a well known maintainer, but instead of using that platform to shame the company in question, he politely emails back to the person sitting with an outdated excel sheet of 500 suppliers. That person didn't decide that the "email everyone on this list demanding info" strategy was a good idea.

To actually make a difference when you have a platform, use it. Tweet-shame them so that the fallout actually reaches the manager in question. This is just complaining about a behahavior while at the same time more or less doing everything possible to encourage that behavior.

zeroesandones|4 years ago

seems like it's automated, they must be aware it's oss

jandrese|4 years ago

I have a feeling that some security automaton at a major corporation is about to have their mind blown when they discover the world of Open Source Software. They had absolutely no idea that non-commercial software was even a thing.

josephcsible|4 years ago

Why black out the company name? Confidentiality notices at the bottom of emails aren't legally binding, especially when it's an unsolicited email from a company you have no relationship with.

jffry|4 years ago

This was addressed directly within the linked blog post:

> In my tweet and here in my blog post I redact the name of the company. I most probably have the right to tell you who they are, but I still prefer to not. (Especially if I manage to land a profitable business contract with them.)

Nicksil|4 years ago

>Why black out the company name? Confidentiality notices at the bottom of emails aren't legally binding, especially when it's an unsolicited email from a company you have no relationship with.

It's explained in the article.