top | item 29877745

What NPM should do to stop a new colors attack

237 points| mfrw | 4 years ago |research.swtch.com

672 comments

order
[+] franciscop|4 years ago|reply
The whole "grab the latest semver compatible version" was designed for a reason, and just pinning to a specific version would bring us to a world where very different articles need to be written. Specifically, when there's a vulnerability in any dependency (a thing that happens much more often than a rogue OSS dev) the upgrade process would be a nightmare for everyone involved.

You have to chose one of the two evils; automatic bug/vulnerability fixes, or protection against rogue OSS devs you depend on. I'd say the former is an order of magnitude more common and important, and so the npm/JS world works this way.

[+] luhn|4 years ago|reply
But the npm/JS doesn't work that way.

NPM will by default create a lockfile that pins the dependencies. `npm install` will install dependencies from the lockfile as long as package.json hasn't been updated, or update the lockfile if it has. `npm ci` will install from the lockfile and fail hard if it can't satisfy the version constraints in package.json.

imo this gives the best of both worlds: Easy, controlled updates to the latest compatible versions and stable CI/CD.

The reason color.js was an issue is because people use `npm install` in their CI and either don't commit their lockfiles or have their lockfile inadvertently out of date—It's easy to do and I'm not sure how many people actually know about `npm ci`.

[+] foobiekr|4 years ago|reply
It's... almost like pulling random code from the internet is a liability and the people doing so need to eat the costs of doing so safely and not pull code that cannot be pulled safety. Like, for example, evidence that the author actually takes security issues seriously and has a formal way to announce them.

The reality here is that just grabbing and running random shit on clients or servers has _always_ been unsafe. The people not pinning and mirroring have gigantic exposure. Pinning and mirroring people have some exposure. None of them should be doing what they are doing now.

[+] zorr|4 years ago|reply
I wonder if both are possible with a trust system where releases can be approved or signed by community members. This way we can have automatic semver updates for releases approved by people we trust.

Something like "I trust NPM so accept all releases that NPM has approved". Large companies could have their own internal approval keys etc.

[+] armchairhacker|4 years ago|reply
IMO what should be done is dependency pinning, but package repositories can send a "critical update" to essentially replace a pinned dependency. Maintainers have to explicitly request that a critical update be applied, and it's ultimately up to the repository owner to approve - generally these critical security fixes shouldn't happen often.

This would handle both Log4Shell and colors/left-pad issues. Additionally, a repository owner could force a mandatory package update without the author's consent, in case someone hides malicious code in their package and it becomes popular.

Of course the repository owner could go rogue, but I trust NPM/Maven/Cargo/OPAM maintainers much more than a random package developer.

[+] wbl|4 years ago|reply
Have you used Go? Updating a module is easy and it uses exactly the solution described above.
[+] jka|4 years ago|reply
Hrm. Generally what you're saying sounds reasonable, although I wonder if there's a chance of a false dichotomy there.

What about human errors introduced by non-rogue developers? (a reasonably common occurrence, probably?)

[+] kristjansson|4 years ago|reply
Sure, but there are points in the design space between “everyone specifies exact version requirements” and “everyone specifies lower bounds and pulls the absolute latest”.
[+] oauea|4 years ago|reply
Yes, and that reason is no good. There's a reason maven3 moved away from it.
[+] jacquesc|4 years ago|reply
It's really tricky when you end up using libraries 4 levels deep, and never consciously chose something.

Looking at one of our production projects, we use colors via: "css-loader" -> "cssnano" -> "svgo" -> "colors"

I wish I could say I spent the hours to go line by line through every dependency of our app. But that wouldnt leave much time for anything else.

[+] mac-chaffee|4 years ago|reply
There are tons of tiny libraries that have gotten themselves into the dependency chain of popular libraries, and any attempt to remove them is treated as a turf war.

I've tried to remove single-line dependencies only to have the PR rejected by the creator of the tiny library who happens to contribute to the parent library.

IMO the Javascript ecosystem really needs a decent standard library to remove the inordinate amount of power granted to these these tiny library squatters who wrote 5 lines of code and a package.json file 10 years ago.

[+] rsc|4 years ago|reply
Exactly. In this case, the package manager should use the version of colors that svgo asked for, not the one that appeared on the internet 5 minutes ago.
[+] numbsafari|4 years ago|reply
If you are unconsciously shipping something, you shouldn't be shipping anything.

Our industry has gotten along for far too long with zero liability or accountability.

[+] jrm4|4 years ago|reply
Imagine a car manufacturer like "I wish I could say I spent the hours looking at every valve and every screw..."
[+] the__alchemist|4 years ago|reply
Nailed it. Shallow dependency trees are much easier to maintain and secure.
[+] bhedgeoser|4 years ago|reply
It's the responsibility of "svgo" to make sure it's direct dependencies are alright.
[+] foobiekr|4 years ago|reply
So.. basically you are OK shipping unvetted, attacker-controlled code to your users?
[+] woutr_be|4 years ago|reply
Reminds me of my time at a global bank. After many security incidents, they finally decided to step up their game, and implemented multiple security scans.

For any project using NPM, it was an absolute nightmare. Dependencies 3 or 4 levels deep would get flagged, and there wouldn't be a way to resolve them. The security teams didn't care, and multiple projects were left stranded. Since resolving those issues would mean having to contribute to open source, which we weren't allowed to do.

On the other hand, because we had specific security teams doing the scanning and assessing the results, you weren't allowed to question them, because they could just block your entire project. So developers were extremely unhappy with them, and what ended up happening is that developers just did what the security teams asked, without questioning. Which to me, let to more security concerns, because we were just pulling in libraries without knowing what had changed.

Sadly, the few devs who stopped relying on third party dependencies all left, because most of their work took longer (since they were implementing stuff that third parties usually do). Business teams took notice and questioned why everything was taking longer.

[+] commandlinefan|4 years ago|reply
> that wouldnt leave much time for anything else

By the time you got to the end, it would be time to go back to beginning and start over again...

[+] zitterbewegung|4 years ago|reply
Unfortunately the way that this can be prevented would be to audit the packages before including them or having your own package management that you control.
[+] howdydoo|4 years ago|reply
Summary:

The author argues that version bounds should be treated as a maximum rather than a minimum, like Go does. e.g., if the latest version of colors is 1.0.3, and you have dependencies that request 1.0.1 and 1.0.2, you would end up with 1.0.2. The end result being, the exact resolved version will have been tested with at least one of your dependencies.

I must admit I like the idea.

[+] freeqaz|4 years ago|reply
I've used NPM shrinkwrap before back in versions 1-3 of NPM. It's a little confusing that the author of this post calls it "new", so I investigated.

Since NPM version ~2 (current is 8), you're allowed to publish a npm-shrinkwrap.json file[0] in your package. That's in contrast to a package-lock.json, which may not be included in a published package.

Back when I used shrinkwrap, it was pre-lockfiles and it was trying to accomplish something similar to what lockfiles accomplish today. (Lockfiles were added in v5 [1])

I dug up this old StackOverflow post to verify that the behavior of shrinkwrap hasn't changed[2] since many years ago.

That doesn't leave me very hopeful. If package maintainers aren't doing this already, and they've been able to for 6+ years, then it's unlikely they will in the near future.

(I'm currently investigating how to deal with bad deps with some Open Source tooling I'm building. Feel free to ping me if you have any thoughts to share)

0: https://docs.npmjs.com/cli/v8/configuring-npm/npm-shrinkwrap...

1: https://nodejs.dev/learn/the-package-lock-json-file

2: https://stackoverflow.com/questions/44258235/what-is-the-dif...

[+] dang|4 years ago|reply
Recent and related:

Dev corrupts NPM libs 'colors' and 'faker', breaking thousands of apps - https://news.ycombinator.com/item?id=29863672 - Jan 2022 (955 comments)

Important NPM package colors from Marak causing console problems at the moment - https://news.ycombinator.com/item?id=29861560 - Jan 2022 (1 comment)

Creator of faker.js pushed an update of colors.js which has an infinite loop - https://news.ycombinator.com/item?id=29855397 - Jan 2022 (1 comment)

Marak adds infinite loop test to popular colors.js - https://news.ycombinator.com/item?id=29851065 - Jan 2022 (7 comments)

Marak's GitHub account suspended after he erased his faker project - https://news.ycombinator.com/item?id=29837473 - Jan 2022 (53 comments)

Faker.js Erased by Author - https://news.ycombinator.com/item?id=29822551 - Jan 2022 (2 comments)

Popular JavaScript package “Faker” replaced with message about Aaron Schwartz - https://news.ycombinator.com/item?id=29816532 - Jan 2022 (3 comments)

Faker.js Has Been Deleted - https://news.ycombinator.com/item?id=29806328 - Jan 2022 (9 comments)

[+] GeekyBear|4 years ago|reply
In this case, Microsoft seized ownership of an open source project and modified the code without the permission of the person who still controls those copyright rights that they haven't specifically waived.

Legally, shouldn't Microsoft limit themselves to banning the developer from npm and forking their repos under a new name?

[+] mfer|4 years ago|reply
Basically, applications should use a lock file for dependencies based on known tested good versions of dependencies.

How is it that people aren't doing that today? For the sake of security and stability, lock files should be used.

[+] danenania|4 years ago|reply
They are used. A lot of the comments in this thread that are about NPM specifically are strawmen, since it has been standard to use lock files for years.

You could still have an issue if you need to update a dependency, for security or other reasons, since it could bring along a bunch of updated sub-dependencies of its own (and sub-sub-dependencies, and so on). But that problem is not unique to NPM and exists in any language or platform that includes package management.

[+] ghostly_s|4 years ago|reply
> Other package managers should take note too. Marak has done all of us a huge favor by highlighting the problems most package managers create with their policy of automatic adoption of new dependencies without the opportunity for gradual rollout or any kind of testing whatsoever.

(Emphasis added) - is this actually a widespread practice? That's certainly not how apt packages are handled...my impression was this is a problem unique to the js ecosystem.

[+] dane-pgp|4 years ago|reply
I don't have an answer to your question, but I'm glad you've highlighted this part of the article. The idea that an attacker "has done all of us a huge favor" by attacking the free software community is so toxic that it needs to be called out, even if it was meant somewhat in jest.

If we don't reject this logic, then we'll get more attackers claiming "just a prank, bro!" and "social experiment!", like the University of Minnesota researchers carrying out human experimentation on kernel developers without their consent:

https://www.theverge.com/2021/4/30/22410164/linux-kernel-uni...

[+] ms4720|4 years ago|reply
Pin your versions, this was so effective because no one pins their version for what they are using.

Anyone saying it was the dick's fault and not yours needs to start owing their project.

I admit that the last time I worked with node it was a cluster fuck of nested dependencies that only works through the prayers of angles above. But standards are standards, own your project.

[+] paulhodge|4 years ago|reply
The many suggestions about what NPM should do differently are universally all really unsatisfying, rearrange-the-deck-chairs-on-the-Titanic level solutions.

At the end of the day, if you're running code that you didn't write or verify yourself, you're taking on risk.

IMO the next hugely successful package manager (in whatever programming language) will be the one with a really good capabilities model, so you can run untrusted code with much less worry.

[+] hn_throwaway_99|4 years ago|reply
One feature I've wanted for some time now is the ability to say "give me all the latest versions that are at least N days old" when you run npm install without a lockfile or npm update.

The idea being "I want the latest version, but not the absolute bleeding edge, I want something that has at least baked for a bit".

[+] lmeyerov|4 years ago|reply
It seems npm and GitHub do take action in cases like leftpad and colors.. and people don't like them (Microsoft) doing so unilaterally. Maybe part of the 0-day fix is go multiparty: allow weighted votes by dependents to take over the namespace to force a patch/reversion. You own your code etc, but the OSS distro service has its own community-owned rules, and you are free to run a competing package manager without them. By using community managed package managers, you signal your intent not to break your users & contributors, and give an explicit remediation mechanism for handling such trust chain violations. Instead of a hundred-message GH thread, we get a voted minor version bump same-day.

Though agreed with the sentiment of 'prefer stable' during install would be A+. 1am package scans was a cruddy way to start my vacation.

[+] acdha|4 years ago|reply
> It seems npm and GitHub do take action in cases like leftpad and colors.. and people don't like them (Microsoft) doing so unilaterally.

A few people don't like this but I think that most people are quite happy not to have their projects compromised. I would not assume that the loudest voices are broadly representative, especially with Microsoft being a lighting rod for complaints.

> Maybe part of the 0-day fix is go multiparty: allow weighted votes by dependents to take over the namespace to force a patch/reversion.

I think there's a lot of promise in this approach, especially if it's all public and deliberative — e.g. freeze the attacker's account, remove the compromised package, etc. but then have a public voting period for a transfer. The other question I'd have is whether it'd make sense to have a restricted set of maintaining organizations — e.g. I'd be a lot more comfortable if, say, the new project got transferred to an organization like the Apache foundation than some random developer who could plausibly be preparing to quietly ship a crypto-miner or something.

[+] omegalulw|4 years ago|reply
> Maybe part of the 0-day fix is go multiparty: allow weighted votes by dependents to take over the namespace to force a patch/reversion. You own your code etc, but the OSS distro service has its own community-owned rules, and you are free to run a competing package manager without them.

Not a good idea. You are pretty much asking to be review bomb. You would need very good moderation to be able to trust votes. That means npm spending a bunch of resources on hiring them, not sure if they are up for that (no, free mods aren't the solution).

[+] gigel82|4 years ago|reply
I take issue with the title of this article. This was not an attack. The project's author committed a change to his own open source project, which he had every right to do.

Was that an asshole move, and did he lose all credibility in the OSS community? Yes, absolutely. But it was not an attack.

In fact, I also take issue with GitHub / Microsoft taking over the package and am very worried about the precedent that sets, regardless if their motives this time appear to be entirely selfless.

[+] spankalee|4 years ago|reply
Just because the package owner had implicit permission to run code on other people's machines, doesn't mean that this wasn't an attack.

A postal carrier may have a key to a building to deliver mail, but if they use it to turn on all the faucets and leave the doors open, it's an attack.

[+] dimgl|4 years ago|reply
I don't agree with the general sentiment on this. This seems like a blatant attack to me. He went out of his way to deliberately sabotage all of the projects that depended on his. I have a few open source libraries (admittedly, not very popular) and I'd never do this. Even if I am giving the project out for free with an MIT license and are not liable for any damages.
[+] sam0x17|4 years ago|reply
> The project's author committed a change to his own open source project, which he had every right to do.

Indeed, and it goes beyond this. I think the real implication is that software published on NPM and similar networks isn't truly free and open source, in the sense that the platform owner can and will step in and make editorial changes without the knowledge or consent of the software maintainers.

[+] sigzero|4 years ago|reply
Of course it was an "attack". Is doesn't matter that he owned his own packages. He knew exactly what he was doing with his changes and the effect it would have on projects that relied on his.
[+] shadowgovt|4 years ago|reply
It's worth asking, I think, how much energy should be spent against guarding against an attack like this in the future. I think every organization's risk model is different.

In this case, the Net interpreted Marak's actions as damage and seems to have effectively routed around them. There was disruption, but we already have systems in place for flagging broken or malicious versions (`npm audit` being one example).

For people willing to have the system work 99% of the time with the occasional Marak-screw, we already have a working solution in what currently exists. For people who cannot tolerate that level of risk, you have to firewall your package sources and only allow use of vetted code dependencies in your software.

Open source relies, in the end, on trust. Marak broke trust and that's all. The question isn't "How do we prevent people breaking trust" (that's impossible; humans have free will) but instead "How do we mitigate damage / protect that which cannot be risked if someone chooses to break trust?"

[+] vbezhenar|4 years ago|reply
Here's my 2 cents how build system should work. I'm not sure that any build system works like that... But anyway.

All dependencies sources must be included with project sources. When you check out repository, you should have everything necessary for building the project. Sources should be in their original form, so you could fix anything quickly.

Libraries builds must be reproducible. Output artifacts hashes must be provided along with sources. It means that one should be able to download pre-built artifacts from the Internet. But if they're not available, that should not be an issue.

Ideally whole build environment should be part of project, something like docker container. So your compiler and necessary build tools are fixed as well.

Of course there should be some tooling to check for new dependencies versions, print changelogs if available, etc. But it must not be automatic.

[+] fold_left|4 years ago|reply
Checking in your dependencies with https://github.com/JamieMason/shrinkpack can help insulate you from these problems until you're ready to face them. I created this before left-pad and thankfully meant that we were unaffected.

A lot of developers, understandably, baulk at checking in dependencies, but there is a concrete benefit in being able to continue uninterrupted during outages.

[+] dusted|4 years ago|reply
Yeah no.. The author chose to change his own project, that's in no way an attack. People not only pulling random code from the Internet, but also integrate it automatically without review and without as much as locking the version is the problem here if there is one.

The author is under absolutely no obligation to maintain his project or to keep in some state desirable by whoever is currently using it.

[+] smasher164|4 years ago|reply
> Essentially every open source software license points out that the code is made available with no warranty at all. Modern package managers need to be designed to expect and mitigate this risk.

It's almost like the package manager's job has become to protect users from their dependencies.

[+] diogenesjunior|4 years ago|reply
when i have any code that requires dependencies, i always use an exact version. i am mainly a python developer, so my requirements.txt would look like this:

    flask==1.1.1
    request=2.3.2
    tensorflow==3.4.2
i tested/built my code with these versions, so i know that these versions work. when i install my dependencies, i use these versions, not the latest version. is there a way to do this with node.js? and wouldn't using this method solve any problems? pardon me if i'm missing something
[+] awinter-py|4 years ago|reply
uhhh yes staying pinned to an old version forever solves some problems, but not other problems? article doesn't mention 'npm audit' and how there are cases where you want to encourage an upgrade

real long-term solve here is a code review community for widely-used public packages I suspect?

am not a huge blockchain fan but this is one thing that blockchain could conceivably do well, because reviews are public, need to be authenticated, exist as compact metadata that can fit on chain, and benefit from public reputation dynamics

[+] Hizonner|4 years ago|reply
The real long-term solution is for projects not to have hundreds of weird dependencies.

If dependencies are pinned until developers update them, you have massive security holes, because developers can't be trusted.

If you take the latest dependencies on every update, you have massive functional exposures (and security holes), because different developers can't be trusted.

If you have "community review", you create a new class of thankless work that nobody will want to do... except power-tripping control freaks who want to gatekeep over whatever their personal obsessions may be.

If users have to pick the versions, nothing will ever work, because users can't be trusted (and wouldn't want to do it anyway).

[+] howdydoo|4 years ago|reply
A blockchain isn't needed for that. Authentication needs "crypto"graphy, but not "crypto"currency.

This wouldn't be a complete thread without someone mentioning Rust, so I'll do it. cargo-crev is a nice web-of-trust type code review system for Rust crates. https://github.com/crev-dev/cargo-crev

[+] philosopher1234|4 years ago|reply
Where did you get forever? The idea in the article is to upgrade dependencies only when the maintainers are ready to/explicitly say to.