There is an open NPM “audit assertions”[1] RFC which would allow the compromised package to be nested one dependency deeper, and would allow the hijacked ua-parser-JS package to assert that it’s unaffected.
Should this proposal go through, it’s quite possible this would have gone much further. It would have allowed the attacker to wait for detection, and to silence any propagation of that to users of the otherwise legitimate package.
I had to bow out of the conversation because I didn’t feel like I could engage it in a healthy way—I’m way too personally affected by my own experience dealing with dismissal of serious security issues—but I think folks here might be interested in it given this compromise.
A malicious audit assertion is grounds to consider the package malicious. Thus you just mark the parent package as malicious.
Edit: I think your misunderstanding is you're considering "malicious config" and "malicious code" significantly different issues, and arguing malicious config won't be noticed as malicious because it's only malicious in the context of what it enables. I believe this is incorrect. Innocuous seeming usage of a legitimate feature that actually has malicious effect is not a new issue.
Looking at the issue in more detail you were told essentially exactly this.
> A far, far easier version that already exists is: put malicious code directly into an otherwise innocuous package (or as a dependency). No vulnerability warning would exist, and thus there'd be no need for assertions. In other words, your "exploit" is just a more complex way of doing something that has always been possible.
> Once you've trusted a maintainer, you have already completely surrendered any protections to their judgement. If they make a mistake with an assertion, they could also make a mistake with code, or by depending on a package with an unknown vulnerability. Similarly, if they're malicious, they already own you because you depend on their code. This RFC does not introduce any new problems or attack kinds - all it does is add another vector that is more complex and easier to fix than existing ones.
Edit2: Another way of saying it is you're essentially arguing "this feature means that given sufficient control of a popular package to make innocuous seeming but actually malicious changes essentially any compromise is possible", and the proponents of this feature are saying there are already so many ways for that level of access to be exploited this makes no difference.
And on the other hand every week I see a few issues about "regex DDOS" for tools that usually go nowhere near user-generated input. The signal/noise ratio of NPM audit is very poor, and right now I think its main impact is to pressure people to update often. All of that creates a very fertile ground to distribute malicious code through NPM.
Maintainer already released clean versions "on top of" the compromised ones, and NPM acted on reports and removed the compromised versions as well.
Compromised (and no longer downloadable from NPM):
- 0.7.29
- 0.8.0
- 1.0.0
Clean:
- 0.7.28 (last version before the hijack)
- 0.7.30
- 0.8.1
- 1.0.1
Compromised versions apparently contained a cryptomining tool capable of running on Linux, and a trojan that extracts sensitive data (saved passwords, cookies) from browsers on Windows. Both are blocked by up-to-date Windows Defender and presumably other AV software.
We pin all of our npm dependencies and upgrade them via dependabot. Dependabot links to the GitHub or GitLab release for each dependency bump, and I typically skim / scan every single commit to each dependency. But there's no guarantee that what's on GH matches what is uploaded to npm (which is what happened in this case; there are no malicious commits).
Does anyone know of a good way to verify that a npm release matches what's on GH? Version controlling the entirety of node_modules/ and running untrusted updates in a sandbox would work in theory, but in practice many packages contain minified js which makes the diffs between version bumps unreadable.
Skip the nonsense and just check your dependencies in directly to your repo. The separation has no real world gains for developers and doesn't serve anyone except the host of your source repo. As it turns out most people's repo host is also the operator of the package registry they're using, so there aren't even theoretical gains for them, either.
Doing it this way doesn't preclude the ability to upgrade your dependencies, it _completely_ sidesteps the intentional or unintentional desync between a dependency's source and its releases, it means people have to go out of their way to get a deployment that isn't reproducible, and in 4 years when your project has rotted and someone tries to stand it up again even if just temporarily to effect some long-term migration, then they aren't going to run into problems because the packages and package manager changed out from beneath them. I run into this crap all the time to the point that people who claim it isn't a problem I know have to be lying.
> Does anyone know of a good way to verify that a npm release matches what's on GH?
I'm not aware of any way to do this, and it's a huge problem. It would be great if they introduced a Docker Hub verified/automated builds[0]-type thing for open source projects. I think that would be the only way we could be certain what we're seeing on GitHub is what we're running.
Honestly it’s hard to believe we all just run unverifiable, untrustable code. At the very least NPM they could require package signing, so we'd know the package came from the developer. But really NPM needs to build the package from GitHub source. Node is not a toy anymore, and hasn't been for some time—or is it?
Why is this a link to Reddit? Usually that’s when the link is to the specific Reddit discussion because that’s what is being shared, but that’s not the case here.
I don't know why anyone would auto-download untrusted packages as part of their own build/release chain and not review the code manually. Anything could be in there.
Totally agree, this is a huge reason I use Yarn. Reproducibility.
The catch is, particularly with yarn 3, if you want the soundness and capability to vendor zip files (completely removing the concept of node_modules) you need to use PnP and deal with some packages that don’t define their dependencies very well.
There’s still packages that will try to import/require stuff they dont declare, which is unsound. So when you get the warning, you need to declare it yourself in the root yarn config.
It’s a bit annoying, but the payoff is soundness.
You can clone to a new machine (be it a for a dev, or ci/cd) and be good to go without touching the network.
It’s like going from js to ts. Extra rigour for a price, but more often than not the price is worth it.
Even if they were trusted, it's still a horrific waste of resources. Maybe it's related to the insane trendchasing attitude ("gotta get the latest!!!!!111") that has infected JS/web development. Combine that with "free" cloud CI services and the general spreading stupidity of not knowing where the data actually is or what's going on (popularity of "quick tutorials" that have completely clueless noobs running one-line commands which can silently manipulate gigabytes of data), and it's not hard to see the result.
Our approach to solving this on our open source project:
1. Enforce use of lock files
2. On any PR that changes any hash in the lock file, run a static analysis that looks for “permission” changes in any source file. E.g., used network, used env vars, used filesystem
3. Comment on the PR with any observed permission changes
It is very alpha-quality and not yet generally available but here are some details: https://deps.semgrep.dev/
I'm only somewhat superficially familiar with JavaScript (and only in the browser, not NodeJS), but you can probably get far more obfuscated than that; or at least, you can in some other similar-ish languages. For example in Ruby on Rails they use (or used? It's been years so may be different now) all sorts of meta-programming stuff, which at times actually made it somewhat difficult to find a method definition if you didn't know where to look; it was all dynamically defined at runtime.
My worry with such a tool would be that malicious actors would go out of their way to hide their code from such tools, and that the tool would give me a false sense of security.
An unfortunate side-effect of this type of system, in my experience, is that you end up being stuck on very old dependencies for a very long time.
When a security issue is discovered, perhaps months or years later, it is extremely difficult to upgrade because
1) the patch will always be applied to the most current release, which could be many major versions beyond what you run, and
2) manually backporting the security fix is impossible without creating your own fork and the code will likely be so different as to not be able to cleanly apply the patch.
You are creating so much friction to update dependencies that you will find yourself unable to update them in the future.
Small example: I recently upgraded a dependency that required us to move from node 12 to node 14. That required us to upgrade more packages, including our database ORM. The ORM was 2 years out of date. Now we have changes that literally affect the entire application and require significant regression testing (automated and manual) because the risk surface is so much larger. This wouldn't have happened if we didn't use a lock file with pinned versions.
------
In the past (node 4.x), lock files weren't written by default, and I (on a small project) had sufficient test coverage that always allowed NPM to install the latest minor version bump of every package, every time. If there was a problem caught by the tests, then I immediately fixed the issue with the upgrade.
The nice thing about this is that I was always on the latest version of every dependency, so I always had the latest security fixes, and the online docs for the dependencies were always accurate to what I was using.
If this particular exploit happened, it would have been more likely to get deployed but it would have also automatically gotten fixed as soon as the patch update rolled around. Whereas with a lock file, someone may have updated it and then not been aware when the "fixed" version came out because npm/yarn don't tell you that.
As a JavaScript dev (full stack) I think lock files are a bad idea by making us LESS secure, not more (keeping around insecure older dependencies that can't be patched).
NPM/any package manager should not let you upload/manage packages with more than 1k downloads a month without 2FA. It's a liability and this is bound to happen.
IMO they—NPM especially—should also prevent publishing arbitrary packages where the source is accessible. In such cases, `npm publish` should be an authorization check and run your publish lifecycle to produce the package.
It’s absolutely bonkers that an ecosystem where the vast majority of packages are permissively licensed allows those packages to be published with basically whatever the heck you want, completely independent of whatever you would find inspecting and building it yourself.
Does any package ecosystem do this? Rust had the opportunity to start off doing the right thing but instead uses long-lived API tokens which can get easily owned.
This can be difficult to integrate into CI systems though. I don't recall ever seeing a way to allow interactive input during a CI pipeline to provide a 2FA token.
I'd argue that a completely automated release pipeline still trump's a manual/local process with 2fa (for reproducibility and transparency mainly).
Being able to have a hold step that also prompts for a token in order to publish in a CI system could be pretty neat though and best of both worlds
How will this work with say typescript? It's pretty common to not commit the transpiled code - do you expect this to change, or would this check basically do a build + pack and then compare?
This appears to affect other popular packages as well, e.g. fbjs.
It‘s time for proper supply chain security, the typo- and namesquatting minefield that is Pypi shows clearly why there is a need. Way too easy to distribute malware this way or register repos for popular packages by appending a version number for example. I‘m constantly running into shady deserted packages. Why not partnering with a large security vendor? The infrastructure is often provided as a donation from cloud providers as well
“ I noticed something unusual when my email was suddenly flooded by spams from hundreds of websites (maybe so I don't realize something was up, luckily the effect is quite the contrary).”
I’ve been running a small deal aggregation and notification website for 15 years. Almost no new users these days. A few months ago I was excited to see dozens of new accounts created each day but no one ever clicked the verification link that was sent to their email. That’s when I first learned that creating unused accounts is a technique used by hackers to flood people’s inboxes to hide fraudulent purchases or other nefarious activity like publishing a hijacked npm package. A simple what is x + y? captcha solved the issue.
For packages that popular, authority of more than one people should required or at least some kind of delay until npm update/install command catch it unless specifically that version set.
If i remember correctly if someone hijacked my npm account and published something i wouldn't even noticed it. Obviously publisher should get an email or maybe even sms.
Pretty sure that every time (or almost every time) when I run `npm install` in a repo downloaded from Github, NPM complains about several security problems in the dependencies. Kinda feels like an unending ‘cry wolf’ situation.
I wish NPM had an option to disallow pulling dependencies from accounts that don’t use 2FA. Even if you trust a publisher it is so easy for their packages to get compromised because they are lax about security.
That the package is downloaded 7 million times a week exacerbates the impact significantly. The CI/CD trend of pulling everything from upstream over and over for builds (despite caching, as the absurd NPM download counts indicate) ensures that any upstream issue is distributed widely and frequently.
Continuous Integration and Continuous Deployment of vulnerabilities.
This is the price of free code. If you don't read the open source you're adding to your system then don't complain when it owns you! You wouldn't take your medicine without reading the label first.
NPM is a miracle for open software development, but it requires diligence.
The JavaScript culture around dependencies is baffling to me.
The simplest of tools will easily have hundreds of transitive dependencies, making these sort of events a mess to deal with, and causing the blast radius to be avoidably large.
I'd encourage everyone to try the npm fund command (https://docs.npmjs.com/cli/v7/commands/npm-fund) on their repos to see how many projects need funding. All of these projects are an opportunity for a malicious actor to offer a lump sum of money to get the package.
I'm usually pretty critical of tiny NPM modules, but I really want to support whoever maintains this package.
Hundreds of lines of regexes, people asking about IE8 support, and a constantly moving target every time a browser changes something. This really must be a horrible package to maintain.
[+] [-] eyelidlessness|4 years ago|reply
Should this proposal go through, it’s quite possible this would have gone much further. It would have allowed the attacker to wait for detection, and to silence any propagation of that to users of the otherwise legitimate package.
I had to bow out of the conversation because I didn’t feel like I could engage it in a healthy way—I’m way too personally affected by my own experience dealing with dismissal of serious security issues—but I think folks here might be interested in it given this compromise.
1: https://github.com/npm/rfcs/pull/422
[+] [-] iudqnolq|4 years ago|reply
Edit: I think your misunderstanding is you're considering "malicious config" and "malicious code" significantly different issues, and arguing malicious config won't be noticed as malicious because it's only malicious in the context of what it enables. I believe this is incorrect. Innocuous seeming usage of a legitimate feature that actually has malicious effect is not a new issue.
Looking at the issue in more detail you were told essentially exactly this.
> A far, far easier version that already exists is: put malicious code directly into an otherwise innocuous package (or as a dependency). No vulnerability warning would exist, and thus there'd be no need for assertions. In other words, your "exploit" is just a more complex way of doing something that has always been possible.
> Once you've trusted a maintainer, you have already completely surrendered any protections to their judgement. If they make a mistake with an assertion, they could also make a mistake with code, or by depending on a package with an unknown vulnerability. Similarly, if they're malicious, they already own you because you depend on their code. This RFC does not introduce any new problems or attack kinds - all it does is add another vector that is more complex and easier to fix than existing ones.
Edit2: Another way of saying it is you're essentially arguing "this feature means that given sufficient control of a popular package to make innocuous seeming but actually malicious changes essentially any compromise is possible", and the proponents of this feature are saying there are already so many ways for that level of access to be exploited this makes no difference.
[+] [-] Zababa|4 years ago|reply
[+] [-] ComputerGuru|4 years ago|reply
[+] [-] EdwardDiego|4 years ago|reply
[+] [-] olex|4 years ago|reply
Compromised (and no longer downloadable from NPM):
- 0.7.29
- 0.8.0
- 1.0.0
Clean:
- 0.7.28 (last version before the hijack)
- 0.7.30
- 0.8.1
- 1.0.1
Compromised versions apparently contained a cryptomining tool capable of running on Linux, and a trojan that extracts sensitive data (saved passwords, cookies) from browsers on Windows. Both are blocked by up-to-date Windows Defender and presumably other AV software.
[+] [-] eemax|4 years ago|reply
Does anyone know of a good way to verify that a npm release matches what's on GH? Version controlling the entirety of node_modules/ and running untrusted updates in a sandbox would work in theory, but in practice many packages contain minified js which makes the diffs between version bumps unreadable.
[+] [-] pwdisswordfish0|4 years ago|reply
Doing it this way doesn't preclude the ability to upgrade your dependencies, it _completely_ sidesteps the intentional or unintentional desync between a dependency's source and its releases, it means people have to go out of their way to get a deployment that isn't reproducible, and in 4 years when your project has rotted and someone tries to stand it up again even if just temporarily to effect some long-term migration, then they aren't going to run into problems because the packages and package manager changed out from beneath them. I run into this crap all the time to the point that people who claim it isn't a problem I know have to be lying.
[+] [-] dmitryminkovsky|4 years ago|reply
I'm not aware of any way to do this, and it's a huge problem. It would be great if they introduced a Docker Hub verified/automated builds[0]-type thing for open source projects. I think that would be the only way we could be certain what we're seeing on GitHub is what we're running.
Honestly it’s hard to believe we all just run unverifiable, untrustable code. At the very least NPM they could require package signing, so we'd know the package came from the developer. But really NPM needs to build the package from GitHub source. Node is not a toy anymore, and hasn't been for some time—or is it?
[0] https://docs.docker.com/docker-hub/builds/
[+] [-] taion|4 years ago|reply
It's also great at doing bulk updates, so you get a lot less spam than you do from Dependabot.
[+] [-] brundolf|4 years ago|reply
[+] [-] eyelidlessness|4 years ago|reply
[+] [-] ComputerGuru|4 years ago|reply
@mods I think the link should be to the actual post here: https://github.com/faisalman/ua-parser-js/issues/536
[+] [-] gnu8|4 years ago|reply
The way people use NPM is pure fucking insanity.
[+] [-] Aeolun|4 years ago|reply
[+] [-] ojkelly|4 years ago|reply
The catch is, particularly with yarn 3, if you want the soundness and capability to vendor zip files (completely removing the concept of node_modules) you need to use PnP and deal with some packages that don’t define their dependencies very well.
There’s still packages that will try to import/require stuff they dont declare, which is unsound. So when you get the warning, you need to declare it yourself in the root yarn config.
It’s a bit annoying, but the payoff is soundness.
You can clone to a new machine (be it a for a dev, or ci/cd) and be good to go without touching the network.
It’s like going from js to ts. Extra rigour for a price, but more often than not the price is worth it.
[+] [-] userbinator|4 years ago|reply
[+] [-] austincheney|4 years ago|reply
https://en.m.wikipedia.org/wiki/Invented_here
[+] [-] imtringued|4 years ago|reply
[+] [-] unknown|4 years ago|reply
[deleted]
[+] [-] ievans|4 years ago|reply
1. Enforce use of lock files 2. On any PR that changes any hash in the lock file, run a static analysis that looks for “permission” changes in any source file. E.g., used network, used env vars, used filesystem 3. Comment on the PR with any observed permission changes
It is very alpha-quality and not yet generally available but here are some details: https://deps.semgrep.dev/
[+] [-] arp242|4 years ago|reply
My worry with such a tool would be that malicious actors would go out of their way to hide their code from such tools, and that the tool would give me a false sense of security.
[+] [-] Osiris|4 years ago|reply
When a security issue is discovered, perhaps months or years later, it is extremely difficult to upgrade because
1) the patch will always be applied to the most current release, which could be many major versions beyond what you run, and
2) manually backporting the security fix is impossible without creating your own fork and the code will likely be so different as to not be able to cleanly apply the patch.
You are creating so much friction to update dependencies that you will find yourself unable to update them in the future.
Small example: I recently upgraded a dependency that required us to move from node 12 to node 14. That required us to upgrade more packages, including our database ORM. The ORM was 2 years out of date. Now we have changes that literally affect the entire application and require significant regression testing (automated and manual) because the risk surface is so much larger. This wouldn't have happened if we didn't use a lock file with pinned versions.
------
In the past (node 4.x), lock files weren't written by default, and I (on a small project) had sufficient test coverage that always allowed NPM to install the latest minor version bump of every package, every time. If there was a problem caught by the tests, then I immediately fixed the issue with the upgrade.
The nice thing about this is that I was always on the latest version of every dependency, so I always had the latest security fixes, and the online docs for the dependencies were always accurate to what I was using.
If this particular exploit happened, it would have been more likely to get deployed but it would have also automatically gotten fixed as soon as the patch update rolled around. Whereas with a lock file, someone may have updated it and then not been aware when the "fixed" version came out because npm/yarn don't tell you that.
As a JavaScript dev (full stack) I think lock files are a bad idea by making us LESS secure, not more (keeping around insecure older dependencies that can't be patched).
[+] [-] meibo|4 years ago|reply
[+] [-] eyelidlessness|4 years ago|reply
It’s absolutely bonkers that an ecosystem where the vast majority of packages are permissively licensed allows those packages to be published with basically whatever the heck you want, completely independent of whatever you would find inspecting and building it yourself.
[+] [-] ferdowsi|4 years ago|reply
[+] [-] mnahkies|4 years ago|reply
I'd argue that a completely automated release pipeline still trump's a manual/local process with 2fa (for reproducibility and transparency mainly).
Being able to have a hold step that also prompts for a token in order to publish in a CI system could be pretty neat though and best of both worlds
[+] [-] koolba|4 years ago|reply
Though rather than mandating it, just show if the publisher has it enabled and let the market decide.
[+] [-] Ros2|4 years ago|reply
I got a PR in my repository a few days ago leading back to a team trying to make it easier for packages to be reproducible from source https://github.com/microsoft/Secure-Supply-Chain
[+] [-] mnahkies|4 years ago|reply
[+] [-] ahuth|4 years ago|reply
Better one might be the GitHub issue discussing it: https://github.com/faisalman/ua-parser-js/issues/536
[+] [-] schleck8|4 years ago|reply
It‘s time for proper supply chain security, the typo- and namesquatting minefield that is Pypi shows clearly why there is a need. Way too easy to distribute malware this way or register repos for popular packages by appending a version number for example. I‘m constantly running into shady deserted packages. Why not partnering with a large security vendor? The infrastructure is often provided as a donation from cloud providers as well
[+] [-] tppiotrowski|4 years ago|reply
I’ve been running a small deal aggregation and notification website for 15 years. Almost no new users these days. A few months ago I was excited to see dozens of new accounts created each day but no one ever clicked the verification link that was sent to their email. That’s when I first learned that creating unused accounts is a technique used by hackers to flood people’s inboxes to hide fraudulent purchases or other nefarious activity like publishing a hijacked npm package. A simple what is x + y? captcha solved the issue.
[+] [-] alienalp|4 years ago|reply
[+] [-] aasasd|4 years ago|reply
[+] [-] paxys|4 years ago|reply
[+] [-] knadh|4 years ago|reply
Continuous Integration and Continuous Deployment of vulnerabilities.
[+] [-] ipnon|4 years ago|reply
NPM is a miracle for open software development, but it requires diligence.
[+] [-] godmode2019|4 years ago|reply
I started a project in react and it installed like 100+ random packages. I had to audit everyone, so silly not to have a standard library.
[+] [-] crehn|4 years ago|reply
The simplest of tools will easily have hundreds of transitive dependencies, making these sort of events a mess to deal with, and causing the blast radius to be avoidably large.
[+] [-] Zababa|4 years ago|reply
[+] [-] technion|4 years ago|reply
Hundreds of lines of regexes, people asking about IE8 support, and a constantly moving target every time a browser changes something. This really must be a horrible package to maintain.