Thing is, node_modules isn't just thirdparty modules.
It's just what the Node path resolution standard uses, which is also how Local Modules (the "/src/node_modules" structure) allows you to import other files with clean paths, without having to add a config in every tool of the toolchain, post-install symlinks, or any other non-crossplatform tricks. It just works because it's literally what Node uses to resolve paths, and all build tools are based on Node, so when you stick to the standard resolution, you can add new tools to the toolchain without needing a bunch of configs for them to find your files. For example, it's now also the default resolution in Typescript as well.
The only time /src/node_modules doesn't work is when tool goes out of its way to break it, and wrongly assumes that node_modules can only ever be used for external thirdparty code (e.g. Jest).
So best of luck to make Node + NPM + Yarn to agree on a new path resolution syntax, but I hope we won't end up with another tool-specific resolution that only works in Yarn.
This doesn't break that, it specifically says it will fall back to Node's module resolution algorithm when the module you're looking isn't in the static resolutions table. That means you can keep using that technique as you have bee.
As an aside, you can also use lerna[1], yarn workspaces[2] or pnpm workspaces[3] to achieve the same effect, depending on your package manager of choice. You might get additional boosts to code organization/productivity, it's explained in the links.
I'm the author behind the proposal, feel free to ask me any question you might have!
As a personal note, I'm super excited and so grateful to have had the chance to work on this project - node_modules have been a longstanding thorn in the side of the Javascript ecosystem, and to finally have a chance to try something completely new is amazing.
This is awesome. I didn't read the full proposal yet, but at first glance it's very close to how we do things where I work (because we needed something long before Node even existed, and never replaced it).
The biggest hurdle we constantly face is tools that make assumptions about the environment (eg: Flow, TypeScript, Webstorm, etc), so changing anything to module resolution breaks everything. We have to spend a lot of time hacking it in every single time. Sometimes the hooks just aren't there (TypeScript has a hook for overriding module resolution, but editors often don't give you an easy way to hook into TypeScript's own hooks).
Any thoughts on how things would work for tools if this was to go forward? Would they all need to be updated to honor this new way to resolve modules?
This is a bit of a tangent, but do you know of good tutorials, blog posts, videos, or books which help build a mental framework of how javascript packaging works internally?
My goal is to be better at debugging development environment issues.
To be more specific, I am asking this from the perspective of a user of the node ecosystem who has repeatedly had the experience of running into problems with my development environment or packages and not really knowing what to do to get myself un-stuck. I can totally search google/stack overflow/github for an answer and sometimes there is a clear one. Sometimes I fond a command to run, sometimes I don't. In either case, I come away with a strong feeling that I've not actually learned anything. If the same thing happened again, I might remember what to google for next time, but I wouldn't be able to reason clearly about what I'm working with. In the past when I run into these issues, I've just thrown time at them until something sticks. That kinda sucks.
I'd like to find a way to put in some work and come away with a solid understanding. When you were starting this project, how did you go about building up your knowledge of the underlying abstractions that npm/yarn/webpack/etc. use?
When you see a misconfiguration or get a bug report with this project, how do you go about investigating it?
Have you ever seen a really good explanation of a bug/misconfiguration which helps the reader build a solid mental model?
I quickly read the paper and I wonder if there's anything in place to deal with packages that generate side effects via postinstall. Can we at least manually create a list of them in package.json or something so that Yarn copies them into node_modules like before?
This seems to be solving a problem quite similar to the one of getting browsers to resolve non-relative imports (without file system access, they also need a precise map from module names to paths). Unfortunately, I can't remember who or what was working on that and I'm not sure how much progress it is making, but it'd be wonderful if this could somehow be framed in a way where it also helps that issue along. Are you taking this into consideration in the design?
Hey, I just saw a tweet about this, it looks super exciting. A question I have is how does this work with yarn workspaces? I have a monorepo setup and would love to try this, but I use both yarn workspaces and lerna to run commands across packages. Will nohoist be respected?
Another question—does this also resolve from directory paths to index.js, add missing extensions, and so on, or does it only map package names to directories? As in, does this get rid of all 'superfluous' file system access during resolution?
Removing `node_modules` would be fantastic, but not at the expense of native modules:
From the PDF:
"On the long term we believe that post-install scripts are a problem by
themselves (they still need an install step, and can lead to security issues),
and we would suggest relying less on this feature in the future, possibly
even deprecating it in the long term. While native modules have their
usefulness, WebAssembly is becoming a more and more serious candidate
for a portable bytecode as the months pass."
WebAssembly is not "more and more a serious candidate" to replace native modules.
The issue with post-install scripts needs a better long-term solution, but simply deprecating native modules is not it.
I mentioned it in another answer, but we'll eject the packages that require postinstall scripts - so they will work as before, but will take extra time to be installed.
As for wasm, I'm curious to hear what you think isn't good enough. I think the two main issues are garbage collection and dynamic linking, and there's ongoing work on them to fix them.
I didn't dive deeply enough, but I'd assume they could just fallback to classic node_modules behavior for the native packages, right? You'd not actually delete the node_modules directory then but still get most of the speed benefits.
npm announced crux yesterday, seems focused on the same problems. Approach seems slightly different, although there aren't as much technical details available for crux (unless someone has a better link):
This is effectively how pub, the package manager for Dart works. At version resolution time, we generate a manifest file that contains the local paths to every depended-upon package. Then, at compile/load time, tools just use that file to locate packages.
Very cool, does this mean you can check in the dependency file to version control? Just like yarn.lock?
As much as I hate node_modules, there are times when I want to see how a library is implemented. Is there a way to have some libraries in node_modules? Say only the ones in listed in the package.json file
Second this. I'd ask that any dependencies in node_modules/ override whatever is cached.
For a work project, I'm updating a dependency written against an undocumented specification. Much fun is being had doing this, I assure you. Being able to crack open the sources in node_modules/, instrument things with logging, and fix fix fix until I get something working is very helpful. I'm sure I can track things down in a cache directory and do the same, but it's nice knowing that my changes are limited to just this particular project, and that a checkout/reinstall of the module in another project will get the original broken sources for comparison.
The .pnp.js file can be checked-in, but needs the cache to work. Right now I'd advise not to check-it in.
`yarn unplug` will eject a dependency from the cache and put it into the project folder. That's a quick and easy way to inspect and debug libraries you use.
I've been trying to clean up my excess node modules and found that when building a web-socket app, you don't need express (native http works fine) and you don't even need socket.io (unless your project needs to work on IE 9 or less) It's hard to find examples so I wrote one: https://github.com/chrisbroski/bare-socket
Bringing packages into your work is your own choice. You should always be doing an evaluation of the benefit given by a package vs. the bloat it brings to your project. For many packages, it is worth it. But if you are in the habit of installing a package just to avoid writing a widget in your UI or writing some helper functions, then... yes, your projects will amass dependencies quickly.
Put your content inside of a nested node_modules folder and you won't have this problem anymore.
I don't think node_modules is perfect, and I get why it gets hate, but IMHO the algorithm is actually kinda nice for nesting packages.
If you set up your folders like so:
----
src--->node_modules--->utils--->helper.js
src--->main.js
----
You can require your helper in main.js with a simple ``require('utils/helper.js');``
What's nice is that you can't do the reverse. So your helper doesn't get the same access to your main.js. I use this a lot for testing - it means that my tests can require my components using the same syntax that I use in normal code, but those components don't have special access to the tests.
A big "aha" moment for me with node_modules was figuring out that it's an entirely separate system from npm. It's not trying to build a flat package structure; it's trying to make it easy to on-the-fly set up packages from anywhere.
I've also gotten into the habit of checking my installed packages into source for projects that I don't expect users to rebuild (ie. websites, games, etc...). That's a much longer conversation, but it mitigates a large number of issues with node_modules on long-lived projects.
while it would be great to have a standard way to do absolute imports, I almost like how it force people into more sensible project structures and splitting large projects into multiple packages. The "smell" of the ../../../ encourages better practices, so it's not all bad IMO.
I just saw two comments comparing Yarn's RFC to Maven - yours and one over on Lobsters.
Having never used Maven, any chance you could point to a document explaining how Maven's cache approach works, and maybe expand on the similarities between that and Yarn's RFC?
Nice work – what's being done to make this the default behavior in node and deprecate node_modules proper, or is this forever destined to live in the awkward place of userland-but-not-really kind of territory?
People have already mentioned native modules. Install scripts are a nuisance, but exist for reasons. If you remove support for them – provided this project takes off, which I suppose it will because bandwagons – you risk people performing install-type tasks on first module load, potentially creating an even worse situation. Has this been considered as a risk?
> what's being done to make this the default behavior in node and deprecate node_modules proper
My hope is that PnP proves that this approach can work and raises the stakes for Node to implement APIs that would allow for a better integration. It's a long term plan and we have yet to prove ourselves on the long run, but I'm quite confident :)
> If you remove support for them
I don't think we'll ever remove support for them. I'll personally advocate for them not to be used because of their subpar developer experience, but apart from that it won't cost us much to support them.
From what I understand: instead of cp -rf from offline cache, or ln -sf (which doesn't work for a large percentage of packages due to node being node), they propose to use a custom resolver to tap into the cache directly.
This will also break for various packages due to fs.readFile* dependencies, gyp, and other things. If your dependency tree is 4k+ node modules, the "vanilla" yarn or npm resolution and reconciliation techniques are already so brittle, that changing them will undoubtedly break things.
Currently I use npm for angular projects. Before writing even hello world , I have to perform npm install on the project and install so many module to serve up small web app. This seems like we are going in the right direction, cant wait to try yarn as package manager.
The problem in javascript seems more an ideological one where programmers instantly reach for a package that has multiple dependencies of which those dependencies have even more dependencies.
There's this culture of not caring about bloat it seems in the vast majority of javascript projects. left_pad comes to the mind as the poster boy for this stuff.
[+] [-] wildpeaks|7 years ago|reply
It's just what the Node path resolution standard uses, which is also how Local Modules (the "/src/node_modules" structure) allows you to import other files with clean paths, without having to add a config in every tool of the toolchain, post-install symlinks, or any other non-crossplatform tricks. It just works because it's literally what Node uses to resolve paths, and all build tools are based on Node, so when you stick to the standard resolution, you can add new tools to the toolchain without needing a bunch of configs for them to find your files. For example, it's now also the default resolution in Typescript as well.
The only time /src/node_modules doesn't work is when tool goes out of its way to break it, and wrongly assumes that node_modules can only ever be used for external thirdparty code (e.g. Jest).
So best of luck to make Node + NPM + Yarn to agree on a new path resolution syntax, but I hope we won't end up with another tool-specific resolution that only works in Yarn.
[+] [-] cprecioso|7 years ago|reply
As an aside, you can also use lerna[1], yarn workspaces[2] or pnpm workspaces[3] to achieve the same effect, depending on your package manager of choice. You might get additional boosts to code organization/productivity, it's explained in the links.
[1]: https://lernajs.io [2]: https://yarnpkg.com/lang/en/docs/workspaces/ [3]: https://pnpm.js.org/docs/en/workspace.html
[+] [-] arcatek|7 years ago|reply
As a personal note, I'm super excited and so grateful to have had the chance to work on this project - node_modules have been a longstanding thorn in the side of the Javascript ecosystem, and to finally have a chance to try something completely new is amazing.
[+] [-] shados|7 years ago|reply
The biggest hurdle we constantly face is tools that make assumptions about the environment (eg: Flow, TypeScript, Webstorm, etc), so changing anything to module resolution breaks everything. We have to spend a lot of time hacking it in every single time. Sometimes the hooks just aren't there (TypeScript has a hook for overriding module resolution, but editors often don't give you an easy way to hook into TypeScript's own hooks).
Any thoughts on how things would work for tools if this was to go forward? Would they all need to be updated to honor this new way to resolve modules?
[+] [-] afarrell|7 years ago|reply
My goal is to be better at debugging development environment issues.
To be more specific, I am asking this from the perspective of a user of the node ecosystem who has repeatedly had the experience of running into problems with my development environment or packages and not really knowing what to do to get myself un-stuck. I can totally search google/stack overflow/github for an answer and sometimes there is a clear one. Sometimes I fond a command to run, sometimes I don't. In either case, I come away with a strong feeling that I've not actually learned anything. If the same thing happened again, I might remember what to google for next time, but I wouldn't be able to reason clearly about what I'm working with. In the past when I run into these issues, I've just thrown time at them until something sticks. That kinda sucks.
I'd like to find a way to put in some work and come away with a solid understanding. When you were starting this project, how did you go about building up your knowledge of the underlying abstractions that npm/yarn/webpack/etc. use?
When you see a misconfiguration or get a bug report with this project, how do you go about investigating it?
Have you ever seen a really good explanation of a bug/misconfiguration which helps the reader build a solid mental model?
[+] [-] russellbeattie|7 years ago|reply
[+] [-] eliseumds|7 years ago|reply
[+] [-] marijn|7 years ago|reply
[+] [-] audiolion|7 years ago|reply
[+] [-] marijn|7 years ago|reply
[+] [-] suchabag|7 years ago|reply
[+] [-] k__|7 years ago|reply
[+] [-] _urga|7 years ago|reply
From the PDF:
"On the long term we believe that post-install scripts are a problem by themselves (they still need an install step, and can lead to security issues), and we would suggest relying less on this feature in the future, possibly even deprecating it in the long term. While native modules have their usefulness, WebAssembly is becoming a more and more serious candidate for a portable bytecode as the months pass."
WebAssembly is not "more and more a serious candidate" to replace native modules.
The issue with post-install scripts needs a better long-term solution, but simply deprecating native modules is not it.
[+] [-] arcatek|7 years ago|reply
As for wasm, I'm curious to hear what you think isn't good enough. I think the two main issues are garbage collection and dynamic linking, and there's ongoing work on them to fix them.
[+] [-] skrebbel|7 years ago|reply
EDIT: just saw a comment by the author that says roughly this: https://news.ycombinator.com/item?id=17977986
So, native packages will keep on working.
[+] [-] vorpalhex|7 years ago|reply
[+] [-] ohitsdom|7 years ago|reply
https://blog.npmjs.org/post/178027064160/next-generation-pac...
edit: found the repo, seems a bit behind yarn's effort and not yet beta status. https://github.com/npm/crux
[+] [-] munificent|7 years ago|reply
[+] [-] gitgud|7 years ago|reply
As much as I hate node_modules, there are times when I want to see how a library is implemented. Is there a way to have some libraries in node_modules? Say only the ones in listed in the package.json file
[+] [-] ndarilek|7 years ago|reply
For a work project, I'm updating a dependency written against an undocumented specification. Much fun is being had doing this, I assure you. Being able to crack open the sources in node_modules/, instrument things with logging, and fix fix fix until I get something working is very helpful. I'm sure I can track things down in a cache directory and do the same, but it's nice knowing that my changes are limited to just this particular project, and that a checkout/reinstall of the module in another project will get the original broken sources for comparison.
[+] [-] arcatek|7 years ago|reply
`yarn unplug` will eject a dependency from the cache and put it into the project folder. That's a quick and easy way to inspect and debug libraries you use.
[+] [-] jannes|7 years ago|reply
It angers me how many dependencies very simple projects amass.
[+] [-] zkochan|7 years ago|reply
pnpm saves one version of a package only ever once on a disk and node_modules consume drastically less space
[+] [-] the_duke|7 years ago|reply
You just need to forgo all the conveniences the ecosystem provides and write stuff yourself instead.
Node.js or the browser have everything you need.
[+] [-] protonfish|7 years ago|reply
[+] [-] codingdave|7 years ago|reply
[+] [-] lucisferre|7 years ago|reply
[+] [-] danShumway|7 years ago|reply
I don't think node_modules is perfect, and I get why it gets hate, but IMHO the algorithm is actually kinda nice for nesting packages.
If you set up your folders like so:
----
src--->node_modules--->utils--->helper.js
src--->main.js
----
You can require your helper in main.js with a simple ``require('utils/helper.js');``
What's nice is that you can't do the reverse. So your helper doesn't get the same access to your main.js. I use this a lot for testing - it means that my tests can require my components using the same syntax that I use in normal code, but those components don't have special access to the tests.
A big "aha" moment for me with node_modules was figuring out that it's an entirely separate system from npm. It's not trying to build a flat package structure; it's trying to make it easy to on-the-fly set up packages from anywhere.
Edit: example - https://gitlab.com/dormouse/dormouse/tree/master/src
I've also gotten into the habit of checking my installed packages into source for projects that I don't expect users to rebuild (ie. websites, games, etc...). That's a much longer conversation, but it mitigates a large number of issues with node_modules on long-lived projects.
[+] [-] shados|7 years ago|reply
[+] [-] MrEfficiency|7 years ago|reply
[+] [-] i386|7 years ago|reply
Hopefully yarn does a better job at validating dependencies than early Maven 1 and 2 did.
[+] [-] acemarke|7 years ago|reply
Having never used Maven, any chance you could point to a document explaining how Maven's cache approach works, and maybe expand on the similarities between that and Yarn's RFC?
[+] [-] olingern|7 years ago|reply
[+] [-] mstade|7 years ago|reply
People have already mentioned native modules. Install scripts are a nuisance, but exist for reasons. If you remove support for them – provided this project takes off, which I suppose it will because bandwagons – you risk people performing install-type tasks on first module load, potentially creating an even worse situation. Has this been considered as a risk?
[+] [-] arcatek|7 years ago|reply
My hope is that PnP proves that this approach can work and raises the stakes for Node to implement APIs that would allow for a better integration. It's a long term plan and we have yet to prove ourselves on the long run, but I'm quite confident :)
> If you remove support for them
I don't think we'll ever remove support for them. I'll personally advocate for them not to be used because of their subpar developer experience, but apart from that it won't cost us much to support them.
[+] [-] lxe|7 years ago|reply
This will also break for various packages due to fs.readFile* dependencies, gyp, and other things. If your dependency tree is 4k+ node modules, the "vanilla" yarn or npm resolution and reconciliation techniques are already so brittle, that changing them will undoubtedly break things.
[+] [-] cryptozeus|7 years ago|reply
[+] [-] unknown|7 years ago|reply
[deleted]
[+] [-] ericintheloft2|7 years ago|reply
[+] [-] aij|7 years ago|reply
I've been thinking about unifying our current ivy+yarn+bower setup, but haven't yet gotten much past thinking about it...
[+] [-] specialist|7 years ago|reply
From https://pnpm.js.org
"Efficient: One version of a package is saved only ever once on a disk. So you save dozens of gigabytes of disk space!"
Without digging, I imagine this will be more like Maven's cache.
NPM's design decision to flatten the version hierarchy baffled me. And has occasionally tripped me up.
[+] [-] Rockslide|7 years ago|reply
[+] [-] sergiotapia|7 years ago|reply
There's this culture of not caring about bloat it seems in the vast majority of javascript projects. left_pad comes to the mind as the poster boy for this stuff.
[+] [-] MrEfficiency|7 years ago|reply
Does any other language have a solution to this?
[+] [-] jhabdas|7 years ago|reply
[deleted]
[+] [-] jensvdh|7 years ago|reply
[+] [-] acemarke|7 years ago|reply