I've seen a lot of people criticise npm and their policies but I've never come across a solution. Npm has its flaws and while there are such abuses like everything package, is-odd, left-pad, etc there are also many useful packages like vue, sortable, etc without which development will be a huge pain.
So not asking rhetorically, if we had all the insight and knowledge we have now, how would you make it different?
I know it's five hours later and this question has already spawned dozens of responses, but it's worth thinking and speaking clearly if we're trying to arrive at a solution for something. We can start by saying exactly what we're talking about—how do you make what different? Because you mention npm "and their policies" but then switch gears and talk about "is-odd", which is not a policy issue. It's rather something else entirely.
If you want answers, state clearly what _specific_ problem you're trying to solve. Whatever the solution to it might be, vague and fuzzy questions—while magnets for chatter since they can stand in for whatever someone wants to read out of them—are not the way to get there.
(You could say that this is needlessly tedious because everyone already knows what we're talking about, but that this isn't true is exactly my position. It's certain that something like half the people reading, thinking, and writing are have in mind one thing, while the other half are thinking of another—and the third half are thinking about something different from either of those. We're also programmers, so dealing with tedium and the constraints of having to be explicit should be second nature.)
The core of the problem is micro dependencies. It seems in the Javascript ecosystem, developers have no awareness of costs of complexity.
When you wonder whether to add a dependency, you should ask yourself: What are the upsides and downsides of adding this dependency. One downside is always that by adding a dependency, you add a potential security problem, a potential point of breakage, and more complexity.
There are situations where these are well justified. If your dependency is stable, from a trustworthy source, and if it is a functionality that you cannot quickly implement yourself. But if you include a dependency that is effectively one line of code, the question answers itself: The costs of adding a dependency is completely unreasonable. It your list of dependencies grows into the 100s, you're doing something wrong.
Devs working with core developers to create more 1st party packages would be a good start. I don't need 12 different implementations for sorting on Vue/React/[insert spa framework of the month]. I just need 1 really good sorting library. With it, we can move to less overall dependencies on random packages.
Don't allow un-publishing package versions. If they are literally malware, they can be manually removed by npm admins. If a court orders a takedown due to copyright, that's also something npm admins can handle. If you want to be able to un-publish, then just publish on your own server (or github etc).
If analyzing the dependencies for showing in the NPM web UI, while analyzing, as you exceed 40 direct or transitive dependencies, abort and highlight this package in red, for having excessive dependencies.
If installing locally, you get what you get, don't install random or crazy packages, stick to well known high-quality minimal-dependencies packages. nodejs does include file reading and writing, http server, http client, json ... that will take you pretty far. Master the basics before getting too fancy. And remember, you don't need some company's client package just to make some http requests to their API.
I’ve started thinking package management has too much trust now. Ideally, but probably unpractically, projects should check in their packages like they used to under /lib or /third_party, and be much more suspicious of new package dependencies.
Basically, you would need to start accepting that you are responsible for any dependencies you choose to include. Any upstream changes you would need to evaluate and bring in or patch yourself.
Definitely an impossible task given how broad and deep modern package dependencies are, but at least you’d start feeling the insanity of having all if them in the first place :P.
IIRC, the Maven crowd was criticizing npm's decisions from the get go because they chose to ignore many of the problems the Java community already solved a decade before.
The entire reason this is a big deal is that people don't know what their dependencies are. The left-pad incident wasn't a big deal because it was pulled, it was a big deal because no one could easily fix their builds and didn't even know they were depending on it, because it was a dependency of a dependency of a dependency.
While it's ridiculous to expect that people will audit every single dependency and sub-dependency, it's not ridiculous to expect tooling to do the same.
Packages should be given an overall quality rating (and honestly it might be great for an ecosystem as large, diverse, and welcoming-to-beginners as JS/TS), part of the score comes from the number of different dependencies/sub-dependencies -- a social package score if you will. If a package causes the dependency graph to explode, give a warning before installing it.
Then, if you're NPM, you don't need all of these convoluted and exploitable policies around un-publishing.
Personally I would like it and the ecosystem to just cease to exist overnight. Nothing on earth has caused so much pain, misery, suffering and agony, apart from possibly PHP.
Our devops guys scream from the seething pain whenever the have to debug some pile of shit that decides it won't build unless all the runes are aligned precisely and all the RAM in the universe is available on the build runners. And pushing this to the developers results in importing more packages thus adding to the burning tyre fire.
And after several hours of builds and 9000 layers of packages you wake up one morning and in that 50 meg chunk of javascript that is excreted from the process, someone managed to inject a "Slava Ukraini!" banner into your web app.
The one thing clear in JavaScript is that if some developers think there is a better way, they will develop it and use it.
That hasn't happened with NPM because its about as useful as it conceivably could be.
The criticisms really amount to nothing, and tend to come from developers who don't even write JavaScript.
It doesn't have to be as painful as C++ without package managers, but should make every developer spend about 5~10 minutes labor work for adding each direct dependency, or one minute for each new dependency in the dependency closure.
When you use 'go get' to add a package to your Go project, it actually fetches the code through a Google proxy which saves a snapshot of the commit in question. Even if the original source goes away, they should have a copy of every version of the library ever fetched via their tool, and devs can continue to build existing stuff.
(If you don't want Google to see what packages you're fetching, you can also turn this off with an environment variable.)
I don't know that there's a solution because the fundamental cause of the problem is that Javascript has a huge dev base and everybody wants to have at least one active NPM package they maintain for their resume. Nobody ever asked me as a Perl programmer what CPAN packages I've created because very few Perl programmers made them, but hiring managers will look at Javascript devs' NPM footprint.
The solution is a proper deprecation mechanism with a grace period for migration. Restricting removal or allowing instant removal are the extremes that cause trouble.
Have smarter users. If your package breaks because it depends on trivial code which got deleted, you shouldn't have depended on that in the first place.
Preventing people from deleting their code -- always, or even just sometimes -- was never the right solution.
Soft deletes. You can delete a package and it stops being advertised but a shadow copy of referenced versions are kept for anything that depends on it. NPM spews warnings when this happens.
Once the referencing packages are updated are deleted or modified the shadow versions can be dropped.
I’m blown away by the reception of this article. It’s wildly low quality, generated SEO spam.
> It was removed, but then reemerged under a different scope with over 33,000 sub-packages. It's like playing whack-a-mole with npm packages!
> This whole saga is more than just a digital prank. It highlights the ongoing challenges in package management within the npm ecosystem. For developers, it's a reminder of the cascading effects of dependencies and the importance of mindful package creation, maintenance, and consumption.
> As we navigate the open source world, incidents like the everything package remind us of the delicate balance between freedom and responsibility in open-source software.
The "delicate balance between X and Y" is an LLM tic[0]. Especially llama -based language models have a habit of ending any longer piece of text with a phrase like that.
Source: have done a bunch of AI-assisted writing to develop my own skills and the tics and specific turns of phrases really pop out to me.
"accidentally broke NPM and all I got was this sweet permanent banner all over my Github
(thats impossible to remove since they probably had to code it up last minute before removing the org/repo)"
When I was consulting for an R&D lab at eBay, we open sourced a bunch of our work in a GitHub org. It was sanctioned by eBay's OSPO; they even linked to it from their main open source directory.
7 years later, long after the team disbanded, someone in eBay's current legal team decided that the (now archival) org violated eBay's trademarks. For the last year+, every time I've opened GitHub, I've been met with the same undismissable banner.
Since the only choice they give you is to contact support, I did. Unfortunately, their support team is not responsive, and has a completely separate notifications system. It took an inordinately long time for them to respond. (I have poor reception here so I can't check, but I think it was months.) Since I'm not in the habit of checking GitHub Support for new messages, when they eventually replied, I missed it. I had to start a whole new ticket. That too was months ago, and I still haven't heard back.
So because I did some work for a skunkworks eBay team in 2015, the top 150px of my GitHub are unusable, and there's apparently nothing I can do about it until some call center decides to write me back.
I think there's a point when you're trying to do something really stupid and hack around the defences (e.g. rate limits and package JSON file sizes) that it's no longer an accident.
Just as a side note about the screenshot at the end. I think it's from this socket thing, but the supply chain security of a package that depends on literally anything on npm having a score of almost 50 really makes me think if that score is just artificially inflated on every other package. Can you even reach a score below 47?
Founder of socket here. npm has since unpublished the chunk packages that the 'everything' package depends on (or perhaps made them private), so those packages are no longer being taken into account in the package score.
You're right that a package that depends on literally everything would absolutely have a score of 0 in our system.
The article is totally misleading there is no storage space running out and system resource exhaustion. btw the total size is around 30MB or less than 50
The only thing is no one can unpublish the npm package because npm have policy if one package is depend on your package you can't unpublish it
> The "everything" package, with its 5 sub-packages and thousands of dependencies, has essentially locked down the ability for authors to unpublish their packages. This situation is due to npm's policy shift following the infamous "left-pad" incident in 2016, where a popular package left-pad was removed, grinding development to a halt across much of the developer world. In response, npm tightened its rules around unpublishing, specifically preventing the unpublishing of any package that is used by another package.
Has no one thought of that? It seems like it should have been obvious that such an absolute rule could be easily abused to troll the system at scale.
Not sure if it's a problem though, perhaps all unpublishing requests should be reviewed by someone at the registry (and granted only when it makes sense).
Go is a little different here- originally it was totally decentralized, there was no central registry, just URLs. So you could depend on everything, no effect on other packages. No rules on publishing or unpublishing either, because you just get to run your repo the way you want.
At some point, Russ Cox got the Fear about this, and now https://proxy.golang.org/ is an on by default, caching proxy in the middle. You can still delete your packages whenever you want to though.
NPM as a soulless entity is easy to bash, while the creator of the package is a popular tech influencer so naturally has the support of the masses. If you’re going to complain about NPM, describe how you would solve it in their shoes.
Switch to a "softer" interpretation of "*" that rather than blocking unpublishing of every version, allows unpublishing as long as one version is still available.
Most articles say the page includes a Skyrim meme, but no one says what the meme is and I can't find anything relating to Skyrim on everything.npm.lol. This is very confusing to me.
He didn't kill someone's puppy, he just published some interesting data that others' code struggled to cope with. He was irresponsible perhaps, but I don't think he foresaw doing any real damage, and I don't think he needs to be especially sorry for it.
Perhaps NPM should apologize for shifting blame and failing to address the root cause.
The wildcard "any version of dependency" preventing unpublish is clearly flawed. The "everything" package folks had no malicious intentions, and nobody would benefit from a long-winded, ashamed apology. If not for NPM's flawed unpublish policy the everything team would've unpublished to resolve the issue.
Do you think he should be ashamed? Granted I may have overlooked something, but as far as I can tell it wasn’t an intentionally malicious act, it was a bit of a curious experiment. Seems rather inline with the HN values to me.
Why does he even need to apologize? If anything, npm should apologize and thank him for revealing a huge issue in their unpublishing policies unmaliciously.
Yeah. I’m honestly not sure where any of this “package chaos” actually exists. I mean… there’s incompetence everywhere in every language, so yes there. But I’ve yet to run into a friend of a friend who has a horror story about these dependencies.
superasn|2 years ago
So not asking rhetorically, if we had all the insight and knowledge we have now, how would you make it different?
cxr|2 years ago
If you want answers, state clearly what _specific_ problem you're trying to solve. Whatever the solution to it might be, vague and fuzzy questions—while magnets for chatter since they can stand in for whatever someone wants to read out of them—are not the way to get there.
(You could say that this is needlessly tedious because everyone already knows what we're talking about, but that this isn't true is exactly my position. It's certain that something like half the people reading, thinking, and writing are have in mind one thing, while the other half are thinking of another—and the third half are thinking about something different from either of those. We're also programmers, so dealing with tedium and the constraints of having to be explicit should be second nature.)
hannob|2 years ago
When you wonder whether to add a dependency, you should ask yourself: What are the upsides and downsides of adding this dependency. One downside is always that by adding a dependency, you add a potential security problem, a potential point of breakage, and more complexity.
There are situations where these are well justified. If your dependency is stable, from a trustworthy source, and if it is a functionality that you cannot quickly implement yourself. But if you include a dependency that is effectively one line of code, the question answers itself: The costs of adding a dependency is completely unreasonable. It your list of dependencies grows into the 100s, you're doing something wrong.
charlie0|2 years ago
ploxiln|2 years ago
If analyzing the dependencies for showing in the NPM web UI, while analyzing, as you exceed 40 direct or transitive dependencies, abort and highlight this package in red, for having excessive dependencies.
If installing locally, you get what you get, don't install random or crazy packages, stick to well known high-quality minimal-dependencies packages. nodejs does include file reading and writing, http server, http client, json ... that will take you pretty far. Master the basics before getting too fancy. And remember, you don't need some company's client package just to make some http requests to their API.
jonorsi|2 years ago
Basically, you would need to start accepting that you are responsible for any dependencies you choose to include. Any upstream changes you would need to evaluate and bring in or patch yourself.
Definitely an impossible task given how broad and deep modern package dependencies are, but at least you’d start feeling the insanity of having all if them in the first place :P.
Bjartr|2 years ago
TobyTheDog123|2 years ago
While it's ridiculous to expect that people will audit every single dependency and sub-dependency, it's not ridiculous to expect tooling to do the same.
Packages should be given an overall quality rating (and honestly it might be great for an ecosystem as large, diverse, and welcoming-to-beginners as JS/TS), part of the score comes from the number of different dependencies/sub-dependencies -- a social package score if you will. If a package causes the dependency graph to explode, give a warning before installing it.
Then, if you're NPM, you don't need all of these convoluted and exploitable policies around un-publishing.
baz00|2 years ago
Our devops guys scream from the seething pain whenever the have to debug some pile of shit that decides it won't build unless all the runes are aligned precisely and all the RAM in the universe is available on the build runners. And pushing this to the developers results in importing more packages thus adding to the burning tyre fire.
And after several hours of builds and 9000 layers of packages you wake up one morning and in that 50 meg chunk of javascript that is excreted from the process, someone managed to inject a "Slava Ukraini!" banner into your web app.
Stevvo|2 years ago
rfoo|2 years ago
Make the cost of reusing software non-zero again.
It doesn't have to be as painful as C++ without package managers, but should make every developer spend about 5~10 minutes labor work for adding each direct dependency, or one minute for each new dependency in the dependency closure.
floren|2 years ago
(If you don't want Google to see what packages you're fetching, you can also turn this off with an environment variable.)
ShadowBanThis01|2 years ago
bandrami|2 years ago
notnmeyer|2 years ago
the community shouldn’t need to write a bunch of tiny utility packages to do common things.
in other words, make it easier to avoid the deeply nested dependency mess that js encourages.
throwawayqqq11|2 years ago
cperciva|2 years ago
Have smarter users. If your package breaks because it depends on trivial code which got deleted, you shouldn't have depended on that in the first place.
Preventing people from deleting their code -- always, or even just sometimes -- was never the right solution.
aaomidi|2 years ago
quickthrower2|2 years ago
Once the referencing packages are updated are deleted or modified the shadow versions can be dropped.
o11c|2 years ago
[deleted]
jasonjmcghee|2 years ago
> It was removed, but then reemerged under a different scope with over 33,000 sub-packages. It's like playing whack-a-mole with npm packages!
> This whole saga is more than just a digital prank. It highlights the ongoing challenges in package management within the npm ecosystem. For developers, it's a reminder of the cascading effects of dependencies and the importance of mindful package creation, maintenance, and consumption.
> As we navigate the open source world, incidents like the everything package remind us of the delicate balance between freedom and responsibility in open-source software.
theshrike79|2 years ago
Source: have done a bunch of AI-assisted writing to develop my own skills and the tics and specific turns of phrases really pop out to me.
[0] https://en.wikipedia.org/wiki/Tic
troupo|2 years ago
"accidentally broke NPM and all I got was this sweet permanent banner all over my Github (thats impossible to remove since they probably had to code it up last minute before removing the org/repo)"
bsimpson|2 years ago
When I was consulting for an R&D lab at eBay, we open sourced a bunch of our work in a GitHub org. It was sanctioned by eBay's OSPO; they even linked to it from their main open source directory.
7 years later, long after the team disbanded, someone in eBay's current legal team decided that the (now archival) org violated eBay's trademarks. For the last year+, every time I've opened GitHub, I've been met with the same undismissable banner.
Since the only choice they give you is to contact support, I did. Unfortunately, their support team is not responsive, and has a completely separate notifications system. It took an inordinately long time for them to respond. (I have poor reception here so I can't check, but I think it was months.) Since I'm not in the habit of checking GitHub Support for new messages, when they eventually replied, I missed it. I had to start a whole new ticket. That too was months ago, and I still haven't heard back.
So because I did some work for a skunkworks eBay team in 2015, the top 150px of my GitHub are unusable, and there's apparently nothing I can do about it until some call center decides to write me back.
philipwhiuk|2 years ago
ruune|2 years ago
feross|2 years ago
You're right that a package that depends on literally everything would absolutely have a score of 0 in our system.
navtoj|2 years ago
https://www.youtube.com/watch?v=IzqtWTMFv9Y&t=465
charcircuit|2 years ago
dang|2 years ago
'everything' blocks devs from removing their own NPM packages - https://news.ycombinator.com/item?id=38873944 - Jan 2024 (102 comments)
SquidJack|2 years ago
yreg|2 years ago
Has no one thought of that? It seems like it should have been obvious that such an absolute rule could be easily abused to troll the system at scale.
Not sure if it's a problem though, perhaps all unpublishing requests should be reviewed by someone at the registry (and granted only when it makes sense).
yoavm|2 years ago
rubyissimo|2 years ago
Is npm specifically vulnerable to this kind of thing? Or is it just a cultural elelemnt of npm that there are more micro-packages?
klooney|2 years ago
At some point, Russ Cox got the Fear about this, and now https://proxy.golang.org/ is an on by default, caching proxy in the middle. You can still delete your packages whenever you want to though.
ramesh31|2 years ago
frabjoused|2 years ago
spion|2 years ago
happens|2 years ago
norskeld|2 years ago
[deleted]
francisduvivier|2 years ago
"Just install the everything package, then you will be sure to have the right package"
leros|2 years ago
nubinetwork|2 years ago
[deleted]
pvg|2 years ago
Affric|2 years ago
> First, just want to apologize about any difficulties this package has caused.
No rationale. No shame. Just the word “apologize” in a sentence.
Who downloaded it though? Surely as a dev if you download such a package it’s on you?
Retr0id|2 years ago
nosefurhairdo|2 years ago
The wildcard "any version of dependency" preventing unpublish is clearly flawed. The "everything" package folks had no malicious intentions, and nobody would benefit from a long-winded, ashamed apology. If not for NPM's flawed unpublish policy the everything team would've unpublished to resolve the issue.
urbandw311er|2 years ago
jonorsi|2 years ago
Waterluvian|2 years ago