top | item 28892933

The JavaScript ecosystem is a hot mess and so is software development

102 points| opsec | 4 years ago |nadh.in

90 comments

order
[+] throwdecro|4 years ago|reply
This is a rant I can appreciate. Two subjective takes on the same subject:

1) As JavaScript became ascendant, it seems like developers lost their fear of dependencies, for some reason I don't understand. Dependencies came to be seen as "time saved" rather than "something out of your control that can hurt you."

2) When dependencies don't fit together, things fail in a way that makes it look as if the code is simply wrong. Yet everyone working in the environment ends up conditioned to run to google with the combination of (circumstances, error string) to find someone who has figured out what the actual dependency issue is. It's automatic behavior. At some point it would make sense to include automated googling in the CI pipeline just to add some necessary information to the errors.

[+] Hermitian909|4 years ago|reply
I agree strongly and think once you regain your fear of dependencies the JS ecosystem actually isn't that bad. The kinds of packages that save me months or years of work are generally of reasonable to great quality, it's packages that (theoretically) save a couple days or weeks that tend to be the real stinkers.
[+] noahtallen|4 years ago|reply
I’ve definitely spent a ton of time on dependency work. I think the web app ecosystem is unique compared to its counterparts because the platform is really not designed for applications; it’s designed to render documents. You get so much more out of the box when developing for proper application platforms like iOS. For example, dealing with dates and internationalization. iOS gives you enough tools to not really need an extra library. Not so with the web.

Plus, the std for these platforms is much larger too, because you get access to the huge API surface the platform provides. Apple has spent a significant amount of resources making the platform decent to develop for. They even include reactive UI patterns out of the box now; something you’ll need a library for on web.

I think that’s the real shame. The web platform is very flexible and ubiquitous, so most people try to target it. But in doing so, you’ll need to account for web standards differences across browsers (using a tool like browserslist), the cost of downloading the source code when navigating into a page (with webpack and minification to make bundle sizes more reasonable or split up), and the cost of not having strong default patterns for stateful, interactive UI.

Every other platform will include nearly all of the tools you need within the IDE and on the platform. Web just doesn’t have that, unfortunately, and it has further problems that really don’t exist on other platforms. (For example, other platforms don’t rely on downloading the app source code from a remote server when launching it every time.) It relies on the open source community to provide solutions to so many problems, resulting in the complex dependency trees we see in large apps.

On the one hand, the OSS spirit is admirable. There are competing projects for nearly everything, similar to Linux programs. (Just like there are many Linux dependency managers, there are multiple JS dependency managers.) This allows projects to “get better” by competing with each other, but at the cost of breaking changes and maintenance over time. (For example, there are many projects aiming to provide the fastest JS bundler.) But there is something to be said for having an opinionated platform with more features like iOS or UWP for windows.

All that to say, I don’t think this is a JS shortcoming, but a shortcoming of the web platform in general. And some of those shortcomings are inherent to the benefits of the platform.

[+] sam0x17|4 years ago|reply
A side rant: for some reason in recent years it has been OK to make breaking changes. I remember a time when these were considered a nuclear option -- now popular libraries will make breaking changes for minor aesthetic reasons, resulting in billions of wasted developer hours changing foobar to foo_bar.
[+] pca006132|4 years ago|reply
> At some point it would make sense to include automated googling in the CI pipeline just to add some necessary information to the errors.

Sounds like a decent feature to include in a compiler!

[+] taurath|4 years ago|reply
I installed ESLint into a new project and my package.lock file expanded to 3000 lines. I understand what problem it was trying to solve but the sheer inelegance of it sort of gives everything a jank feel, like you have no idea what’s running.

Or perhaps it’s more that it’s exposing a bunch of things that are hidden or tucked in standard libraries in other languages.

[+] austincheney|4 years ago|reply
> for some reason I don't understand

Fear of writing original code. There is less perceived risk (actual risk is slightly increased) if you can defer blame.

[+] nayuki|4 years ago|reply
I'm embarrassed to admit that for web development, I'm afraid to try frameworks like React/Vue/Angular as well as package managers like NPM/Yarn. So I settled on writing raw (X)HTML code and using only vanilla JavaScript APIs provided by the browser (ECMAScript standard library, DOM, multimedia, etc.).

Because of this, I can't bang out complicated software quickly and easily like the stereotypical front-end developer. In exchange, the techniques that I learn essentially don't go out of date and don't randomly break. I settled for only applying what I understand.

I made an exception for TypeScript; I acknowledge the compiler is enormously complicated and requires Node.js to run. But static type checking has saved me from my own silly mistakes very often. And if armageddon ever comes, I can always strip out the annotations from existing TypeScript code and continue with plain JavaScript.

[+] recursivedoubts|4 years ago|reply
I work on a dependency-free, build-less library called htmx:

https://htmx.org

It's more an extension of HTML than a JS library. It might be a good way to gently introduce more dynamic behavior without blowing out the complexity of your projects.

[+] pixelgeek|4 years ago|reply
I am with you. Don't be embarrassed. I spent a lot of time looking at Vue, React and the others and frankly I don't see how the time and effort are worth it unless you work in a large firm that has a lot of developers working on a large project.

I use npm and node because it allows me to write local tools that I write to help me manage data and convert it. I have a lot of other tools that I have access to but I can thankfully access it via CodeKit on my Mac so I don't have to waste time installing and learning them all.

[+] dimgl|4 years ago|reply
> I made an exception for TypeScript

I was nodding my head until TypeScript. I still don't like it even after years of people telling me it's the future. Maybe this is what it feels like to be older.

[+] spurgu|4 years ago|reply
Sounds like a solid/healthy strategy to me.

With ignoring package managers, do you mean that you instead of running "npm/yarn install/add <package>" you inspect said package (which does what you need) and cherry-pick the functionality into your own codebase (or just outright copy it if its functionality is elementary to you), or that you try to write/reimplement it yourself?

[+] 1ark|4 years ago|reply
The browser environment enables you to do that, so why not. No dependencies. But do you build everything in house then (time unlimited), or the apps are not that complex? It is the same mindset as writing assembly or small Win32 in C, which I both admire and respect btw.
[+] taurath|4 years ago|reply
To get a good setup in JS there’s a ton of things that are nativeish in other languages (at least via IDE) that are wholly exposed via the package management stuff. The upside of the configurability is you can mix and match, and build something that works great for your needs and your project. Different framework, different test system, different linter, different SQL wrapper, etc. The downside is that it’s overwhelming, even when experienced with it.

I would suggest starting with a bundle created by create-react-app - you will get most of the cool tooling out of the box, and discover the usefulness of some of the libraries.

Having a live loading web app is a really neat and productive way to develop - but getting there can be daunting.

[+] fny|4 years ago|reply
Despite the bandwagoning here, I'm going to say it's the authors fault.

They updated dependencies blindly upping the version of packages a leading digit. Of course things will break! That's what the leading digit is for!

Yes, there are problems with too many dependencies, but upgrading packages just because you feel like it is a recipe for disaster even with very mature dependencies with good governance.

[+] mmcnl|4 years ago|reply
I agree. I also think the author blindly followed up on some automatic GitHub bot suggestions only to find out you can't just upgrade everything.

My experience is actually the opposite: it's never been easier to develop at scale in frontend. People forget how hard frontend is. You need to ship everything and it needs to work everywhere. You have zero control over the code execution environment. The fact that build tools like Webpack and frameworks like Vue/React exist made it a breeze to quickly develop high-quality web applications, and that is actually a triumph of the JavaScript ecosystem.

Ofcourse it's difficult to understand the entire build chain, but often it's not necessary to understand every detail. If you use vue-cli/create-react-app you get a perfect bootstrapped environment that works out of the box. If you really need to upgrade, I would suggest creating a new empty project and migrate all the code to the new project, instead of changing dependency version numbers in package.json.

My personal experience is that I've run into quite some issues, but not necessarily more than when doing anything backend related. I've spent weeks trying to understand the Python module system. And worse, Python doesn't even have declarative dependency management built-in. Also it's very difficult to find high quality packages because there's hardly anything like npmjs.com for the Python ecosystem.

That being said, I love both JS and Python, but I don't think of the quirks of both languages and their ecosystems as a "mess", but rather an understandable reality we have to deal with it that is definitely worth the trade-off. Innovating and backwards compatibility are often orthogonal direction so it's definitely not easy facilitating both.

[+] koonsolo|4 years ago|reply
Exactly. The real "problem" is how easy it is to install and upgrade packages.

They took the painful part out of library management, and now they expect breaking changes should magically be resolved.

The funny part is that there is no comparison to other development ecosystems. "Oh yeah, in C++ we don't have this problem, since everything is even more painful"

[+] pmoleri|4 years ago|reply
So, `yarn upgrade` broke something, probably because some package down the road didn't respect SemVer. I guess failing to respect SemVer can happen in any ecosystem, now in other ecosystem you could have sealed dependencies that an upgrade can't break, but then it makes the whole upgrade action moot. In summary, you either, not upgrade, go deep in your lock-file and fix it manually, our you update your main depenency like you did. If your UI library doesn't support the new version of the framework, you either change the library or you bail out on upgrade for the time being.

It seems pretty reasonable process to me, I feel your pain on the wasted time and I wish github issues have something like "current answer" so we don't need to go through pages of comments to find out how to fix or workaround something. In any case, lock files are there to help and make everything more static like in other package managers that don't support the same kind of dynamism. Upgrade is just like wiping your lock file.

[+] thdc|4 years ago|reply
Yeah, while I think the blame lies on both Vue and the author, I feel like the author is shifting too much blame off of himself.

Vue messed up because eslint wasn't pinned I assume (looks to be pinned now at ^6.7.2, so it shouldn't happen anymore) so upgrading gave you an incompatible eslint version.

Author messed up by blindly upgrading all dependencies, jumping major versions, and expecting everything to work fine without putting in work to double check. Yarn even has a kind of `interactive-upgrade` command that shows versions/links to changelogs iirc. There is also documentation for upgrading project dependencies at [this link](https://cli.vuejs.org/guide/installation.html#project-depend...) that I came across while playing around with vue cli.

I took the liberty of generating a Vue 2 project and seeing what dependencies are in the package.json

    "dependencies": {
      "core-js": "^3.6.5",
      "vue": "^2.6.11"
    },
    "devDependencies": {
      "@vue/cli-plugin-babel": "~4.5.0",
      "@vue/cli-plugin-eslint": "~4.5.0",
      "@vue/cli-service": "~4.5.0",
      "babel-eslint": "^10.1.0",
      "eslint": "^6.7.2",
      "eslint-plugin-vue": "^6.2.2",
      "vue-template-compiler": "^2.6.11"
    },
I don't think it's too bad if you've ever seen create-react-app unloaded or whatever, although I've never used Vue so I could be missing the equivalent unload step that generates a million more dependencies.
[+] Gigachad|4 years ago|reply
Upgrading dependencies is never a safe operation regardless of what semver tells you. The version number is a suggestion and not a promise. I have had minor patches break things because it turns out that my app is depending on a behaviour which is a bug. But it’s no big deal, upgrading is something you plan out and spend the time on. Upgrade them all at once and spend the time testing everything and reading change logs.
[+] shams93|4 years ago|reply
Yeah for my new startup I decided to avoid javascript except where its absolutely necessary and then use css plus lots of server side python including template generation and system and database management. I was using spa+soa with node and angular, react and vue for years but decided to use django especially because it has a great security track record and changes very slowly and doesn't require tons of extra libraries outside of django.
[+] ch4s3|4 years ago|reply
I'm starting to do this with Elixir, even though I used to have a very favorable opinion of React on the front end. When you have a small, focused ecosystem when dependencies often have few or no upstream dependencies, things just work better. The story around testing in my preferred stack is miles better than anything in JS land, and runs in parallel. One of the apps I work on has thousands of tests and the test run is under 4 minutes for all of them. They almost never break when dependencies are upgraded. It's really nice.
[+] framecowbird|4 years ago|reply
This is a great strategy. Kudos to Django for simplicity and stability!
[+] musingsole|4 years ago|reply
JustPy has been my dream come true for python based web-development.
[+] mistersys|4 years ago|reply
The JS Ecosystem + The AWS / Serverless ecosystem is a love story of the likes of Harley Quinn and the Joker.

This week I picked up a gig working on a website for a non-profit organization, the needed some basic changes to a few forms. The reason they had to hire me, a web app developer, the site was built by some agency with Next.js / Jam Stack technologies, Contenful, AWS Lambda, AWS Amplify etc.

This simple website, is one of the most complicated monstrosities I've ever worked with. Every page downloads multiple megabytes of JSON from the Contentful API just to render dead simple pages with text and images. Every time there's a change in contenful, a new build in AWS Amplify is triggered to generate the site to static content... only the React code is so complicated with so many round-trip, nested server requests (n+1+1+1 etc) that the builds fail constantly because the API server (on lambda of course) which serves the Contentful content basically gets DOSed. I had to write logic to retry all app API requests just to get the builds to consistently work so I could work on the website. The code that transforms these megabytes of JSON into webpages must have been written by a mad man.

Also screw AWS for not building proper tools to debug AWS Lambda. Not having to manage servers sounds awesome, so everyone has started using lambda, but the developer experience of having to wait 10 - 15 minutes just to add a console.log line is like pulling out your own finger nails and then scraping them on a chalkboard. Then AWS built the most noisy, difficult to use and hard to navigate log management solution on the planet (CloudWatch), and started sipping their beers.

You can avoid these technologies all you want, but one day you're going to have to work with a project that choose to use them, and you can't control that. I believe library authors and service providers have a responsibility to think about developer experience that's being ignored. Or perhaps, since developers don't always have a choice in the technologies they use, building easy to use developer tools isn't worth it.

[+] pixelgeek|4 years ago|reply
It isn't just JS apps that are like this though. I had to redo a site for a client a few years ago that was done in some horrific php CMS that had a whole site management architecture applied overtop of it. It was a nightmare to even change a page.
[+] hn_throwaway_99|4 years ago|reply
I can definitely appreciate rants like this, and I think anyone who has been in JS development for a while has hit some sort of dependency incompatibility headache like this. That said, I'm also always left a bit annoyed when the author isn't really introspective to think about what benefits came with the environment he is bitching about.

As much as everyone loves to complain about "dependency hell" in the JS/Node ecosystem, the fact that there are tons of high quality (or, at least discernible quality) components, largely for free, is something that really didn't exist before say the mid '00s. I'd also highlight that "component reuse" was literally the holy grail of software engineering for decades prior.

The simple fact is that to get the benefit of a very robust dependency ecosystem requires more work managing those dependency versions. Running the equivalent of `npm update` on a regular schedule is pretty much required lest your dependencies get too far behind, and then when you are forced to upgrade due to a security issue you find you have tons of backwards incompatible changes.

I'm not saying things like NPM can't be greatly improved, but I think it's important to acknowledge the benefits that the JS ecosystem provides.

[+] eatonphil|4 years ago|reply
I stick to pretty minimal dependencies (and more recently, simple build systems like esbuild or parcel) and I don't have a terrible time maintaining things over the years. But I can sympathize (to a point) with folks who haven't learned this lesson.

The more packages you bring in the harder it will be for you to upgrade (anything).

You have to learn over time what you must have implemented in a third party library and what would be better for you to maintain yourself.

I do wish there were a public grading system for JavaScript libraries that failed them if they bring in X dependencies or have low tests coverage. Any automated way that publicly guided/shamed library developers toward fewer dependencies would be welcome.

[+] jonathan-adly|4 years ago|reply
I agree and have quit Javascript completely. The ecosystem just feeds on layers and layers of abstractions. It seems the mantra is complex is better than simple in JS land.

Been developing with an HTMX/Django stack[0], and it has been a wonderful experience so far!

https://htmx-django.com

[+] sam0x17|4 years ago|reply
What's worse: the JavaScript community has normalized making frequent breaking changes, something that used to be forbidden and considered a no-no, and this cultural practice is bleeding over into other languages, unfortunately.
[+] mattgreenrocks|4 years ago|reply
Life is too short to put up with this nonsense. My take is the JS ecosystem is dominated by devs who like programming more than they like solving problems via programming. And that’s fine, but the culture doesn’t reign them in, and everyone takes a maximalist approach to things, believing that just a bit more JS would make things a little less crazy.

Fundamentally, the web is not a great application platform. Decent, yes. This is the fact that cannot be papered over by more software.

I left webdev back in 2012 after doing Rails. Best decision I have ever made for my career.

[+] amir-h|4 years ago|reply
I like how the giant monorepos (like Google's) serve as a forcing function against these problems.

When all of your dependencies are developed on the same repo and are part of the same CI they just won't break you (at least not your tests). And when you'll come back to the project 6 months later you'll find it building and passing tests against the latest versions of all dependencies.

This forces you to have a decent test coverage (otherwise they'll break you).

It makes library authors feel the pain of their downstream dependents and carefully consider breaking changes (otherwise they will need to spend time fixing downstream).

It makes people hesitant to bring in third party libraries that aren't very stable compatibility-wise (as the "internal owner" of the third party becomes responsible for updating it and unbreaking the world).

[+] darepublic|4 years ago|reply
plot twist: javascript frameworks are an invention of a cabal of backend engineers attempting to sabotage the client side of things
[+] duncan-donuts|4 years ago|reply
“Just one dependency”. The author makes it sound like this is trivial because they have a single dependency therefore it should Just Work™. Author should have included how many node modules were installed for the one dependency.

The closing statements are just mind blowing. To say vue/js don’t bother you but continue to shit on everything is a tired argument. Even bringing up PHP! What are you actually doing to make it better or are you just piling on to the same shit we’ve all been saying for at least 5 years?

Be the change you want to see, accept the bull shit, or just don’t use it. No one forced you to write a vue app.

[+] minusf|4 years ago|reply
while this might be a valid criticism of the blog post, i think it kind of sidesteps the issue that what it describes is a common experience while working on javascript projects. for me, pinning days are dreadful. sure, hunting in changelogs to fix issues can happen in my django projects as well, but the frequency is orders of magnitudes lower, and more likely well documented. ymmv
[+] pixelgeek|4 years ago|reply
I have recently come back into web development after a long hiatus and this article really touches on a lot of issues I have had with JS development and trying to learn new tools and libraries. I really feel what that author is saying.

All of these cli tools that people use make me think of that boiling frog metaphor.

[+] valbaca|4 years ago|reply
> Do developers have to take the overhead of worrying about the numerous dependencies in the toolchain as much as their own business logic?

I know this was meant to be rhetorical...but the answer is YES! ABSOLUTELY!

[+] stevehind|4 years ago|reply
Out of curiosity what is the best counter argument to the general thesis, eg either what positives come entangled in the hot mess or reasons it’s not a hot mess?
[+] tyingq|4 years ago|reply
High tool churn creates an ecosystem where you get paid well for knowing the new hotness.
[+] runarberg|4 years ago|reply
The other side of the coin is that we are getting better tools. Vue 3 with the composition API is a lot easier to work with, reason about and maintain then the legacy API.

However the real counter argument is that vanilla javascript apps are also easier to write these days. If you have only a small interface to something that you have to maintain but only infrequently, perhaps a vanilla app is better suited. Vue apps are better suited if you want/need to a) get things running quickly (e.g. a prototype) and/or b) have a team of developers which need to maintain a large codebase together.

[+] loic-sharma|4 years ago|reply
A developer today can create better apps with less experience today than they could 20 years ago.
[+] Phaedor|4 years ago|reply
Its a result of a lot of trial and error and finding better way of doing things. I dont think anyone likes it, but its byproduct of moving fast and of requirements changing.

Personally I think the web-frontend ecosystem is getting closer to stabilizing. For example React has not been changing much recently in its API and most of its development happens under the hood. A counterexample right now is bundlers. Webpack was the king for a long time, but it has gained more and more features and the code bases it is working on are bigger so its getting slow and complex. Competitors like esbuild and vite that are simpler and faster are popping up. However this is not the fault of webpack but simply that the requirements has changed so much and webpack always had to follow the requirements. Now that the requirements are starting to stabilize new alternatives can be built that can skip all the cruft.

I find it interesting how people in this thread are praising python frameworks for being so stable, I'm in my mid thirties and I clearly remember a time when they weren't.

[+] musingsole|4 years ago|reply
That while it might be difficult to articulate, the churn of technologies does represent a meaningful search and forward march of progress. The churn is the result of ever more rapid progress on an increasingly abstract knowledge system. However, it is hard to recognize positive structural change as just another worker bee in the hive.
[+] jitl|4 years ago|reply
Don’t install a UI toolkit dependency because you heard it’s the new hotness on Twitter?
[+] koonsolo|4 years ago|reply
Either you move slow and keep things stable, or you move fast and break things.

Pick one.

[+] pixelgeek|4 years ago|reply
well you can always write your own framework. I'ver had to do that a few times and the main benefit is that it fits your coding style and you're the only one that can break it.