One of the things that I've found to be important for (and generally under-documented by) package managers is resolution strategy.
pip, for instance, eagerly resolves the highest version of a dependency first found in the tree and doesn't backtrack AFAIK [1]. Cargo sounds as though it behaves similarly, but will actually backtrack [2]. I've also seen package managers like 0install that will use SAT solvers to solve dependency constraints [3].
I've found lack of backtracking on conflicts to be frustrating, but I'm curious how the other two (and other conflict resolving strategies) compare in practice.
pip's lack of backtracking is doubly irritating, because it installs packages as it goes. It would be great if it would fetch all of the required manifests and make sure that the install was going to succeed before it starts spewing stuff everywhere, but instead it will happily back itself into a corner where A->B->C(v1) and A->D->C(v2) conflict and so it leaves the system with A, B and C(v1) installed and prints a nice red stack trace.
and graydon wrote a POC of using the Z3 solver to find a solution where semver constraints were satisfied, optimizing for the latest versions within those constraints:
Resolution strategies are really interesting in Nix; specifically, because it doesn't have any! All dependencies, etc. are passed in explicitly (either as function arguments, or by importing files).
You are absolutely right about solver strategies. Most of the existing package managers in Go have no solver. It's all manual and typically the tip of master or latest commit.
Glide has a dumb solver in that it goes with the first best version it finds. It will tell you if there is a later conflict that you can manually resolve.
There is a package, discussed here called gps, that is being prepped to merge into Glide. This solver has a back-off strategy. It's loosely based on the solver in dart. The idea is to find the latest newest version that works for the complete tree.
We've been incrementally working to get a good solver and are almost there.
At least three things wrong with respect to Bundler/RubyGems:
1. RubyGems is the package manager. Bundler is a meta-layer on top of that which does full dependency resolution to find the right version before installing. Bundler builds on the capabilities provided by RubyGems (and will be integrated into RubyGems in the future).
3. RubyGems and Bundler can use alternative sources (my work applications use both public and private sources simultaneously).
One thing I don’t see that would love to see tracked and solved is authenticity verification. RubyGems has support for signed gems but it’s not widely used and hasn’t really been validated as Correct.
They should probably also look at CocoaPods and Carthage (iOS and macOS build dependencies) and whatever Swift provides.
2. Write apps that need to work in disconnected environments -- ie with no internet connectivity.
I work on the Cloud Foundry buildpacks team for Pivotal. Being able to stage and launch apps in a disconnected environment is A Big Deal for a lot of companies.
Call me old fashioned, but I find it annoying that every new language nowadays seems to want to pull in its own package manager, redundantly parallel to the perfectly good one my operating system provides. I am already perfectly fine with installing my software through apt-get. Why oh why do I need pip, and gem, and npm, and CocoaPods, and cargo, and NuGet, and on and on and on...
Using a new language is no longer a matter of cracking open a book, installing a compiler, and firing up vim. You've got to change your whole lifestyle now.
Because apt-get doesn’t exist on the system I use, and most system packagers can’t/won’t keep up with the rate of change of development, anyway.
CPAN, RubyGems, etc. all exist because system packaging systems are (a) specific to a particular distribution, (b) generally many versions behind the development versions, (c) really hard to get into unless you are already a system packager for a particular distribution, and (d) effectively don’t exist for two of the largest platforms where you will find users.
I’d suggest making some sort of meta-system packaging system that works for all the languages, but xkcd 927 (and I wouldn’t use it myself, anyway, because it would probably not install Ruby packages correctly and would solve things worse than NPM already does).
Oh you know, just some silly little use cases like...
* building on not-linux -- you know, where almost all personal computers exist. Gee, wouldn't it be nice if most computers could actually compile your code?
* Having only one build config, only having to maintain one publication (ugh need to maintain deb, rpm, nix, chocolatey, brew, whatever the hell arch and suse use...)
* reproducible builds -- distros are well-known to randomly modify libs, making "foo x.y.z" in yum and apt basically random. It's not even clear to me I'll get the same thing from apt on different days!
* Iterating and collaborating on fledgling projects which rightly wouldn't be accepted into a distro -- or just iterating on a timescale smaller than "months"
* local deps -- using system-global deps (as preferred by most linux package managers) is known to be a huge nightmare for any language package manager that's tried. Hence pyvenv. Hence why so many have thrown up their hands in disgust and just started dockering all the things.
(inb4 someone comes to explain that Nix or whatever probably magically solves all these problems)
Although, the pony is still valid that every programming language has decided to implement its own pkg mgr.
Like all things in languages, the world needs to be reimplemented. Which is too bad, but I also never see a great alternative. Eventually people want everything they work on to be in the same language, which of course has the advantage that anyone in that language can hack on it, but the disadvantage that we always start anew every time a language gets created.
Where's the redundancy? When you pull down a project from github, grabbing all the specific versions of libraries it needs in a way which doesn't interfere with the versions of libraries used by other projects, is a different use case than the one system package managers perform, surely?
Yep. I always feel extremely uneasy when I encounter a codebase that asks me to install its language's package manager (npm I'm looking at you) as root. I just go 'nope' the only thing that touches my real system is portage. Even using npm or pip as a non-root user worries me because there is an entire extra level of validation missing and it signals to me that developers are not doing due diligence and working with distro maintainers. Yes I know it is more work, but it is an incredibly important step for quality control. The kids these days seem to think that containerization will solve everything, but these problems cannot be avoided.
Prepare for the hate-train from developers. But I agree with you though. It's frustrating because each PM wants to install it's own version, which ends up conflicting with another PM's version, and who knows where they end up, or in what state. Apt-get rarely fails for me, but pip, gem, npm fail all the time. And who knows what version of the package they just installed, or where they got it from.
I wish devs would just tell me the libraries I need, and I'll get the packages myself.
Unless your on the bleeding edge of development, you don't need the newest versions of a lib. And you probably shouldn't be developing on the bleeding edge anyway. There's a reason apt-get is so stable, and is used everywhere.
The caveat to all that is: I hate making packages for apt-get. It's a terrible process, and I completely understand why so many devs want to avoid it.
Global installs are the root of a lot of headaches when building software in the same way that global mutable state is the root of a lot of headaches when developing it. Nix may be the one system package manager that is the exception to this rule (I don't have experience with it, so I can't vouch towards that, but I hear great things). However apt-get is incredibly bad at getting you the dependencies you need for a reproducible build.
I've seen this argument come up a few times now, and I have a hard time reconciling this rosy view of system package managers at the center of a development process with my own experiences. That process was never as easy as "cracking open a book, installing a compiler, and firing up vim" and usually entailed a fair amount of trial and error, environment variable fiddling, and sadness from having to chase down a derelict PPA (or the equivalent) hosting the actual dependency version I need. Then more sadness as the number of projects I work on increases and my dependency versions bifurcate.
I know Dart isn't super widely used, but it's package manager, pub, is probably a good reference for this. We put a ton of work into it, and Dart users regularly single it out as one of the compelling parts of the Dart platform.
It hews pretty closely to Bundler's model (versions, version ranges, lockfiles, shared dependencies), though. So I don't know if it gives you much insight beyond "Yes, another language team things Bundler is on the right track."
Personally, I have a hypothesis that states that all programming languages either have a package manager that works like Bundler or will end up getting one.
Perl's CPAN has tooling for diffing versions through the use of MetaCPAN (a top-notch site which every language should try to emulate). For example, here is a diff of the URI distribution:
I don't get why "pip" is checked in "Has separate manifest and lock files". Actually it doesn't have that feature (which Bundler for Ruby has, for instance). This very feature comes with a third-party package called "pip-tools", or possible alternatives, but raw pip doesn't have this ability directly afaik.
Fwiw pip isn't even able to enforce versions correctly (packages are installed as the file is read, and can conflict with previously expressed constraints). Or report installed versions correctly (it's possible that packages are half-installed or installed but not reported as such by pip commands).
On the CF buildpacks team in NYC, lack of vendoring prevented us from completing a buildpack that would work in disconnected Cloud Foundry installations.
Since I had starting looking for an alternative to NPM I have discovered a couple of things:
* All current package managers are either language or OS specific. What if you have an application with code written in multiple languages?
* NPM didn't have any kind of integrity checks for its packages, and I assume most package managers don't either. If you download a corrupt package, for example, you won't have any idea and it will still install.
* Some package managers do better than others with regards to managing packages. I found NPM encourages dependency hell and very little management tools for dependent or installed packages.
* A lot of package managers seem to intermix packaging, distribution, and a registry. The registries tend to have limited names to pick from (like real estate) and can result in legal problems. Also if registration to the service catalog is required you cannot self-host or self-manage the distribution of your application.
> * All current package managers are either language or OS specific. What if you have an application with code written in multiple languages?
guix and nix both work cross-language and cross-distro. Still OS specific though since only linux AFAIK. Also, containers partially solve this problem.
> * NPM didn't have any kind of integrity checks for its packages, and I assume most package managers don't either. If you download a corrupt package, for example, you won't have any idea and it will still install.
Any package manager that doesn't do integrity checks is a bad package manager. The only one I know of currently that doesn't is npm, but I haven't looked deeply into every available package manager.
> * A lot of package managers seem to intermix packaging, distribution, and a registry. The registries tend to have limited names to pick from (like real estate) and can result in legal problems. Also if registration to the service catalog is required you cannot self-host or self-manage the distribution of your application.
What package managers don't let you self host? I'm truthfully not aware of any. Even NPM does according to a quick google.
From your readme: "biddle is inspired by the incredible awesomeness of NPM". Since NPM is literally the worst package manager I have ever used, that line makes me want to stop reading and never touch biddle. I'd word it differently.
Edit: Reading biddle further. Dependency management and central hosting are some of the primary reasons to have a package manager. At least for me, that kills any interest at all. I imagine there's a niche market though?
It would be nice if the Go crowd and the Rust crowd, both of which are developing new package managers, had at least a common spec on how to describe dependencies.
This should probably use Stack for Haskell as well, which was designed to solve various common issues that arise when using cabal as a package manager.
I'm confused as to why composer (PHP) got an X for "Central package repository"... isn't Packagist a central registry? And other package managers allow packages from source.
I'm not arguing the point just trying to understand the decision.
You're not actually downloading the package from Packagist. It acts only as a metadata repository that points you to a source repository. For the others, I would assume they act as both a metadata and a source repository.
Which one allows to import multiple versions of a lib/module/package with different versions? (Not primarily but also an issue of the package manager.)
The difference is in the central repository: Elm will compare your types to the previous versions and how the versions diff, and refuse to let you upload a package that makes a backwards incompatible change without bumping the version.
At least, that's my understanding of what they meant.
knucklesandwich|9 years ago
pip, for instance, eagerly resolves the highest version of a dependency first found in the tree and doesn't backtrack AFAIK [1]. Cargo sounds as though it behaves similarly, but will actually backtrack [2]. I've also seen package managers like 0install that will use SAT solvers to solve dependency constraints [3].
I've found lack of backtracking on conflicts to be frustrating, but I'm curious how the other two (and other conflict resolving strategies) compare in practice.
[1]: https://github.com/pypa/pip/issues/988
[2]: http://doc.crates.io/cargo/core/resolver/index.html
[3]: http://0install.net/solver.html
haldean|9 years ago
surrealize|9 years ago
https://github.com/rust-lang/cargo/issues/2064
and graydon wrote a POC of using the Z3 solver to find a solution where semver constraints were satisfied, optimizing for the latest versions within those constraints:
https://github.com/graydon/z3-rs/blob/master/tests/semver_te...
chriswarbo|9 years ago
mfer|9 years ago
Glide has a dumb solver in that it goes with the first best version it finds. It will tell you if there is a later conflict that you can manually resolve.
There is a package, discussed here called gps, that is being prepped to merge into Glide. This solver has a back-off strategy. It's loosely based on the solver in dart. The idea is to find the latest newest version that works for the complete tree.
We've been incrementally working to get a good solver and are almost there.
halostatue|9 years ago
1. RubyGems is the package manager. Bundler is a meta-layer on top of that which does full dependency resolution to find the right version before installing. Bundler builds on the capabilities provided by RubyGems (and will be integrated into RubyGems in the future).
2. Bundler does support vendoring. It is widely discouraged, but I take advantage of it in my app packager for deploys at work (Cartage: https://github.com/KineticCafe/cartage, specifically cartage-bundler https://github.com/KineticCafe/cartage-bundler).
3. RubyGems and Bundler can use alternative sources (my work applications use both public and private sources simultaneously).
One thing I don’t see that would love to see tracked and solved is authenticity verification. RubyGems has support for signed gems but it’s not widely used and hasn’t really been validated as Correct.
They should probably also look at CocoaPods and Carthage (iOS and macOS build dependencies) and whatever Swift provides.
jacques_chester|9 years ago
It is also widely encouraged. Some engineers:
1. Don't trust dependencies to always be there.
2. Write apps that need to work in disconnected environments -- ie with no internet connectivity.
I work on the Cloud Foundry buildpacks team for Pivotal. Being able to stage and launch apps in a disconnected environment is A Big Deal for a lot of companies.
ryandrake|9 years ago
Using a new language is no longer a matter of cracking open a book, installing a compiler, and firing up vim. You've got to change your whole lifestyle now.
halostatue|9 years ago
CPAN, RubyGems, etc. all exist because system packaging systems are (a) specific to a particular distribution, (b) generally many versions behind the development versions, (c) really hard to get into unless you are already a system packager for a particular distribution, and (d) effectively don’t exist for two of the largest platforms where you will find users.
I’d suggest making some sort of meta-system packaging system that works for all the languages, but xkcd 927 (and I wouldn’t use it myself, anyway, because it would probably not install Ruby packages correctly and would solve things worse than NPM already does).
Gankro|9 years ago
* building on not-linux -- you know, where almost all personal computers exist. Gee, wouldn't it be nice if most computers could actually compile your code?
* Having only one build config, only having to maintain one publication (ugh need to maintain deb, rpm, nix, chocolatey, brew, whatever the hell arch and suse use...)
* reproducible builds -- distros are well-known to randomly modify libs, making "foo x.y.z" in yum and apt basically random. It's not even clear to me I'll get the same thing from apt on different days!
* Iterating and collaborating on fledgling projects which rightly wouldn't be accepted into a distro -- or just iterating on a timescale smaller than "months"
* local deps -- using system-global deps (as preferred by most linux package managers) is known to be a huge nightmare for any language package manager that's tried. Hence pyvenv. Hence why so many have thrown up their hands in disgust and just started dockering all the things.
(inb4 someone comes to explain that Nix or whatever probably magically solves all these problems)
bluejekyll|9 years ago
Like all things in languages, the world needs to be reimplemented. Which is too bad, but I also never see a great alternative. Eventually people want everything they work on to be in the same language, which of course has the advantage that anyone in that language can hack on it, but the disadvantage that we always start anew every time a language gets created.
jameshart|9 years ago
hyperion2010|9 years ago
CaptSpify|9 years ago
I wish devs would just tell me the libraries I need, and I'll get the packages myself.
Unless your on the bleeding edge of development, you don't need the newest versions of a lib. And you probably shouldn't be developing on the bleeding edge anyway. There's a reason apt-get is so stable, and is used everywhere.
The caveat to all that is: I hate making packages for apt-get. It's a terrible process, and I completely understand why so many devs want to avoid it.
knucklesandwich|9 years ago
I've seen this argument come up a few times now, and I have a hard time reconciling this rosy view of system package managers at the center of a development process with my own experiences. That process was never as easy as "cracking open a book, installing a compiler, and firing up vim" and usually entailed a fair amount of trial and error, environment variable fiddling, and sadness from having to chase down a derelict PPA (or the equivalent) hosting the actual dependency version I need. Then more sadness as the number of projects I work on increases and my dependency versions bifurcate.
jlarocco|9 years ago
Apt is no more standard than pip, gem, or any of the others.
Try maintaining packages for Debian, Red Hat, Arch, Gentoo, OSX, Windows, and a dozen others, for a while, and see how quickly you change your mind.
> You've got to change your whole lifestyle now.
Oh please.
munificent|9 years ago
https://www.dartlang.org/tools/pub
It hews pretty closely to Bundler's model (versions, version ranges, lockfiles, shared dependencies), though. So I don't know if it gives you much insight beyond "Yes, another language team things Bundler is on the right track."
Personally, I have a hypothesis that states that all programming languages either have a package manager that works like Bundler or will end up getting one.
jacques_chester|9 years ago
sivoais|9 years ago
https://metacpan.org/diff/file?target=ETHER%2FURI-1.71%2F&so...
This information is also available through an API for integration into command line tools.
mfer|9 years ago
In fact, the original creator of Glide (Go package manager) wrote about Perl and CPAN when talking about Go at http://technosophos.com/2015/09/02/dont-let-go-be-condemned-....
jbbarth|9 years ago
Fwiw pip isn't even able to enforce versions correctly (packages are installed as the file is read, and can conflict with previously expressed constraints). Or report installed versions correctly (it's possible that packages are half-installed or installed but not reported as such by pip commands).
calvins|9 years ago
steveklabnik|9 years ago
jacques_chester|9 years ago
We were sad.
anonymousguy|9 years ago
* All current package managers are either language or OS specific. What if you have an application with code written in multiple languages?
* NPM didn't have any kind of integrity checks for its packages, and I assume most package managers don't either. If you download a corrupt package, for example, you won't have any idea and it will still install.
* Some package managers do better than others with regards to managing packages. I found NPM encourages dependency hell and very little management tools for dependent or installed packages.
* A lot of package managers seem to intermix packaging, distribution, and a registry. The registries tend to have limited names to pick from (like real estate) and can result in legal problems. Also if registration to the service catalog is required you cannot self-host or self-manage the distribution of your application.
I am trying to work on a solution to these problems at https://github.com/prettydiff/biddle
mordocai|9 years ago
guix and nix both work cross-language and cross-distro. Still OS specific though since only linux AFAIK. Also, containers partially solve this problem.
> * NPM didn't have any kind of integrity checks for its packages, and I assume most package managers don't either. If you download a corrupt package, for example, you won't have any idea and it will still install.
Any package manager that doesn't do integrity checks is a bad package manager. The only one I know of currently that doesn't is npm, but I haven't looked deeply into every available package manager.
> * A lot of package managers seem to intermix packaging, distribution, and a registry. The registries tend to have limited names to pick from (like real estate) and can result in legal problems. Also if registration to the service catalog is required you cannot self-host or self-manage the distribution of your application.
What package managers don't let you self host? I'm truthfully not aware of any. Even NPM does according to a quick google.
> I am trying to work on a solution to these problems at https://github.com/prettydiff/biddle
From your readme: "biddle is inspired by the incredible awesomeness of NPM". Since NPM is literally the worst package manager I have ever used, that line makes me want to stop reading and never touch biddle. I'd word it differently.
Edit: Reading biddle further. Dependency management and central hosting are some of the primary reasons to have a package manager. At least for me, that kills any interest at all. I imagine there's a niche market though?
Animats|9 years ago
ericlathrop|9 years ago
riffraff|9 years ago
unknown|9 years ago
[deleted]
nilkn|9 years ago
moosingin3space|9 years ago
throwaway2016a|9 years ago
I'm not arguing the point just trying to understand the decision.
jadell|9 years ago
stewbrew|9 years ago
olalonde|9 years ago
w4tson|9 years ago
the_mitsuhiko|9 years ago
steveklabnik|9 years ago
At least, that's my understanding of what they meant.
jacques_chester|9 years ago
Probably because of this[0]:
> Note that Cargo does not yet support vendoring in a first-class fashion, but this is a hotly desired feature and coming soon!
[0] http://doc.crates.io/faq.html#how-can-cargo-work-offline
mfer|9 years ago
unknown|9 years ago
[deleted]