Go ecosystem is a complete mess. I understand what Go maintainers tried to do, but Go isn't like Gnome lib,or Linux core lib where dependencies are stable. And the "just vendor" stuff pushed by some Go pundits is stupid. Clutter or Gtk don't vendor Gdk or Glib. So Go right now is about everybody reinventing the wheel, because you can't write reproducible build with 3rd party packages thus create an ecosystem with Go get. The solution ? deprecate Go get in favor of a new tool that supports at least fetching tags for packages. Project based build Tools (like Gb) work for apps, they don't work for libraries. Glide and co have a big problem : everybody is using a different package manager that is incompatible with others.
The question isn't being solved yet because to solve it seriously it would require to change how the language works at first place.
> And the "just vendor" stuff pushed by some Go pundits is stupid.
I see it as an excellent way to allow me to ensure that no matter what my project continues to build and work. And dont say "just pin a version" because there have been packages that lose or update their tags and break things, or the distribution channel (npmjs, github, w/e) go down and suddenly I am up a creek for a deploy/build.
When I vendor, I have what I need to build and deploy my code with a lot fewer points of failure.
> The question isn't being solved yet because to solve it seriously it would require to change how the language works at first place.
And Go 1 compatibility promise [0] would prevent any change.
In theory, you could define a GOPATH to install custom packages. You install half a dozen or so packages, and it's not trivial to determine what all you installed. A Go update arrives. Now some of those packages fail to update. Either spend hours debugging, reporting bugs, or rm -rf $GOPATH and reinstall everything.
The language is wonderful in so many ways, but a number of seemingly silly things make it annoying.
I understand that the tools will need to be changed, but what needs to be changed in the language? Because import paths are strings interpreted by the tools and not the language, there seems to be good separation between the language and issues of source code organization, packaging, etc.
Hi folks, My name is on that list and I work on one of the Go package managers (Glide - it's like npm, crates, composer, bundler, etc... but for Go... https://glide.sh).
If you have questions or input those who are working on this problem are listening.
[This question is purely informational and doesn't directly relate to golang, but I want to ask someone who has been involved in domain-specific package management.]
Why is the protocol side of package management not a solved problem already?
At least Go reuses the git protocol to its advantage. But the general problem of package management applies to a lot of things.
"There is a package with a name and a description. That package has versions, which can be greater/lesser than others. Those versions may have deltas from one-another. They contain one or more files. Those files must have their integrity checked. etc, etc..." -> Why does every single language reimplement that?
Nevermind languages actually. Why does everyone reimplement that? Firefox/Chrome extensions, Android/ios app stores, a bazillion different game addons and app-specific stores...
We have a mostly-identical problem being solved a million different ways. pypi, npmjs, glide, they should be running the same server software. It'd be a first step towards standardization and avoid a bunch of mistakes repeated every time a new solution comes out.
Is anyone even trying to do something about this? (and before anyone says "do it yourself", a package management protocol is something I used to work on, not anymore. It keeps getting worse - it's not just a technical problem, also a political one.)
Personally I don't think we need yet another package manager with painful external dependencies, non-reproducible builds, conflicts, lack of rollbacks, etc. We have that with "go get". What we need is to actually solve all those thing. Make sure there are no fragile external dependencies, but everything is in the package manager, even gcc, binutils. You get the idea.
(I've tried Glide but it slowness and few bugs made me move to Govendor. Nothing personal, it just didn't work out that well for me.)
A few questions:
- Is the intention desincentive the use vendoring?
- Is the intention to make a single solution, and then deprecate existing tools (Glide, Govendor and like)?
- Do you have the intention to make a central registry for Go packages? Or the solution will use GitHub and like, as it does today? It would be not backward compatible, but the document don't make this clear.
What are some of the proposed solutions other than vendoring, which tends to be very difficult for large distributions such as Fedora. I was just reading a few days ago about some of the extreme challenges Fedora is having packaging the golang ecosystem.
I never understood why Go conflates where something is (its Github url) with what something is (a specific version of a specific bit of code, what Maven calls an "artifact," and what lots of other solutions call a "package," or "module," or "gem", which has a version and never changes).
This creates a lot of problems that simply don't need to be there:
1. Versioning dependencies is pretty much impossible unless you make your own copy of them, and keep them updated. Good dependency management solutions do not require you to "vendor" your dependencies. They enable reproducible builds by keeping every version of an artifact around.
2. Github could change its URL schema, and most Go projects would stop compiling.
3. The maintainer could delete the Github repo.
4. Source control, specifically Github, is tightly coupled with dependency management, and that should not be necessary.
These are not unsolved problems. Maven solved every single one of these problems years ago. I don't expect people to use Maven to build Go applications, but clearly there is an opportunity to use Maven's good ideas and build a better dependency management system for Go.
I have to say, in defense of Maven, that it is very robust: artifacts are mirrored in many places, including in your own local repo; versioning works well, including availability of older versions; and nobody can screw you by deleting package or changing its Github url. My company uses Maven for Node.js as well as Java, because we've had so many problems with NPM.
I should also point out that the expectation that the core Go team should develop a dependency management system is a bit odd. The dependency management systems for Java, Ruby, Python, Node.js, and Perl were all developed by the community, not the core team. Some of them were later absorbed by the core team, but the point still stands.
You, too, seem to misunderstand the concept of a Go import path. It's a string that identifies a package. It's not a URL. To provide for sane namespacing, people tend to use DNS names and URL paths to name their packages, just like they do with Java package names. There's no requirement that there's any correlation between the package import path and where you fetch that package's code from.
Thus, your statement:
2. Github could change its URL schema, and most Go projects would stop compiling.
is wrong. No single Go project that compiled before would now fail to do so. The common build tools use the import path to determine a path on the local filesystem where source or object code may be found. Neither a change of Github's URL schema, nor a repo deletion by some maintainer would cause your build to fail.
The thing that would break is of course the re-fetching of source code from that repo, but that's kinda obvious, don't you think? You can't clone from a deleted Git repo. A common solution to that problem (language-independent) is to mirror the code somewhere else.
Now, there's one single thing in the standard Go tooling universe that interacts with remote Git repos. It's 'go get', a small convenience wrapper around several VCS commands. It allows you to clone a remote repo to the right place on your disk while using the same syntax for Git, Mercurial and SVN alike. That's basically all it does. Yet, some people expect it to be some kind of elaborate package management system. It's not.
With Go being a new language (1.0 in 2012), there is no excuse for not having had a solid pkg solution in 1.0.
It's one of the things holding Go back, in my opinion, producing makeshift solutions like Glide and gopkg.in, which are only minimally adopted.
I hope they also specify some mandatory versioning scheme (like npm with semver), because there are so many Go packages out there which don't even do any versioning!
The idea that somehow every single language and community needs its own crappy package management systems is just utterly broken.
There's no reason you can't use the same package format and tools for code written in multiple languages. You might get some additional convenience by having small language-specific plugins (like the automatic dependency extractors in RPM) but that's just a tiny amount of the necessary tooling. The vast majority is not language-specific at all.
With these language-community systems, there's also the problem of depending on code written in another language. My Go program might well depend on a C library, just as my C program can require a shell script.
I'm new to Go and I read some documents [1][2][3] on what 'vendoring' means but I still don't understand.
What problem does it solve? Why is it needed?
Is there an 'explain for a 5-year-old' version, or better yet, an 'explain for someone who is used to other packaging systems that may not be entirely elegant but work pretty okay most of the time, like Java's Maven or Node's npm'
It's common practice for folks building Go binaries (but not libraries) to check in the code for their dependencies. So whenever you build Docker, you get the same thing unless Docker made a change (either by changing Docker itself or explicitly checking in an upgrade to a dep).
There are a few nice things about it; vendoring means you won't have a dependency on other repos or archives being available, and if you're a binary maintainer you can see source changes in your own code and that of your dependencies side-by-side if you're, say, investigating a bug that came up between certain revisions.
The not-so-nice things are pretty much what you'd think. There isn't a command to say "give me upgrades, but not ones that the author flagged as breaking changes". If you're writing a _library_ that is depended on by other _libraries_ there is no one conventional way to safely do breaking changes, so there's a lot of effort to avoid those changes plus sometimes hacks like different repo paths for different versions.
Part of the history is that, internally, Google's got one version of third-party code in the latest rev of their source control tree, by company policy. You upgrade something for one, you upgrade it for the whole codebase. Sounds daunting, but they have lots of testing, canary servers, and so on to catch problems. There is a great talk about some of this by Rachel Potvin (can't recall if it covers the third_party/ stuff specifically):
So Go's initial model didn't consider multiple versions of packages because Google had approached upgrades a different way internally. Of course, unlike Google, Go users on the outside will never have any central mechanism to keep all the world in sync when there are API changes. Hence the conversation today about package management.
I'm grateful for anyone who serves on the committee, as it could be a thankless job (some people may not like the agreed upon tool/process, but hopefully a small group that doesn't feel entirely disenfranchised).
I really hope whatever they come up with is welcomed by the community and projects migrate to using it. Basically becoming the "Standard Go Way" to do things.
One of the things I love about Golang is that it's opinionated; gofmt forces your code to conform to a uniform style, the linters are top-notch, and the go command line tool builds your code in exactly one way. Some of the decisions made (the lack of generics sticks out) are frustrating at times, but the language authors having opinions stops many developer quibbles from becoming a problem. Now it's time to add packaging to this list.
When we use go at work, everyone conforms to the same coding standards, everyone uses the same documentation tools, everyone deals with the same build system, and everyone (for the most part) writes cleaner, more uniform code because of it. I get why some developers hate Golang: it forces lots of its own opinions on how to do things on you, but in exchange it removes almost completely the bikeshed problem; everyone has to paint their bike shed the same color, so we can move onto the meat of solving the problems. I personally have been fine with go get and the vendoring tools people have created because we're using almost entirely internal libraries so getting the latest version of the master branch is usually the right choice, but I understand the need for a better package management solution. I just hope, like the parent commenter, that the go package solution is as opinionated as the rest of the language; give us an easy, sensible default so that the tooling gets the hell out of my way. I hate working on C++ and Java projects at my job because of how heavy the tooling often is (CMake and Maven, respectively), and that's just for the build system; forget autocomplete or linting or refactoring support unless you're in a commercial IDE. On the other hand, Go is simple enough that emacs is powerful enough to take care of every IDE feature I could ever want. Hell, the command line and nano is plenty to edit Go code, because the build system works exactly one way, and that is so incredibly important to me.
So yeah, I want to see a good solution. But I will take a less complete solution over one that adds more complexity.
If you want to model a real package manager, don't do a comparison of existing tools, all of which have flaws. Start with the feature set of RPM, and then ask a professional software packager + developer what is missing.
In order to have a feature-complete package manager (that is, it does everything you will concievably need) you need at least the following:
- Unique file/object tracking
- Upgrade & Rollback
- Dynamic string interpolation (file paths, for example)
- Identification of architecture and OS
- ACL and extended permission support
- Tracking of both source and binary dependencies & versions
- Before & after script hooks for every single operation
- Cryptographic signing & verification
- Support for multiple central & distributed repositories
This is, of course, completely different from a build system, but has a few features in common. You may not need a package manager at all; you may just need a feature-complete build system, which is how you end up with CPAN. (There is no Perl package manager, though PPM attempts one, and Carton is just a bundler)
vendoring is dependency storage. Management, especially when you have transitive dependencies (deps of deps), is more than storage. We need to solve the non-storage related problems.
Are you familiar with the diamond dependency problem? That's one example.
[+] [-] aikah|9 years ago|reply
The question isn't being solved yet because to solve it seriously it would require to change how the language works at first place.
[+] [-] maerF0x0|9 years ago|reply
I see it as an excellent way to allow me to ensure that no matter what my project continues to build and work. And dont say "just pin a version" because there have been packages that lose or update their tags and break things, or the distribution channel (npmjs, github, w/e) go down and suddenly I am up a creek for a deploy/build.
When I vendor, I have what I need to build and deploy my code with a lot fewer points of failure.
[+] [-] 0xmohit|9 years ago|reply
And Go 1 compatibility promise [0] would prevent any change.
In theory, you could define a GOPATH to install custom packages. You install half a dozen or so packages, and it's not trivial to determine what all you installed. A Go update arrives. Now some of those packages fail to update. Either spend hours debugging, reporting bugs, or rm -rf $GOPATH and reinstall everything.
The language is wonderful in so many ways, but a number of seemingly silly things make it annoying.
[0] https://golang.org/doc/go1compat
[+] [-] BarkMore|9 years ago|reply
[+] [-] Thaxll|9 years ago|reply
This doesn't make any sense. What problem are you talking about?
[+] [-] paulsmith|9 years ago|reply
And yet, people manage to get work done in it.
[+] [-] 0xdeadbeefbabe|9 years ago|reply
Who is reinventing wheels anyway?
[+] [-] unknown|9 years ago|reply
[deleted]
[+] [-] mfer|9 years ago|reply
If you have questions or input those who are working on this problem are listening.
[+] [-] scrollaway|9 years ago|reply
Why is the protocol side of package management not a solved problem already?
At least Go reuses the git protocol to its advantage. But the general problem of package management applies to a lot of things.
"There is a package with a name and a description. That package has versions, which can be greater/lesser than others. Those versions may have deltas from one-another. They contain one or more files. Those files must have their integrity checked. etc, etc..." -> Why does every single language reimplement that?
Nevermind languages actually. Why does everyone reimplement that? Firefox/Chrome extensions, Android/ios app stores, a bazillion different game addons and app-specific stores...
We have a mostly-identical problem being solved a million different ways. pypi, npmjs, glide, they should be running the same server software. It'd be a first step towards standardization and avoid a bunch of mistakes repeated every time a new solution comes out.
Is anyone even trying to do something about this? (and before anyone says "do it yourself", a package management protocol is something I used to work on, not anymore. It keeps getting worse - it's not just a technical problem, also a political one.)
[+] [-] zzzcpan|9 years ago|reply
Personally I don't think we need yet another package manager with painful external dependencies, non-reproducible builds, conflicts, lack of rollbacks, etc. We have that with "go get". What we need is to actually solve all those thing. Make sure there are no fragile external dependencies, but everything is in the package manager, even gcc, binutils. You get the idea.
[+] [-] ahacker15|9 years ago|reply
(I've tried Glide but it slowness and few bugs made me move to Govendor. Nothing personal, it just didn't work out that well for me.)
A few questions:
- Is the intention desincentive the use vendoring?
- Is the intention to make a single solution, and then deprecate existing tools (Glide, Govendor and like)?
- Do you have the intention to make a central registry for Go packages? Or the solution will use GitHub and like, as it does today? It would be not backward compatible, but the document don't make this clear.
[+] [-] SEJeff|9 years ago|reply
[+] [-] grosbisou|9 years ago|reply
[+] [-] twblalock|9 years ago|reply
This creates a lot of problems that simply don't need to be there:
1. Versioning dependencies is pretty much impossible unless you make your own copy of them, and keep them updated. Good dependency management solutions do not require you to "vendor" your dependencies. They enable reproducible builds by keeping every version of an artifact around.
2. Github could change its URL schema, and most Go projects would stop compiling.
3. The maintainer could delete the Github repo.
4. Source control, specifically Github, is tightly coupled with dependency management, and that should not be necessary.
These are not unsolved problems. Maven solved every single one of these problems years ago. I don't expect people to use Maven to build Go applications, but clearly there is an opportunity to use Maven's good ideas and build a better dependency management system for Go.
I have to say, in defense of Maven, that it is very robust: artifacts are mirrored in many places, including in your own local repo; versioning works well, including availability of older versions; and nobody can screw you by deleting package or changing its Github url. My company uses Maven for Node.js as well as Java, because we've had so many problems with NPM.
I should also point out that the expectation that the core Go team should develop a dependency management system is a bit odd. The dependency management systems for Java, Ruby, Python, Node.js, and Perl were all developed by the community, not the core team. Some of them were later absorbed by the core team, but the point still stands.
[+] [-] slrz|9 years ago|reply
Thus, your statement: 2. Github could change its URL schema, and most Go projects would stop compiling.
is wrong. No single Go project that compiled before would now fail to do so. The common build tools use the import path to determine a path on the local filesystem where source or object code may be found. Neither a change of Github's URL schema, nor a repo deletion by some maintainer would cause your build to fail.
The thing that would break is of course the re-fetching of source code from that repo, but that's kinda obvious, don't you think? You can't clone from a deleted Git repo. A common solution to that problem (language-independent) is to mirror the code somewhere else.
Now, there's one single thing in the standard Go tooling universe that interacts with remote Git repos. It's 'go get', a small convenience wrapper around several VCS commands. It allows you to clone a remote repo to the right place on your disk while using the same syntax for Git, Mercurial and SVN alike. That's basically all it does. Yet, some people expect it to be some kind of elaborate package management system. It's not.
[+] [-] the_duke|9 years ago|reply
With Go being a new language (1.0 in 2012), there is no excuse for not having had a solid pkg solution in 1.0.
It's one of the things holding Go back, in my opinion, producing makeshift solutions like Glide and gopkg.in, which are only minimally adopted.
I hope they also specify some mandatory versioning scheme (like npm with semver), because there are so many Go packages out there which don't even do any versioning!
[+] [-] slrz|9 years ago|reply
There's no reason you can't use the same package format and tools for code written in multiple languages. You might get some additional convenience by having small language-specific plugins (like the automatic dependency extractors in RPM) but that's just a tiny amount of the necessary tooling. The vast majority is not language-specific at all.
With these language-community systems, there's also the problem of depending on code written in another language. My Go program might well depend on a C library, just as my C program can require a shell script.
[+] [-] zzzcpan|9 years ago|reply
[+] [-] niftich|9 years ago|reply
What problem does it solve? Why is it needed?
Is there an 'explain for a 5-year-old' version, or better yet, an 'explain for someone who is used to other packaging systems that may not be entirely elegant but work pretty okay most of the time, like Java's Maven or Node's npm'
[1] https://engineeredweb.com/blog/2015/go-1.5-vendor-handling/ [2] https://github.com/golang/go/wiki/PackageManagementTools [3] https://docs.google.com/document/d/1rMoZ0rmpxw6dShalHnkuPtia...
[+] [-] twotwotwo|9 years ago|reply
There are a few nice things about it; vendoring means you won't have a dependency on other repos or archives being available, and if you're a binary maintainer you can see source changes in your own code and that of your dependencies side-by-side if you're, say, investigating a bug that came up between certain revisions.
The not-so-nice things are pretty much what you'd think. There isn't a command to say "give me upgrades, but not ones that the author flagged as breaking changes". If you're writing a _library_ that is depended on by other _libraries_ there is no one conventional way to safely do breaking changes, so there's a lot of effort to avoid those changes plus sometimes hacks like different repo paths for different versions.
Part of the history is that, internally, Google's got one version of third-party code in the latest rev of their source control tree, by company policy. You upgrade something for one, you upgrade it for the whole codebase. Sounds daunting, but they have lots of testing, canary servers, and so on to catch problems. There is a great talk about some of this by Rachel Potvin (can't recall if it covers the third_party/ stuff specifically):
https://www.youtube.com/watch?v=W71BTkUbdqE
So Go's initial model didn't consider multiple versions of packages because Google had approached upgrades a different way internally. Of course, unlike Google, Go users on the outside will never have any central mechanism to keep all the world in sync when there are API changes. Hence the conversation today about package management.
[+] [-] unknown|9 years ago|reply
[deleted]
[+] [-] grey-area|9 years ago|reply
It only really gets complex on big projects with lots of dependencies which themselves have dependencies.
[+] [-] howeyc|9 years ago|reply
I'm grateful for anyone who serves on the committee, as it could be a thankless job (some people may not like the agreed upon tool/process, but hopefully a small group that doesn't feel entirely disenfranchised).
I really hope whatever they come up with is welcomed by the community and projects migrate to using it. Basically becoming the "Standard Go Way" to do things.
[+] [-] kaeshiwaza|9 years ago|reply
[+] [-] nameless912|9 years ago|reply
One of the things I love about Golang is that it's opinionated; gofmt forces your code to conform to a uniform style, the linters are top-notch, and the go command line tool builds your code in exactly one way. Some of the decisions made (the lack of generics sticks out) are frustrating at times, but the language authors having opinions stops many developer quibbles from becoming a problem. Now it's time to add packaging to this list.
When we use go at work, everyone conforms to the same coding standards, everyone uses the same documentation tools, everyone deals with the same build system, and everyone (for the most part) writes cleaner, more uniform code because of it. I get why some developers hate Golang: it forces lots of its own opinions on how to do things on you, but in exchange it removes almost completely the bikeshed problem; everyone has to paint their bike shed the same color, so we can move onto the meat of solving the problems. I personally have been fine with go get and the vendoring tools people have created because we're using almost entirely internal libraries so getting the latest version of the master branch is usually the right choice, but I understand the need for a better package management solution. I just hope, like the parent commenter, that the go package solution is as opinionated as the rest of the language; give us an easy, sensible default so that the tooling gets the hell out of my way. I hate working on C++ and Java projects at my job because of how heavy the tooling often is (CMake and Maven, respectively), and that's just for the build system; forget autocomplete or linting or refactoring support unless you're in a commercial IDE. On the other hand, Go is simple enough that emacs is powerful enough to take care of every IDE feature I could ever want. Hell, the command line and nano is plenty to edit Go code, because the build system works exactly one way, and that is so incredibly important to me.
So yeah, I want to see a good solution. But I will take a less complete solution over one that adds more complexity.
[+] [-] zellyn|9 years ago|reply
[+] [-] mmkx|9 years ago|reply
[+] [-] jvehent|9 years ago|reply
We even do it with just an S3 bucket for go.mozilla.org: https://go.mozilla.org/
[+] [-] dlsniper|9 years ago|reply
[+] [-] peterwwillis|9 years ago|reply
In order to have a feature-complete package manager (that is, it does everything you will concievably need) you need at least the following:
This is, of course, completely different from a build system, but has a few features in common. You may not need a package manager at all; you may just need a feature-complete build system, which is how you end up with CPAN. (There is no Perl package manager, though PPM attempts one, and Carton is just a bundler)[+] [-] randomsofr|9 years ago|reply
[+] [-] themihai|9 years ago|reply
[+] [-] 0xdeadbeefbabe|9 years ago|reply
Edit: Seriously, I'm not being a jerk.
[+] [-] mfer|9 years ago|reply
Are you familiar with the diamond dependency problem? That's one example.
[+] [-] leonardinius|9 years ago|reply
[+] [-] zellyn|9 years ago|reply