I completely see the benefits of using guix, transactional package management seems at the very least an interesting alternative to docker in some scenarios, and could be very interesting from a tool automation perspective.
I don't want to sound negative, but I think the biggest problem with getting me or other ruby developers to package our gems to guix would be the following. The bundler/rbenv system I've been using for years is pretty darn good, personally I haven't had problems with it. I've used it in massive and small projects, and admittedly the only inconvenience is having to manually or through orchestration install the dependencies of certain gems, of which there are also only a few (mysql2 with libmysqlclient-dev and his nokogiri mention are probably the most notable). He does point this out but if you're using docker or some automation tool this is a pretty minor point.
Also I already ensure my gems run their tests on mri, jruby, rubinius, etc. and this seems like extra work for little benefit, especially when most in house deployment scenarios are running with docker or chef/puppet on top of the classic rbenv/rvm and bundler setup. My biggest question is after upgrading a gem version how easy is it for me to push the new version of the package to guix?
So essentially, I wish this article had more arguments to show why guix will better the life of ruby devs.
I don't know if it's the same with Guix but in NixOS it's just a matter of running `bundix` and all the Gemfile.lock dependencies are converted to a nix expression. Gem developers can keep developing as usual.
Maybe the only restriction is to make sure the gem has configurable paths if it tries to access system files.
I'm experimenting with it at the moment and converted my blog to be managed by nix on OSX. It's not a walk in the park but it's working. Clone the repo ( https://github.com/zimbatm/zimbatm.com ) and type `nix-shell` (needs to be installed first) and you should get all the blog dependencies available in your environment.
As someone who deploys Ruby software to our servers, I've come up with a formula that's served us well.
We take all gems associated with the app (like a Rails app, but we do it with others), and stuff them along with the code into a package created by fpm.
In other words, the packages are completely self-contained; no gems are used from the environment.
The key to this is the --path option to bundle install, i.e.:
bundle install --path vendor/bundle
This command makes sure all my gems are in a sub directory of my project called vendor/bundle. When creating the debian package using fpm which includes all the source code, vendor/bundle also comes along with the package.
After installing the app on our Linux servers, 'bundle exec rails server' uses the gems from vendor/bundle.
This approach does require that I create the debian package on a machine that closely mirrors the deployed servers, so that compiled gems make any sense.
Most all the issues described in the article have not been problems because of this approach. I used to use rvm as part of my build process; it was very brittle. I'm completely happy now; just wanted to share in case that helps someone else.
>We take all gems associated with the app (like a Rails app, but we do it with others), and stuff them along with the code into a package created by fpm.
That is exactly what I've started doing at work, actually. It's a short-term win and rather simple, but it's a pretty big hack. Package bundling is quite poor form as it leads to file system duplication and for each duplication you have an additional location to patch when a security vulnerability is inevitably discovered. Ideally each Ruby gem would be its own package. I'm also concerned with reproducibility, so I build packages in as isolated of an environment as possible where I run 'bundle install' with a fresh vendor/bundle directory. Builds take longer than they should because of all the re-downloading. Using Guix, where I can easily represent each Ruby component in a package object, I can take advantage of its content-addressable storage to save time on repeated builds by using the cached gem builds from previous times.
What is exactly wrong with piping curl into bash? I'm trusting RVM anyway. As for functional package managers, I'll believe it when I see it. The reason things are brittle is that any time something in the middle of a dependency chain changes the whole chain may break. The more you freeze things, the greater chance there is to freeze up a security hole (see what is going on with Docker). So you have two choices: be ok with the freeze, or update the whole chain. In this respect I prefer the latter, and I especially prefer the way Ruby does it where, due to the global namespace, I'm forced to upgrade the gem everywhere so I end up fixing things and keeping the whole project up to date. While this seems frustrating at first, once you realize that everyone in the Ruby community is doing it, you realize that the reason it is usually painless to update your gems is because other people have fixed those other holes. What is especially nice is that you only have to remember things that are broken at T0, unlike, say, Node, where a popular nested node_module may have 10 different versions in your dependencies and you keep hitting the same bug or security hole over and over again.
I've found all the hate surrounding this practice to be pretty misdirected. As long as it's served over https, it's functionally equivalent to "download and run this random package installer!", which is generally tolerated.
I'm not sure I want to install guix on my main Archlinux box, as I don't want conflicts with the existing libraries and applications, but I would very much play with it if it were possible to do something like this:
guix virtualenv myapp
cd myapp
guix activate
guix install postgres # installs a local postgres installation available only on this "environment"
guix environment -l myapp.scm # install app dependencies
<do work>
guix deactivate
This would be a killer feature for me and I would ditch virtualenv, pip, npm, perhaps Docker in a heartbeat.
> I'm not sure I want to install guix on my main Archlinux box, as I don't want conflicts with the existing libraries and applications
That's the beauty of functional package management: no global state is modified. Whatever Guix does is restricted to its own namespace, e.g. the /gnu directory. Every package that is built or installed ends up there and is completely isolated from the rest of the system.
To actually use the software you use a link to a profile (which is located in the "local state" directory of Guix).
You can install packages into separate profiles with Guix:
Yes, the 'guix environment' tool is like a language-agnostic virtualenv. Guix does not use or interfere with any components from the host distro, so you can install Guix on top of Arch Linux without fear. We have a binary installation method[0] that isn't too difficult that you can try out. I use Guix on top of an Ubuntu host at work.
For your PostgreSQL example, you'd still need to do things like initialize the DB cluster and start the daemon on your own (the GuixSD distro has a system service for this[1]), but to spawn a shell with the postgres daemon and client programs available you would run:
guix environment --ad-hoc postgresql
Exiting the sub-shell would do essentially what you had in mind with 'guix deactivate': Make postgresql available for garbage collection (via 'guix gc') if/when nothing else is using it.
Notice the striking similarities of package management definition files and configuration management, where the latter does more related to state, coordination, attributes and templating files. Packages are just more generic containers of resources and metadata, configuration management makes those resources concrete on specific systems and applies tweaks to converge on the desired catalog state.
Btw, has anyone written a usable Guile-based cfg mgmt tool yet?
>Btw, has anyone written a usable Guile-based cfg mgmt tool yet?
That would be Guix. We use the same primitive utilities for package management and system configuration management. I'm typing this from a laptop running the GuixSD distro, and if a system upgrade were to break things, I can roll-back to a previous, working generation of the system at boot time and keep on going.
Configuration management is slowly coming to Guix. Guix is the foundation of the Guix System Distribution, which is a system instantiated from a declarative system configuration file /etc/config.scm. There is work under way to extend this to managing multiple machines.
I like the approach of using gs (simple gemsets) [1] and dep (simple dependencies) [2]. I've been using this approach of handling dependencies for a while now.
I've written a simple shell script as a replacement for gs (since I constantly forget if I was in a sub-shell or not). All it does is change $GEM_PATH and $GEM_HOME when $PWD changes (while staying in the same shell). If the current $PWD has a directory ".gs", `gem install` will install them there instead of globally. All I have to remember is to create a .gs directory when I start a new project.
It does not address all issues highlighted in the article, nor do I know how well this would work in a production environment with automated deployments. It's probably more of a work around than a solution. I vastly prefer it for personal projects, though. Hate solving gem conflicts with "bundle exec".
Instead of changing environment variables when $PWD changes, I went the other way: make it obvious when you're in a subshell by strapping $VIRTUAL_ENV onto $PS1. Again, simple shell scripts win[0][1].
I'm still interested in a cross-language approach, but I'm suspicious of anything which requires an explicit import step for rubygems that doesn't involve resolving the dependency tree. Unless Gemfile.lock is involved, it's going to need to copy Bundler's resolution algorithm, which seems fraught.
no 'ruby version manager', eg 'chruby, rvm, or rbenv' make any sense in the nix/guix world. the article would have done a disservice to itself to go into detail on each one, when the goal was really to show how to manage ruby projects in guix.
I imagine everything said in the article also applies to the Nix package manager[0]. One of its nice features is the Nix expression language (aka Nix) [1]. It has an elegant mixture of purity and pragmatism. I imagine guile scheme would have a lot more integration opportunities outside of just system/build/configuration management. It'd be interesting to see Guix expressions that actually mix other scheme libraries or concepts, rather than just taking Nix expressions and removing the syntax.
I very much like how Guix is just a Guile Scheme library and can itself be integrated with other Scheme code. There is, for example, guix-web, a web interface to the package manager; there also is an Emacs interface using geiser.
Using Guile has turned out to be a great feature in itself and it is still fueling the development of new features that otherwise would be awkward to implement, such as the new "guix graph" command to visualise dependencies.
>It'd be interesting to see Guix expressions that actually mix other scheme libraries or concepts, rather than just taking Nix expressions and removing the syntax.
We already do this. Our build scripts are written in Guile, not Bash like in Nix, and they take advantage of Guile's standard library which has things like a pattern matcher, an HTTP client and server, a POSIX interface, a foreign-function interface, an XML parser, etc. We use third-party libraries such as guile-json and guile-charting in some of our tools, as well.
> I'm not sure how you feel, dear reader, but my Ruby environments feel like one giant, brittle hack
Of course things are not perfect (and guix may have a few cool ideas), but I think we should keep in mind that both rvm and bundler have been huge improvements.
The hacks people needed to come up with before rvm and bundler existed were an order of a magnitude more brittle.
Yeah, while I think the approach outlined in the post makes sense and could be really great, I really haven't had any of the problems outlined since the bad old `config.gem` days. In my experience, everyone defaults `~> X.Y.Z` dependencies, with more lenient patterns justified by actual testing, and I can't remember the last time I saw a blanket `>= X` requirement. This has meant that all, or very nearly so, of the dependency problems I find myself handling are due to actual incompatibility, which I'm glad were caught.
[+] [-] jfaucett|10 years ago|reply
I don't want to sound negative, but I think the biggest problem with getting me or other ruby developers to package our gems to guix would be the following. The bundler/rbenv system I've been using for years is pretty darn good, personally I haven't had problems with it. I've used it in massive and small projects, and admittedly the only inconvenience is having to manually or through orchestration install the dependencies of certain gems, of which there are also only a few (mysql2 with libmysqlclient-dev and his nokogiri mention are probably the most notable). He does point this out but if you're using docker or some automation tool this is a pretty minor point.
Also I already ensure my gems run their tests on mri, jruby, rubinius, etc. and this seems like extra work for little benefit, especially when most in house deployment scenarios are running with docker or chef/puppet on top of the classic rbenv/rvm and bundler setup. My biggest question is after upgrading a gem version how easy is it for me to push the new version of the package to guix?
So essentially, I wish this article had more arguments to show why guix will better the life of ruby devs.
[+] [-] zimbatm|10 years ago|reply
Maybe the only restriction is to make sure the gem has configurable paths if it tries to access system files.
I'm experimenting with it at the moment and converted my blog to be managed by nix on OSX. It's not a walk in the park but it's working. Clone the repo ( https://github.com/zimbatm/zimbatm.com ) and type `nix-shell` (needs to be installed first) and you should get all the blog dependencies available in your environment.
[+] [-] ownedthx|10 years ago|reply
We take all gems associated with the app (like a Rails app, but we do it with others), and stuff them along with the code into a package created by fpm.
In other words, the packages are completely self-contained; no gems are used from the environment.
The key to this is the --path option to bundle install, i.e.:
bundle install --path vendor/bundle
This command makes sure all my gems are in a sub directory of my project called vendor/bundle. When creating the debian package using fpm which includes all the source code, vendor/bundle also comes along with the package.
After installing the app on our Linux servers, 'bundle exec rails server' uses the gems from vendor/bundle.
This approach does require that I create the debian package on a machine that closely mirrors the deployed servers, so that compiled gems make any sense.
Most all the issues described in the article have not been problems because of this approach. I used to use rvm as part of my build process; it was very brittle. I'm completely happy now; just wanted to share in case that helps someone else.
[+] [-] davexunit|10 years ago|reply
That is exactly what I've started doing at work, actually. It's a short-term win and rather simple, but it's a pretty big hack. Package bundling is quite poor form as it leads to file system duplication and for each duplication you have an additional location to patch when a security vulnerability is inevitably discovered. Ideally each Ruby gem would be its own package. I'm also concerned with reproducibility, so I build packages in as isolated of an environment as possible where I run 'bundle install' with a fresh vendor/bundle directory. Builds take longer than they should because of all the re-downloading. Using Guix, where I can easily represent each Ruby component in a package object, I can take advantage of its content-addressable storage to save time on repeated builds by using the cached gem builds from previous times.
[+] [-] regularfry|10 years ago|reply
[+] [-] buren|10 years ago|reply
Perhaps this could make http://devblog.avdi.org/2015/08/11/what-its-like-to-come-bac... easier ;)
[+] [-] davexunit|10 years ago|reply
[+] [-] 3pt14159|10 years ago|reply
[+] [-] burke|10 years ago|reply
[+] [-] 1_player|10 years ago|reply
I'm not sure I want to install guix on my main Archlinux box, as I don't want conflicts with the existing libraries and applications, but I would very much play with it if it were possible to do something like this:
This would be a killer feature for me and I would ditch virtualenv, pip, npm, perhaps Docker in a heartbeat.[+] [-] rekado|10 years ago|reply
That's the beauty of functional package management: no global state is modified. Whatever Guix does is restricted to its own namespace, e.g. the /gnu directory. Every package that is built or installed ends up there and is completely isolated from the rest of the system.
To actually use the software you use a link to a profile (which is located in the "local state" directory of Guix).
You can install packages into separate profiles with Guix:
Or you can just run a shell in a volatile environment as specified in `myapp.scm` To exit this environment, just exit the shell.In summary: it can already be used just like you want, but it's much less verbose :)
[+] [-] davexunit|10 years ago|reply
For your PostgreSQL example, you'd still need to do things like initialize the DB cluster and start the daemon on your own (the GuixSD distro has a system service for this[1]), but to spawn a shell with the postgres daemon and client programs available you would run:
Exiting the sub-shell would do essentially what you had in mind with 'guix deactivate': Make postgresql available for garbage collection (via 'guix gc') if/when nothing else is using it.[0] https://gnu.org/software/guix/manual/html_node/Binary-Instal...
[1] https://gnu.org/software/guix/manual/html_node/Database-Serv...
[+] [-] rokgarbas|10 years ago|reply
[+] [-] bro-stick|10 years ago|reply
Btw, has anyone written a usable Guile-based cfg mgmt tool yet?
[+] [-] davexunit|10 years ago|reply
That would be Guix. We use the same primitive utilities for package management and system configuration management. I'm typing this from a laptop running the GuixSD distro, and if a system upgrade were to break things, I can roll-back to a previous, working generation of the system at boot time and keep on going.
[+] [-] rekado|10 years ago|reply
[+] [-] josteink|10 years ago|reply
When it comes, I'm certain it will come to Emacs first :)
[+] [-] waxjar|10 years ago|reply
I've written a simple shell script as a replacement for gs (since I constantly forget if I was in a sub-shell or not). All it does is change $GEM_PATH and $GEM_HOME when $PWD changes (while staying in the same shell). If the current $PWD has a directory ".gs", `gem install` will install them there instead of globally. All I have to remember is to create a .gs directory when I start a new project.
It does not address all issues highlighted in the article, nor do I know how well this would work in a production environment with automated deployments. It's probably more of a work around than a solution. I vastly prefer it for personal projects, though. Hate solving gem conflicts with "bundle exec".
[1]: https://github.com/soveran/gs [2]: https://github.com/cyx/dep
[+] [-] regularfry|10 years ago|reply
I'm still interested in a cross-language approach, but I'm suspicious of anything which requires an explicit import step for rubygems that doesn't involve resolving the dependency tree. Unless Gemfile.lock is involved, it's going to need to copy Bundler's resolution algorithm, which seems fraught.
[0]: https://github.com/regularfry/gemsh [1]: https://github.com/regularfry/rv
[+] [-] lighthawk|10 years ago|reply
[+] [-] miah_|10 years ago|reply
[+] [-] davexunit|10 years ago|reply
[+] [-] frontsideair|10 years ago|reply
[+] [-] roller|10 years ago|reply
[0]: http://nixos.org/nix/ [1]: http://nixos.org/nix/manual/#chap-writing-nix-expressions
[+] [-] rekado|10 years ago|reply
Using Guile has turned out to be a great feature in itself and it is still fueling the development of new features that otherwise would be awkward to implement, such as the new "guix graph" command to visualise dependencies.
[+] [-] davexunit|10 years ago|reply
We already do this. Our build scripts are written in Guile, not Bash like in Nix, and they take advantage of Guile's standard library which has things like a pattern matcher, an HTTP client and server, a POSIX interface, a foreign-function interface, an XML parser, etc. We use third-party libraries such as guile-json and guile-charting in some of our tools, as well.
[+] [-] davidroetzel|10 years ago|reply
Of course things are not perfect (and guix may have a few cool ideas), but I think we should keep in mind that both rvm and bundler have been huge improvements.
The hacks people needed to come up with before rvm and bundler existed were an order of a magnitude more brittle.
[+] [-] transfire|10 years ago|reply
[+] [-] rokgarbas|10 years ago|reply
[+] [-] schneems|10 years ago|reply
[+] [-] sanderjd|10 years ago|reply
[+] [-] paublyrne|10 years ago|reply
[+] [-] davexunit|10 years ago|reply
[+] [-] unknown|10 years ago|reply
[deleted]