top | item 40763640

Ruby: A great language for shell scripts

442 points| lucasoshiro | 1 year ago |lucasoshiro.github.io

353 comments

order
[+] codesnik|1 year ago|reply
I sometimes wonder why we don't see ruby used for shell stuff more often. It inherited most of the good stuff for shell scripting from Perl, and Perl took a lot of it's syntax from sh and sed and awk, so almost anything you can do in shell script you can do in ruby, but with an option of making it gradually less terse and more readable, while having sane variables and data handling from the start.

Also ruby is great in allowing complexity to grow smoothly, no sudden hiccups. You start with just one line (everything goes into module main implicitly), extend it to a single-file script, require some built-in libraries, then add a module or helper class in the same file, and only then maybe extract those files to required files, add gems, whatever. No boilerplate whatsoever, no jumps, no big rewrites.

meanwhile, a lot of tooling nowadays is written in Go, and I have no idea why, it's not friendly for os manipulation at all, and number crunching power is not needed in many, many tasks of that sort.

[+] NullInvictus|1 year ago|reply
I think the quality of a language for shell scripting is often secondary. What’s of greater significance is where it is at. I.e., does it have it already installed? The answer with Linux and Bash is almost always “yes”. Not so with ruby.

The moment you start asking the user to install things, you’ve opened up the possibility for writing a program rather than a shell script. The lifecycle of a piece of software is almost always one of growing responsibility. This cycle is devastating when it happens to shell scripts. What was once a simple script slowly becomes creaking mass of untestable, poorly understood code playing in the traffic of swimming environments (which grep you got, buddy?).

I guess I’m saying that once you open up the possibility of writing a program, you generally take that option and are usually happier for it. In the “write a program” world, ruby is still good, but it becomes a far harder question to answer whether ruby is still the right choice. There are a lot of languages with a lot of features engineers like.

[+] asa400|1 year ago|reply
> I sometimes wonder why we don't see ruby used for shell stuff more often.

The reason we don't see Ruby used more for shell stuff is because Python won this particular war. It's already installed on basically every Linux distribution out there, and this simple fact outweighs all other language considerations for probably >95% of people who are writing shell scripts in something that isn't Bash.

Personally, I don't much like Python, and even though Ruby is not my favorite language either, I find it much better than Python for this kind of work. But I don't get to decide how Debian builds their infrastructure, so in the end, I tend to use Python.

[+] seabrookmx|1 year ago|reply
I think golang is used because you can easily create a single static binary, which is incredibly easy to distribute. I often find non-trivial CLI tools written in Python cumbersome because of the dependency wrangling necessary.
[+] shagie|1 year ago|reply
> ... maybe extract those files to required files, add gems, whatever.

CPAN is the killer feature of Perl. It just works. First off, most of the time I don't need a CPAN module for doing shell scripting in perl. Perl itself is rich enough with the file manipulations that are needed for any script of less than 100 lines.

My experiences with Ruby and installing gems have been less pleasant. Different implementations of Ruby. Gems that don't compile / work on certain architectures. Breaking changes going forward where a script that was written 2 years ago doesn't work anymore. Sometimes it's someone was doing something clever in the language that doesn't work anymore. Other times its some gem got updated and can't be used that way anymore. ... which brings us to ...

I believe that Go's advantages come into play when the program gets more complex that that 100 line size and it becomes a "program" rather than a "script" that has complexity to deal with. Furthermore, executables built in Go are most often statically linked which means that someone upgrading the libraries doesn't break what is already working.

[+] solidsnack9000|1 year ago|reply
Instability. Ruby has not been the same language for very long. Migrating to 1.9 was a huge hassle for many firms. This may seem like a long time ago in tech years; but then there was Ruby 2.0; and shell scripts, meanwhile, have stayed the same the whole time.

A secondary reason is that Ruby has been very slow for much of its life, which means that for situations where you need to run a huge stack of scripts -- init systems, for example -- it would be punishing.

Ruby does have a terse and intuitive syntax that would make for a good system shell. Although it has some magic, it is less magical and confusing than shell itself. Ruby provides many basic data types that experience has proven are useful for shell scripting -- like arrays and dictionaries -- and they are integrated in a much cleaner and clearer way than they are integrated into widely used shells like Bash.

System tools that are written in Go may still make sense to write in Go, though. Go, it is true, does not have a nice terse syntax for short scripts and one liners; and it doesn't have a default execution model where everything is in main and so on; but that is because it is not a scripting language. Other languages used to write system tools and system services -- like C, C++, Java and Rust -- don't have those things either.

[+] braza|1 year ago|reply
> I sometimes wonder why we don't see ruby used for shell stuff more often

The best piece of code that I worked on was an ETL in pure Ruby. Everything in modules, simply to read, no crazy abstractions, strange things like __main__, abstract clssses or whatever.

Maybe others can chime in, but the main difference that is found in ruby developers is that they really have fun with the language making everything with a higher lever of software craftsmanship that other folks in the data space, e.g. Python of Julia.

[+] yen223|1 year ago|reply
It wasn't that long ago that all the interesting infrastructure projects (vagrant, chef) were written in Ruby.
[+] giraffe_lady|1 year ago|reply
You can kind of figure it out by skimming the comments here. Most mainstream languages have decent-to-great tools built in for scripting, so the difference isn't that huge. So people just prefer to script in the language they already prefer in general, or that the project is written in.
[+] pdimitar|1 year ago|reply
> meanwhile, a lot of tooling nowadays is written in Go, and I have no idea why

No-dependencies final static binary.

> it's not friendly for os manipulation at all

If you say so. I'd love to hear how did you get to that conclusion.

> and number crunching power is not needed in many, many tasks of that sort.

You are aiming very wrongly, it's about startup time. I got sick of Python's 300+ ms startup time. Golang and Rust programs don't have that problem.

[+] SoftTalker|1 year ago|reply
> why we don't see ruby used for shell stuff more often

Simple, ruby is not installed by default. Even Python, while it is on (almost?) all modern Linux distributions, is not installed on the BSDs.

[+] lelanthran|1 year ago|reply
> meanwhile, a lot of tooling nowadays is written in Go, and I have no idea why, it's not friendly for os manipulation at all

I'm not sure where you're going with this: My experience of Ruby and Go is that:

1. Go is a lot easier to do OS manipulation type stuff.

2. Go is a lot easier to modify down the line.

TBH, #2 is not really a consideration for shell-scripts - the majority of the time the shell script is used to kick off and monitor other programs, transforming an exact input into an exact output.

It's glue, basically, and most uses of glue aren't going to require maintenance. If it breaks, it's because the input or the environment changed, and for what shell is used for, the input and the environment change very rarely.

[+] zarzavat|1 year ago|reply
What tooling do you use that’s written in Go? I’d have said that Python is the most popular language for tooling, by a country mile.

The only tooling I know that’s written in Go is Docker.

[+] zarathustreal|1 year ago|reply
Just because you don’t see it doesn’t mean it’s not the most-used shell scripting language. For example, when I was at AWS it was used for templating in something like 90% of all pipeline tooling
[+] SPBS|1 year ago|reply
> meanwhile, a lot of tooling nowadays is written in Go, and I have no idea why

What? Go is used because distributing a static binary without any dependencies is way better than asking each and every user to download an interpreter + libraries.

[+] cyclotron3k|1 year ago|reply
Totally agree! Other tricks I rely on:

a) put a `binding.irb` (or `binding.pry`) in any rescue block you may have in your script - it'll allow you to jump in and see what went wrong in an interactive way. (You'll need a `require 'irb'` in your script too, ofc)

b) I always use `Pathname` instead of `File` - it's part of the standard library, is a drop in replacement for `File` (and `Dir`) and generally has a much more natural API.

c) Often backticks are all you need, but when you need something a little stronger (e.g. when handling filenames with spaces in them, or potentially hostile user input, etc), Ruby has a plethora of tools in its stdlib to handle any scenario. First step would be `system` which escapes inputs for you (but doesn't return stdout).

d) Threads in Ruby are super easy, but using `Parallel` (not part of the stdlib) can make it even easier! A contrived example: `Parallel.map(url_list) { |url| Benchmark.measure { system('wget', url) }.real }.sum` to download a bunch of files in parallel and get the total time.

MacOS has Ruby 2.6 installed by default which is perfectly serviceable, but it's EOL and there are plenty of features in 3+ that make the jump more than worthwhile.

[+] MatthiasPortzel|1 year ago|reply
> You'll need a `require 'irb'` in your script too, ofc

irb is a part of Ruby core, so this isn’t true. (It may have been at one point? I’m not sure.)

I love binding.irb. I use it all the time.

[+] xavdid|1 year ago|reply
Ruby's a great language- I've always enjoyed its ergonomics and clarity. But its editor tooling hasn't kept up with its one-time competitor, Python.

I've mostly been in the Python ecosystem for the past few years and the LSP investment from Microsoft has really shown. Rich Python support in VSCode is seamless and simple. Coming back to Ruby after that caught me off guard - it feels like I'm writing syntax-highlighted plain text. There's an LSP extension from Shopify, but it's temperamental and I have trouble getting it working.

Editor support isn't everything (the actual language design is still the most important), but it definitely affects how eager I am to use it. I basically never choose Ruby over Python given the option, which is too bad. Ruby's a cool language!

[+] kazinator|1 year ago|reply
It seems like a waste of precious syntax to dedicate backticks to running shell commands.

What's between the backticks is not even portable; the commands rely on an operating-system-specific command interpreter.

> puts `ls`.lines.map { |name| name.strip.length } # prints the lengths of the filenames

Fantastic example, except for the commandment violation: "thou shalt not parse the output of 'ls'"!

You really want to read the directory and map over the stat function (or equivalent) to get the length.

  2> (flow "."
       open-directory
       get-lines
       (mapcar [chain stat .size]))
  (727 2466 21 4096 643 16612 5724 163 707 319 352135 140 51 0 4096
   114898 1172428 1328258 4096 4096 4096 29953 4096 4096 0 27 4096
   4096 35585 8450 968 40960 14610 4096 14 755128 1325258 4096 17283
   218 471 104 4096 99707 1255 4096 129 4096 721 9625 401 15658
   4096 235 98 1861 664 23944 4286 4096 1024 0)
[+] Too|1 year ago|reply
They have inherited the second biggest mistake of shellscripts; Requiring the user to manually check $? after each command.

No thanks. Anything that doesn't have error handling enabled by default goes straight in the trash bin.

[+] lucasoshiro|1 year ago|reply
> It seems like a waste of precious syntax to dedicate backticks to running shell commands.

Except from Bash (where backticks also have the same purpose as Ruby), I only remember seeing backticks in:

- Kotlin, for naming functions with phrases. The only use case that I remember for it was to creating more meaningful names for test functions. I don't think that it was so useful... - Lisp dialects for quasiquotes, which is meaningless in Ruby - Haskell, for making functions infix, and I can't see why would it be useful in Ruby (Ruby has it own way to make infix methods) - JS, for creating templates. In Ruby we can use double quotes for that

> "thou shalt not parse the output of 'ls'"

Yes, but it was only an example of associating backticks and the language features. Of course it is not ideal, but it is an example that everyone will understand. You'll probably won't want to use it (even because you can do it with Dir.glob('*').map { |f| f.length }).

The same happened later in the text when I used a regex to find the Git branch.

> You really want to read the directory and map over the stat function (or equivalent) to get the length.

But that is not Ruby.

[+] kouteiheika|1 year ago|reply
> You really want to read the directory and map over the stat function (or equivalent) to get the length.

Indeed. Equivalent in Ruby:

    Dir["*"].map { |x| File.size(x) }
[+] lkuty|1 year ago|reply
Is this Common Lisp? `mapcar` makes me think of it but I do not know about `flow`.
[+] pdntspa|1 year ago|reply
And what psychopath is trying to parse the output of ls?

None of this makes any sense to me, and I write Ruby for my day job.

[+] dcchambers|1 year ago|reply
I work for a company that has a large Rails monolith. Although we use many more languages than just ruby these days, we still have a ton of scripting, config, and tooling that is all in Ruby. It's a joy to work with IMO.

Another common pattern I see is people using embedded ruby in a shell script. It does make it a little harder to read/understand at a glance, but it's nice for being able to do most simple things in sh but drop into ruby for things that sh/bash suck at.

That said, I get a feeling that the people that joined once we'd added stuff outside of the Rails monolith and don't know/use Ruby are...not big fans.

[+] mberning|1 year ago|reply
I had the same experience. Somebody in our company inherited a Ruby script and was trying to modify it and was stuck. They came to me exasperated. The error message was something really trivial like addition is not defined for some object type. If you don’t understand the base level concepts of the language it’s going to be a very bad time. Sadly people are not that interested in learning about Ruby nowadays and look at it as a huge imposition to deal with. I love it still.
[+] drusepth|1 year ago|reply
Ruby is an amazing language. I've seen some systems it's not already installed on and hop over to something like perl/python in those cases, but Ruby is by far my preferred hammer for small scripts. The code is beautiful.

Small nit: your note in Feature 4 is actually supposed to be in Feature 5, I assume.

[+] nightpool|1 year ago|reply
Ruby is my favorite shell scripting language, I used it last year for a complex ffmpeg automation script (use blackdetect to detect long stretches of power-off and split a video file into much smaller components), and Ruby made it a breeze when I know it would have been a real struggle to get working in bash or powershell
[+] hiAndrewQuinn|1 year ago|reply
I have what probably sounds like a niche use case, where most of the boxes I work on don't have access to the Internet.

So for me, "is it installed in the base distribution" is the difference between being able to start immediately no matter which box I'm concerned with, and spending months trying to upstream a new program to be installed with our OS image team.

I took a look around a vanilla Debian 12 box, and didn't see Ruby in there [1]. So, sadly, although I really like the way Ruby looks, I'm going to have to stick with Bash and Python 3 for the hard stuff.

[1]: https://hiandrewquinn.github.io/til-site/posts/what-programm...

[+] djbusby|1 year ago|reply
These are great points, if you already have Ruby in your stack.

For me, when Bash ain't enough I upgrade to the Project language (PHP, Python, JS, etc). For compiled project I reach for LSB language (Perl, Python) before introducing a new dependency.

[+] phendrenad2|1 year ago|reply
Sadly, 9 out of 10 environments lack a Ruby interpreter out of the box. Are you going to add 5 minutes to your docker build to compile Ruby? Probably not.

Luckily, I've found that Perl has most of the best features of Ruby, and it's installed everywhere. It's time to Make Perl Great Again.

[+] bigstrat2003|1 year ago|reply
Why on earth would you add the compilation of a Ruby interpreter to your docker build? Just install it through the package manager of whatever distro your image is built on.
[+] manume|1 year ago|reply
> Sadly, 9 out of 10 environments lack a Ruby interpreter out of the box.

Please name those 10 environments you are talking about. In my experience, a reasonably recent Ruby version is present almost everywhere.

> add 5 minutes to your docker build

Why on earth would it take 5 minutes to install anything? If you install Ruby through a package manager (it's present in pretty much all of them: https://www.ruby-lang.org/en/documentation/installation/#pac...) it takes only seconds.

[+] corytheboyd|1 year ago|reply
Overall a nice lite write up! Bash is great, but it occasionally becomes untenable, usually around the time where HTTP and whatnot becomes involved. Same goes for shell exec exit codes, you can use an API like popen3 for this: https://ruby-doc.org/stdlib-2.4.1/libdoc/open3/rdoc/Open3.ht...

You mention using threads and regex match global variables in the same write up. Please use the regex match method response instead of the $1 variables to save yourself the potential awful debugging session. It even lets you access named capture groups in the match response using the already familiar Hash access API. Example: https://stackoverflow.com/a/18825787

In general, just don’t use global variables in Ruby. It’s already SO easy to move “but it has to be global” functionality to static class methods or constants that I’ve encountered exactly zero cases where I have NEEDED global variables. Even if you need a “stateful constant” Ruby had a very convenient Singleton mixin that provides for a quick and easy solution.

Besides, if you actually WERE to take advantage of them being global VARIABLES (reassigning the value) I would confidently bet that your downstream code would break, because I’m guessing said downstream code assumes the global value is constant. Just avoid them, there’s no point, use constants. This applies to any language TBH, but here we’re talking about Ruby :)

[+] samatman|1 year ago|reply
The things which make Ruby good for shell scripts are, to a large degree, things it inherited from Perl. Which was, and is, a great language for scripting.

People use it a lot less these days, for a lot of reasons, some better than others. I myself do simple stuff in bash, and pull out some Python for more complex scripting. My Perl chops have withered, last time I was paid to use it was 21 years ago, but it really is a great scripting language, and you'll find it installed on a great deal more systems than Ruby.

One of these days I'll give Raku a spin, just for old time's sake.

[+] bingemaker|1 year ago|reply
My first application of Ruby was to use it for shell auto-completions. I'm so grateful that I learnt Ruby first, and then Rails. Ruby is a great language to get some utility working out real fast. Rails is great for MVP. I fail to understand why people bitch about Ruby/Rails by comparing them to other languages/frameworks.
[+] lucasoshiro|1 year ago|reply
Yeah, as I mentioned, every time someone says something bad about Ruby for me, it is about Rails...
[+] derefr|1 year ago|reply
I love using Ruby for shell scripting, but there are also a ton of little nits I have to fix whenever I'm doing it.

For example: Ruby has no built-in for "call a subprocess and convert a nonzero exit status into an exception", ala bash `set -e`. So in many of my Ruby scripts there lives this little helper:

    def system!(*args, **kwargs)
      r = system(*args, **kwargs)
      fail "subprocess failed" unless $?.success?
      r
    end
And I can't ask "is this command installed" in an efficient built-in way, so I end up throwing this one in frequently too (in this instance, whimsically attached to the metaclass of the ENV object):

    class << ENV
      def path
        @path ||= self['PATH'].split(':').map{ |d| Pathname.new(d) }
      end

      def which(cmd)
        cmd = cmd.to_s
        self.path.lazy.map{ |d| d + cmd }.find{ |e| e.file? && e.executable? }
      end
    end
I have other little snippets like this for:

• implicitly logging process-spawns (ala bash `set -x`)

• High-level wrapper methods like `Pathname#readable_when_elevated?` that run elevated through IO.popen(['sudo', ...]) — the same way you'd use `sudo` in a bash script for least-privilege purposes

• Recursive path helpers, e.g. a `Pathname#collapse_tree` method that recursively deletes empty subdirectories, with the option to consider directories that only contain OS errata files like `.DS_Store` "empty" (in other words, what you'd get back from of a git checkout, if you checked in the directory with a sensible .gitignore in play)

...and so forth. It really does end up adding up, to the point that I feel like what I really want is a Ruby-like language or a Ruby-based standalone DSL processor that's been optimized for sysadmin tasks.

[+] Alifatisk|1 year ago|reply
Do you have a gist with all your tricks?
[+] nunez|1 year ago|reply
I agree. Ruby is a _fantastic_ language for getting things done quickly whose credibility was unfairly maligned by Rails.

Unbelievably easy to read, and, with rspec, it is stupid easy to write tests for. No need to fuss with interfaces like you do with Golang; yes, that is the right thing to do, but when you need to ship _now_, it becomes a pain and generates serious boilerplate quickly.

I've switched to Golang for most things these days, as it is a much safer language overall, but when shell scripts get too hard, Ruby's a great language to turn to.

[+] nomilk|1 year ago|reply
If you never coded in ruby before, but use macOS, you already have ruby installed. Just open terminal and type

  irb
to bring up the ruby interpreter and try out the code in the article.
[+] pipeline_peak|1 year ago|reply
Being able to call external commands with backticks alone makes it better suited than Python for shell scripting.
[+] bdcravens|1 year ago|reply
I spend most of my time writing Rails or other backend Ruby, and I prefer my system-level scripts in bash. Philosophically I don't want to have to manage dependencies or broken gems (though inline deps obviate that, and it's not like I've never had to wrestle with apt)
[+] lucasoshiro|1 year ago|reply
Well, but what you do with Bash can be also done with vanilla Ruby...
[+] manume|1 year ago|reply
What do you mean by "broken gems"?
[+] blahgeek|1 year ago|reply
Perl also satisfies all listed features
[+] Alifatisk|1 year ago|reply
Not surprising since Ruby was inspired by Perl.