WingNews logo WingNews
top | item 23193952

Polyglot Makefiles

153 points| sciencerobot | 5 years ago |agdr.org

43 comments

order

brandmeyer|5 years ago

That's interesting.

Author probably wants to use `private` for those target-local variables, though.

For example,

    R: .SHELLFLAGS := -e
    R: SHELL := Rscript
    R:
        greeting = "bonjour"
        message(paste0(greeting, ", R!"))
Everything that target `R` depends on will also have SHELL and .SHELLFLAGS over-ridden. If `R` depends on some data generated by another program, it probably wants to be built and executed with the default SHELL (or another shell, perhaps).

    R: private .SHELLFLAGS := -e
    R: private SHELL := Rscript
    R:
        greeting = "bonjour"
        message(paste0(greeting, ", R!"))
Now, `R`'s dependencies will be generated with the makefile's defaults.

Usually I prefer to build up richer stages like this using the system shell anyway, though. Build a target which in turn is executed by the shell normally to traverse the next edge in the graph. But I can see how this mechanism has its uses.

See also https://www.gnu.org/software/make/manual/html_node/Target_00...

sciencerobot|5 years ago

I didn't know about private. Thanks for the tip.

gregwebs|5 years ago

Make was designed for building dependencies. I think it is always problematic to use it as a command runner (for example there is no standard way to list out the available commands).

[just](https://github.com/casey/just) is a tool that feels similar to make but is designed explicitly for the purpose of running commands.

I think of this as a simple CLI for your project workflow. You still really want to avoid putting code into a Justfile and put it into scripts. But the Justfile helps provide a slightly nicer UX and automatically invoke dependencies.

chubot|5 years ago

Yes I agree with this. I use shell instead of make, because make wrapps shell and its syntax collides very poorly with it. For example, the PID is now $$$$ and not $$.

Most people forget to mark their targets .PHONY, so they have a subtle bug in their build (touch build; touch test).

----

But shell also suffers from the problem where it doesn't list the commands. I filed a bug for Oil shell here:

https://github.com/oilshell/oil/issues/751

I mentioned a couple other "frameworks" there like just, go, Taskfile, etc.

But it should really just be built into the shell, since it's so common. And there should be command completion too, which I think bash-completion has for Makefiles on many distros.

Apparently there is no standard name for this kind of "task runner". But I think shell makes a lot more sense than a custom format, because there are many instances where you need a simple loop or conditional. It scales better. (And Oil also fixes bad shell syntax while remaining compatible: http://www.oilshell.org/blog/2020/01/simplest-explanation.ht...)

If anyone wants to help let me know :) The code is plain Python and pretty hackable. However it generates fast C++, so you get the best of both worlds (in progress)

dima55|5 years ago

The tool you want is remake: http://bashdb.sourceforge.net/remake/

This is GNU Make + a few patches. So it's 100% compatible. And you get an interactive debugger, and lots more stuff. For instance, to list out the commands:

  remake --targets
No idea why this hasn't been merged upstream.

Your larger point really stands, though: if you're just running commands, you shouldn't be using Make. But it is abused in that way often, so...

sciencerobot|5 years ago

I didn't generate any files in the examples for simplicity. But you could imagine a workflow where Python generates some data and then you use R to plot it + run some statistical tool.

epistasis|5 years ago

For complicated pipelines that I want to reuse multiple times, I have turned Makefiles into executables by putting this at the top:

    #!/usr/bin/make -f
And then putting them in my $PATH. I run them with arguments like:

    $ process-data.mk INTSV=a.tsv DB=largefile.gz OUTDIR=finished
This makes me feel like I've sold my soul to the devil, and that I'm just living on borrowed time until it all fails and falls apart. It hasn't yet, however...

wahern|5 years ago

-f is guaranteed by POSIX, and #! is de facto portable.[1] My criteria for shame is, "will this silently break in the future?". I think you're good. It's not my style, but if it were something I came across at work, so long as it worked well it wouldn't even cross my mind to try to "fix" it.

FWIW, using make -f in the shebang is also done for debian/rules in Debian package builds. I don't know if it serves any real purpose. I suppose it permits one to write a bespoke script for building targets without using make.[2] I guess I wouldn't be surprised if someone, somewhere depended on that capability, given how old and widespread Debian packages are.

[1] /usr/bin/env make -f would be better, but then you run afoul of the problem that you can't portably pass more than a single explicit shebang command argument.

[2] Which I see now is a bonus to your process-data.mk script. It could be replaced with a non-make version without effecting callers.

asveikau|5 years ago

I can see two areas in which it might break.

I might put #!/usr/bin/env make -f in case it's somewhere in else in PATH.

Also some systems (BSD, old commercial Unix) have non-gnu-compatible make and sometimes call their gnu make port "gmake" or "gnumake".

haolez|5 years ago

In your scenario, "make processed_data" makes more sense semantically than rules commonly seem in the wild like "make run" or "make deploy"

ainar-g|5 years ago

Note that this article (like many, many others) assumes GNU Make. POSIX Make has neither .ONESHELL nor local macros. Neither do most built-in Make implementations in other OSes, like OpenBSD's bmake.

ddevault|5 years ago

POSIX shell is also notoriously obtuse and difficult to use. As a big advocate of POSIX as a target, I don't blame anyone for using GNU make - or perhaps BSD make is a better lowest common denominator.

Personally, I try to use POSIX Makefiles, but I often find that they're most useful as a target for Makefile generators (in my case, these are usually a shell script called configure).

wahern|5 years ago

bmake is the implementation of make in NetBSD and FreeBSD. OpenBSD dropped bmake a long time ago and wrote their own implementation. OpenBSD make doesn't support ONESHELL, either, though.

IshKebab|5 years ago

I wish someone would write a modern alternative to GNU Make. I've looked and there don't seem to be any. The closest is Ninja but it doesn't seem to be intended to be hand written.

theshrike79|5 years ago

There are a lot of options, but make is just everywhere.

Sometimes it's just simpler to bite the ancient bullet and go with a Makefile, with all its included pains and gotchas rather than try to figure out how to get the fancy new makefile replacement installed in all the relevant environments.

epistasis|5 years ago

A lot of people in bioinformatics use SnakeMake. In this field you often want to restart analysis after something changes somewhere along a pipeline (for example the pipeline is under active development and changing frequently), and individual steps can take hours or more, so automatically rerunning just the right stuff is a great feature.

However, SnakeMake, Nextflow, etc feels excessively verbose compared to standard make. And the prior workflow managers of last decades were far worse. With standard make, you type pretty much exactly what you would for shell commands, and not too much more.

All other alternatives are going to be more verbose than make, and to me that's a negative.

mehrdadn|5 years ago

I think this might have repercussions? Like if you do "make foo bash bar" then can you predict what that SHELL is used for?

sciencerobot|5 years ago

The shell is specific to each target. So doing `make ruby bash python docker` works. It even works in parallel if you do `make -j`.

Edit: I'm the author.