Libc on macOS invokes Perl as a subprocess for string processing (2017)

[coldtea]

Well, FreeBSD does it by calling a sh builtin:

  		/*
  		 * We are the child; make /bin/sh expand `words'.
  		 */
  		(void)__libc_sigprocmask(SIG_SETMASK, 
  &oldsigblock, NULL);
  		if ((pdes[1] != STDOUT_FILENO ?
  		    _dup2(pdes[1], STDOUT_FILENO) :
  		    _fcntl(pdes[1], F_SETFD, 0)) < 0)
  			_exit(1);
  		if (_fcntl(pdesw[0], F_SETFD, 0) < 0)
  			_exit(1);
  		execl(_PATH_BSHELL, "sh", flags & WRDE_UNDEF ? "-u" : "+u",
  		    "-c", "IFS=$1;eval \"$2\";"
  		    "freebsd_wordexp -f \"$3\" ${4:+\"$4\"}",
  		    "",
  		    ifs != NULL ? ifs : " \t\n",
  		    flags & WRDE_SHOWERR ? "" : "exec 2>/dev/null",
  		    wfdstr,
  		    flags & WRDE_NOCMD ? "-p" : "",
  		    (char *)NULL);
  		_exit(1);
    	}

[self_awareness]

Seems Linux version also depends on forking to the shell:

https://code.woboq.org/userspace/glibc/posix/wordexp.c.html

[w0utert]

But why? Looking at the wordexp synopsis it doesn't seem like a particularly advanced function. Why don't the various libc's just implement this function directly in C?

[rurban]

The macOS libc is the FreeBSD libc. FreeBSD is upstream. FreeBSD changed it recently to use /bin/sh instead of /usr/bin/perl, because it's a bit smaller and faster.

[MichaelMoser123]

funny that glob doesn't do that http://xr.anadoxin.org/source/xref/macos-10.14-mojave/Libc-1... however wordexp does.

[fouronnes3]

Slightly OT, you know how people hated on C++ ranges for readability? Honestly this messy soup of C is no better.

[self_awareness]

Seems it's very old code. New Libc doesn't do this. This is Libc from Mojave:

http://xr.anadoxin.org/source/xref/macos-10.14-mojave/Libc-1...

The 'perl' code was a part of Libc v825.24, which seems to be included between 10.7 (Lion) and 10.8 (Mountain Lion).

Of course I still find it hilarious that even the old code did that!

[opencl]

The current version replaces the perl subprocess with an sh subprocess. Doesn't seem like much of an improvement.

[panic]

Discussed in 2015 (the code was out of date even then): https://news.ycombinator.com/item?id=9025572

[josteink]

That's obviously wrong.

It should use Emacs instead.

[protomyth]

I’m actually surprised there isn’t a library called libemacs. It would fulfill the mythos and be really useful for a lot of tools.

[fouronnes3]

Perl doesn't depend on libc?

[detaro]

It does, but that's "not a problem" as long as it doesn't use this function to implement something that's executed during this function.

[jslabovitz]

Turns out this is actually documented in the manpage for wordexp()! (And refers to the mentioned fact that it now calls 'sh' directly.)

> BUGS

> Do not pass untrusted user data to wordexp(), regardless of whether the WRDE_NOCMD flag is set. The wordexp() function attempts to detect input that would cause commands to be executed before passing it to the shell but it does not use the same parser so it may be fooled.

> The current wordexp() implementation does not recognize multibyte characters, since the shell (which it invokes to perform expansions) does not.

[senozhatsky]

Shall somebody send a pull request [0]

    -    /* XXX this is _not_ designed to be fast */
    +    /* XXX this is _not_ designed to be safe */

[0] https://github.com/Apple-FOSS-Mirror/Libc/blob/2ca2ae7464771...

[api]

/* This function computes the expansion rate of spacetime. This version contains an additional factor that causes it to accelerate to allow for rapid testing of other aspects of the physics engine. This MUST be removed before the production release or spacetime will accelerate forever and experience heat death instead of reaching steady state. -God */

[acura]

Isn't the line bellow that line enough? /* wordexp is also rife with security "challenges",

[MichaelMoser123]

Now I see why you can't change /usr/bin on macos. actually there is both perl5 and python2.7 in /usr/bin, libc does have a choice (that is if the tweet is true)...

https://en.wikipedia.org/wiki/System_Integrity_Protection

[olliej]

You can’t change use/bin because that’s a common malware attack vector.

It also has the nice effect of forcing user installed utilities to install in the /local/ variants (which user build projects should be doing on Linux iirc), so an OS update doesn’t overwrite user data.

[fit2rule]

I find it hard to believe that there's any software out there that doesn't, eventually, invoke Perl as a subprocess .. I mean, its Perl.

[leejo]

That's essentially what the OG tweet is saying: Pinnacle of software development: you can solve the problem with three lines of Perl, but you don’t, because of a non-argument against Perl. Since there didn't used to be that many arguments against perl/Perl it worked its way into a lot of systems even if it wasn't actually implementing the system.

Of course Perl, having fallen out of vogue, probably wouldn't be used today but it used to be everywhere so its footprint is still pretty large.

Also - I can't help but see the irony in shelling out to perl given experienced Perl developers always tell the less experienced ones to avoid shelling out from Perl if possible and to only do that as a last resort if there isn't an existing library to solve the problem.

[the_mitsuhiko]

Not exactly related to the link but apparently the author of that tweet blocked me. I don’t recall having ever had any interactions with them. Is there a way to contact that person and figure out why? I have no idea what I did and I’m quite puzzled.

[woodman]

There is a good chance they subscribe to a blocklist, so you could be blocked by anyone of a thousand people. Image the old PGP web of trust, but for crafting perfect echo chambers. I wonder if anybody has ever done the math on that.

[stonogo]

You find out someone has blocked you, and your instinct is to communicate with them? The specific thing they have explicitly disallowed? I'd reconsider this, and just move on.

[acura]

Not good looking at all but, latest commit to that repo is Updated on Oct 11, 2012.

So how well does it reflect reality?

[raimue]

The official sources are published by Apple on https://opensource.apple.com

This repository is just a snapshot that somebody else prepared and uploaded to GitHub, but apparently it is not maintained.