I believe that scripts without a shebang line cannot be executed by execve. They will only run if the shell sees them before the kernel does. It's an annoying limitation of this approach and it wasn't clear to me whether this technique includes a workaround for this. Also the push for code signing everywhere is going to make this harder in the future as well. (I heard a rumor that the workaround to run unsigned executables on macOS no longer works when running on Apple Silicon.) Otherwise, I love it!
My approach was slightly different; I created a polyglot script header you can paste at the beginning of a C source file to make it essentially a C script which is just-in-time compiled when it is executed. Works on Windows, Linux, and Mac as long as a C compiler is installed. https://gist.github.com/jdarpinian/6860ddfd92b5b458a20ab6055...
Author here. I considered doing that. Then I learned about things like how MSVC 2017 would wrap the main() function of C programs with telemetry. The push for code signing, retpoline, aslr, etc. worries me too, from a software freedom standpoint, and the same goes for how difficult it's become to build native software from scratch, thanks to the shift from autotools to a whole new set of competitors like bazel, cmake, ninja, gin, etc. Secure boot is another red flag. By 2018 it almost felt like we were entering a world in which, for all practical purposes, we couldn't share native software easily in either source or binary forms. If PC platforms ever do manage to get consumers to accept the iPhone App Store restriction model, it'll likely stay that way. Best way to prevent that from happening is to use Actually Portable Executable. The more people who continue to depend directly on canonical long-standing stock platform abis, the more impossible it becomes for platforms to re-imagine them every few years. I'll take UNIX+WIN32 over things like XUL any day, since I don't want to fall into the JS dev trap of having to rewrite my app with each passing moon.
Also note that the execve() shebang thing is a one-time cost. The printf statement in the header will overwrite the first 64-bytes of the executable, the first time it's invoked, so it becomes a canonical ELF or Mach-O executable for subsequent invocations. The one-time cost is also cheap. The shell script only needs three system calls (open, write, and execve) to patch the binary appropriately. The generation of this header is also abstracted by the GNU ld script, which made it easy for me to encode the ELF header relocations as octal printf codes!
> I believe that scripts without a shebang line cannot be executed by execve.
This was once true on old systems, but current ones (including Linux) have native support for scripts starting with #!. See fs/binfmt_script.c in Linux, which has existed for several decades.
On my Big Sur machine, `bash ./hello.com` works the first time, but then it overwrites itself(!) with a Mach-O, so further executions have to be `./hello.com`.
Apparently people on Stack Exchange have been working together on a single polyglot program for four years and have gotten a single codebase up to validity in nearly 300 languages.
>Apparently people on Stack Exchange have been working together on a single polyglot program for four years and have gotten a single codebase up to validity in nearly 300 languages.
Wow, what a link. I submitted it just now as a separate HN post using your phrasing:
> the x86_64 patents should expire this year. Apple could have probably made their own x86 chip without paying royalties. The free/open architecture that we've always dreamed of, might turn out to be the one we're already using.
I've never seen anyone mention this before. Is this really in the cards? Or are all the extensions patented as well, so that a hypothetical Apple (or whoever else takes it on) x86 wouldn't be able to use x86 without licensing it?
Can someone eli5 how this works? I sort of get that you can create one big binary file with all the necessary file headers for each platform and the actual x86 code being shared, but what about stuff like syscalls? The post also lost me when it started mentioning the need for shell scripts and emulators.
Back when all you had was minicom dialing a BBS, there was a chicken and egg problem where you needed uudecode to decode any executable you downloaded. If you did not have uudecode you were stuck. If you are unfamiliar, unencoding a file allows you to send binary data through text transmission.
Then a clever version of a DOS .COM file was posted which implemented uudecode, but it only used x86 instructions that were also ASCII characters. You could copy/paste between the --cut here-- lines into a file, save it as uudecode.com, and then get your other binaries like pkzip.
Great memories you brought back. Minicom times were so simple, the BBS sysop dialed upr2.clu.net for us so we had a nice UUCP connection and got our mails pushed out.
I had to undergo the same chicken egg situation with uudecode
> I believe the best chance we have of doing that, is by gluing together the binary interfaces that've already achieved a decades-long consensus, and ignoring the APIs.
Yeah, this isn't going to fly for OpenBSD, which doesn't have ABI stability, programs are expected to use libc. Program for the API, not the ABI. The author is clever, but misguided.
It's a very cool hack and technically interesting, but I can't say that I share the author's "serious" rationale:
>So I think it's really the best of times to be optimistic about systems engineering. We agree more on sharing things in common than we ever have. There are still outliers like the plans coming out of Apple and Microsoft we hear about in the news, where they've sought to pivot PCs towards ARM.
[...]
>If a microprocessor architecture consensus finally exists, then I believe we should be focusing on building better tools that help software developers benefit from it
Considering ARM an outlier in a world were smartphones outnumber desktop PCs and where the x86 stronghold is under attack from several sides seems like a strange take to me.
Also, there's more to portability than managing to run a hello world from the same binary on Windows and Linux. I used to use FreeBSD a lot on the desktop, which usually involved using Linux binary compatibility to run Linux closed source binaries that weren't available on FreeBSD. It worked generally pretty well, but it wasn't without friction at times even though it was coded by very smart people and between two systems that were overall very similar.
These "compatibility" applications often end up as 2nd class citizen that don't integrate well with native applications, or at least not as easily. See Wine for instance, managing to run Windows applications on un*x is a huge undertaking.
>One of the reasons why I love working with a lot of these old unsexy technologies, is that I want any software work I'm involved in to stand the test of time with minimal toil. Similar to how the Super Mario Bros ROM has managed to survive all these years without needing a GitHub issue tracker.
I love emulation, so I definitely understand where this is coming from, but I don't really see what that has to do with the rest. In particular if you're willing to go down the emulation route, why do you care about everybody running x86? Super Mario Bros was not written to run on a Celeron. Besides CPU emulation for something like a NES video game is only a tiny part of the story, you also have to emulate the GPU, sound chip etc...
I feel like there's a mishmash of interesting ideas in there, but it's a bit confusing to understand.
Author here. The intention behind the post is actually very simple. I want to be able to write stdio socket programs that run at native speed, and share them with my friends online. I don't want to have conversations like here's how you install steel bank common lisp and compile tensorflow and remove the toolbar that oracle bundled in the installer. I'm also not concerned about running software on telephones since doing so requires obtaining authorization from Google and Apple beforehand.
Before I built Actually Portable Executable, PCs behaved de facto similar to mobile development restrictions, where for example if you want to build for Apple you download XCode, and if you want to build for Windows you have to download MSVC. But it thankfully has never been a hard requirement for creating first class native programs. It simply required the will to ignore official tooling and encode binaries the hard way that directly talk to canonical stock kernel abis, so that we don't need things like jenkins to compile release binaries n times for the same microprocessor architecture.
I had enough free time to do it. I've also shared a tool that makes it easier for everyone else to do it too. Enjoy!
> I couldn't resist the temptation of making it a reality, since it means that high-performance native code can be almost as pain-free as web apps.
> Please note this is intended for people who don't care about desktop GUIs, and just want stdio and sockets without devops toil.
My understanding was that people are using web-apps and electron precisely to provide cross-platform GUIs. I feel like this is really missing the forest for the mountains...
Is hosting web URLs at storage.googleapis.com a thing? After finding a persistent phishing campaign using it, I ended up restricting it from being loaded in the web browser in my environment. (Embedded images from it, like on blog.google, still work.)
This is the first time I've seen a legitimate website with one of these URLs.
- If you want to be able to distribute binaries in binary only form, then please send $1,000 to Justine Tunney [email protected] on PayPal, for a license lasting 1 year.
If the goal of this project is to have something similar to cross platform web development this is the wrong way to go about it imo.
Author here. I'm sorry you feel that way. If you'd rather it be licensed ISC/MIT/BSD/etc. then please encourage your favorite member of FAANG or some other huge company to acquire Actually Portable Executable and then make it freely available under those terms, for the sake of benevolence and commitment to serving public interest. I come included.
It's limited to a feature-poor "unified" API (that might as well be seen as a crippled VM).
It's complex in both interface and implementation. The bad aspects of Unix made even worse.
The justifications make no sense whatsoever. It wants to do future-proofing but invents an adhoc "VM" with hardcoded OS interfaces that are meant to be treated as private. For macOS, I doubt binaries will work in the next release or two, nevermind 10 years.
It's a fun toy or troll, but if you're looking at this for serious work I suggest you steer well clear.
> If a microprocessor architecture consensus finally exists
I don't think that's a good goal to have. Without competing architectures we'll enter a microprocessor dark age. Although the glory days of Alpha and IA64 are over, plenty of that experience was used to bolster today's instruction sets. Diversity is a good thing.
Microarchitectures are the new architectures. The game these days is all about finding a way to make your code go fast on the latest and greatest AVX thing, while still maintaining compatibility for old machines via CPUID. Getting those 10x speedups (sometimes 50x w/ things like crc32 pclmul) is much more fun than caring about things like s390x (i.e. doing IBM's work for them), SH (i.e. folks emulating Sega Saturn on FPGAs for fun), MIPS (hacked home routers), etc.
I'm not sure I buy that, you can have a lot of competition on the same ISA. After all on the desktop/server x86 has been king for a while, and on embedded architectures ARM has all but taken over the competition. Yet there's significant competition on both fronts.
Meanwhile attempts at creating new ISAs over the past couple of decades has been met with relatively little success, or only in very specific niches. Itanium being an obvious example. People talk about RISC V a lot but in practice it's not really making a dent into the ARM market share yet.
These days competition appears to come mostly from companies reimplementing and extending existing ISAs with better performance.
I think the closest to that would be https://en.wikipedia.org/wiki/Shar and I've seen ones that were both valid shell scripts and Windows batch files. Of course, the actual polyglot part is tiny and serves only to direct execution to continue with the appropriate platform-specific binary.
Finally, you can produce a single cross-platform executable in C.
> we've basically reconfigured the stock compiler on Linux so it outputs binaries that'll run on MacOS, Windows, FreeBSD, OpenBSD too. They also boot on bare metal
What's the challenges involved in making a actually portable executable that is a compiler that can create actually portable executables. Why muck with getting a gcc running on windows or whatever if you don't have to? If you only mess with c once a year or so, getting compilers set up is a pain.
Right now I'm solving the problem by only doing my dev work on Linux. That works fine for me, since the output binaries run on other platforms too. But I'd love to support more flexibility for more people! Not everyone is accustomed to doing their coding via an SSH terminal. So the challenge is getting more eyeballs on the Cosmo source code so we can add features and iron out the long tail of bugs.
Earlier this year, I somehow managed to successfully compile an x86_64-linux-gnu toolchain under MinGW. Using that, pretty much 90% of the Cosmopolitan built fine under Windows without needing a Linux VM (which actually goes faster than native WIN32 due to how Make+GCC assumes processes work). I've tried and failed several times to compile GCC from source on Mac.
But I think compiling GCC is just so hard, that I'd probably rather embed the GNU/Linux compiler binaries in the PKZIP embedded file structure of EMULATOR.COM, since that should create a GPL boundary that's kosher per the GPLv3 and GCC RTEv3 terms.
It's funny. In China there used to be a time when high school students like to use rarely used Chinese characters, which are only similar to the intentional character in shape instead of meaning.
To be clear, "intentional" should be "intended" right? Had to read this a few times (I'm also a non-native speaker, I wonder if that makes it easier or harder for me to understand a text with meaning-altering typos/mistakes) but I think it must be this.
Interesting point about the x86-64 patents expiring and the possibility of third parties making chips for that instruction set. Is it just so big and clunky that the energy is with ARM / RISC-V?
[+] [-] modeless|5 years ago|reply
My approach was slightly different; I created a polyglot script header you can paste at the beginning of a C source file to make it essentially a C script which is just-in-time compiled when it is executed. Works on Windows, Linux, and Mac as long as a C compiler is installed. https://gist.github.com/jdarpinian/6860ddfd92b5b458a20ab6055...
[+] [-] jart|5 years ago|reply
Also note that the execve() shebang thing is a one-time cost. The printf statement in the header will overwrite the first 64-bytes of the executable, the first time it's invoked, so it becomes a canonical ELF or Mach-O executable for subsequent invocations. The one-time cost is also cheap. The shell script only needs three system calls (open, write, and execve) to patch the binary appropriately. The generation of this header is also abstracted by the GNU ld script, which made it easy for me to encode the ELF header relocations as octal printf codes!
[+] [-] ChrisSD|5 years ago|reply
It's a great idea but that part is a big ask for any Windows user who doesn't develop desktop applications.
[+] [-] saagarjha|5 years ago|reply
Was there ever a workaround for this? I thought you just ran them…
[+] [-] JoshTriplett|5 years ago|reply
This was once true on old systems, but current ones (including Linux) have native support for scripts starting with #!. See fs/binfmt_script.c in Linux, which has existed for several decades.
[+] [-] saagarjha|5 years ago|reply
[+] [-] comex|5 years ago|reply
[+] [-] schoen|5 years ago|reply
https://en.wikipedia.org/wiki/Polyglot_(computing)
Apparently people on Stack Exchange have been working together on a single polyglot program for four years and have gotten a single codebase up to validity in nearly 300 languages.
https://codegolf.stackexchange.com/questions/102370/add-a-la...
Edit: also, Yusuke Endoh is a master at this art form
https://github.com/mame/quine-relay
and has even written a book about it
http://uguu.org/words/2015_10_01.html
which I would still love to have available in English!
[+] [-] chrisdalke|5 years ago|reply
A quine that prints itself as scrolling text in an AVI file: https://www.youtube.com/watch?v=rMlJy5He-Tk
A self-similar quine that behaves like a fractal zooming out infinitely: https://www.youtube.com/watch?v=6J7u2G52scc
A quine with source code that contains sheet music to a Bach song in comments and plays it: https://www.youtube.com/watch?v=OryXKUgRVOg
[+] [-] logicallee|5 years ago|reply
Wow, what a link. I submitted it just now as a separate HN post using your phrasing:
https://news.ycombinator.com/item?id=24257630
[+] [-] jaclaz|5 years ago|reply
PoC||GTFO International Journal of Proof-of-Concept or ... https://www.alchemistowl.org/pocorgtfo/
Corkami: https://github.com/corkami
[+] [-] donaldcuckman|5 years ago|reply
[deleted]
[+] [-] celrod|5 years ago|reply
I've never seen anyone mention this before. Is this really in the cards? Or are all the extensions patented as well, so that a hypothetical Apple (or whoever else takes it on) x86 wouldn't be able to use x86 without licensing it?
[+] [-] krackers|5 years ago|reply
[+] [-] voltagex_|5 years ago|reply
Well yeah, but that might be more a function of it existing before git and github.
https://www.mariowiki.com/List_of_Super_Mario_Bros._glitches
[+] [-] snarfy|5 years ago|reply
Then a clever version of a DOS .COM file was posted which implemented uudecode, but it only used x86 instructions that were also ASCII characters. You could copy/paste between the --cut here-- lines into a file, save it as uudecode.com, and then get your other binaries like pkzip.
[+] [-] userbinator|5 years ago|reply
That was a somewhat common approach back then. It's hard to find references to that technique now, but here's something I did find: https://news.ycombinator.com/item?id=16312562
[+] [-] latchkey|5 years ago|reply
[+] [-] InafuSabi|5 years ago|reply
I had to undergo the same chicken egg situation with uudecode
[+] [-] lostmyoldone|5 years ago|reply
[+] [-] notaplumber|5 years ago|reply
Yeah, this isn't going to fly for OpenBSD, which doesn't have ABI stability, programs are expected to use libc. Program for the API, not the ABI. The author is clever, but misguided.
[+] [-] simias|5 years ago|reply
>So I think it's really the best of times to be optimistic about systems engineering. We agree more on sharing things in common than we ever have. There are still outliers like the plans coming out of Apple and Microsoft we hear about in the news, where they've sought to pivot PCs towards ARM.
[...]
>If a microprocessor architecture consensus finally exists, then I believe we should be focusing on building better tools that help software developers benefit from it
Considering ARM an outlier in a world were smartphones outnumber desktop PCs and where the x86 stronghold is under attack from several sides seems like a strange take to me.
Also, there's more to portability than managing to run a hello world from the same binary on Windows and Linux. I used to use FreeBSD a lot on the desktop, which usually involved using Linux binary compatibility to run Linux closed source binaries that weren't available on FreeBSD. It worked generally pretty well, but it wasn't without friction at times even though it was coded by very smart people and between two systems that were overall very similar.
These "compatibility" applications often end up as 2nd class citizen that don't integrate well with native applications, or at least not as easily. See Wine for instance, managing to run Windows applications on un*x is a huge undertaking.
>One of the reasons why I love working with a lot of these old unsexy technologies, is that I want any software work I'm involved in to stand the test of time with minimal toil. Similar to how the Super Mario Bros ROM has managed to survive all these years without needing a GitHub issue tracker.
I love emulation, so I definitely understand where this is coming from, but I don't really see what that has to do with the rest. In particular if you're willing to go down the emulation route, why do you care about everybody running x86? Super Mario Bros was not written to run on a Celeron. Besides CPU emulation for something like a NES video game is only a tiny part of the story, you also have to emulate the GPU, sound chip etc...
I feel like there's a mishmash of interesting ideas in there, but it's a bit confusing to understand.
[+] [-] jart|5 years ago|reply
Before I built Actually Portable Executable, PCs behaved de facto similar to mobile development restrictions, where for example if you want to build for Apple you download XCode, and if you want to build for Windows you have to download MSVC. But it thankfully has never been a hard requirement for creating first class native programs. It simply required the will to ignore official tooling and encode binaries the hard way that directly talk to canonical stock kernel abis, so that we don't need things like jenkins to compile release binaries n times for the same microprocessor architecture.
I had enough free time to do it. I've also shared a tool that makes it easier for everyone else to do it too. Enjoy!
[+] [-] jdmichal|5 years ago|reply
> I couldn't resist the temptation of making it a reality, since it means that high-performance native code can be almost as pain-free as web apps.
> Please note this is intended for people who don't care about desktop GUIs, and just want stdio and sockets without devops toil.
My understanding was that people are using web-apps and electron precisely to provide cross-platform GUIs. I feel like this is really missing the forest for the mountains...
[+] [-] zelly|5 years ago|reply
[+] [-] ocdtrekkie|5 years ago|reply
This is the first time I've seen a legitimate website with one of these URLs.
[+] [-] sdan|5 years ago|reply
[+] [-] renewiltord|5 years ago|reply
[+] [-] kokooo|5 years ago|reply
- License is GPL2
- If you want to be able to distribute binaries in binary only form, then please send $1,000 to Justine Tunney [email protected] on PayPal, for a license lasting 1 year.
If the goal of this project is to have something similar to cross platform web development this is the wrong way to go about it imo.
[+] [-] jart|5 years ago|reply
[+] [-] pjmlp|5 years ago|reply
Long term we will be back to the shareware days and we will have the anti-GPL task force to thank for.
[+] [-] gargalatas|5 years ago|reply
So I read this like: actmally pthrtable execmtable.
[+] [-] IgorPartola|5 years ago|reply
[+] [-] armitron|5 years ago|reply
It's incompatible with modern debuggers.
It's incompatible with shared libraries.
It's limited to a feature-poor "unified" API (that might as well be seen as a crippled VM).
It's complex in both interface and implementation. The bad aspects of Unix made even worse.
The justifications make no sense whatsoever. It wants to do future-proofing but invents an adhoc "VM" with hardcoded OS interfaces that are meant to be treated as private. For macOS, I doubt binaries will work in the next release or two, nevermind 10 years.
It's a fun toy or troll, but if you're looking at this for serious work I suggest you steer well clear.
[+] [-] als0|5 years ago|reply
I don't think that's a good goal to have. Without competing architectures we'll enter a microprocessor dark age. Although the glory days of Alpha and IA64 are over, plenty of that experience was used to bolster today's instruction sets. Diversity is a good thing.
[+] [-] jart|5 years ago|reply
[+] [-] simias|5 years ago|reply
Meanwhile attempts at creating new ISAs over the past couple of decades has been met with relatively little success, or only in very specific niches. Itanium being an obvious example. People talk about RISC V a lot but in practice it's not really making a dent into the ARM market share yet.
These days competition appears to come mostly from companies reimplementing and extending existing ISAs with better performance.
[+] [-] Rebelgecko|5 years ago|reply
[+] [-] userbinator|5 years ago|reply
[+] [-] naikrovek|5 years ago|reply
"Need cred or accessibility? Nerd cred! Screw you if your eyes don't work!"
[+] [-] stavros|5 years ago|reply
[+] [-] fouc|5 years ago|reply
> we've basically reconfigured the stock compiler on Linux so it outputs binaries that'll run on MacOS, Windows, FreeBSD, OpenBSD too. They also boot on bare metal
[+] [-] edgyquant|5 years ago|reply
[+] [-] sagebird|5 years ago|reply
[+] [-] jart|5 years ago|reply
Earlier this year, I somehow managed to successfully compile an x86_64-linux-gnu toolchain under MinGW. Using that, pretty much 90% of the Cosmopolitan built fine under Windows without needing a Linux VM (which actually goes faster than native WIN32 due to how Make+GCC assumes processes work). I've tried and failed several times to compile GCC from source on Mac.
But I think compiling GCC is just so hard, that I'd probably rather embed the GNU/Linux compiler binaries in the PKZIP embedded file structure of EMULATOR.COM, since that should create a GPL boundary that's kosher per the GPLv3 and GCC RTEv3 terms.
[+] [-] dracodoc|5 years ago|reply
They think that's cool. This style was called brain damaged style. https://zhidao.baidu.com/question/63502990
什庅bai湜焱暒妏？举些瑺鼡哋唎ふ。du 什庅湜悩残軆？举些瑺鼡哋唎ふ。 什庅湜zhi悱炷蓅？举些瑺鼡哋唎ふ。 莓兲想埝祢巳宬儰⒈种漝惯
[+] [-] Aachen|5 years ago|reply
[+] [-] aasasd|5 years ago|reply
[+] [-] angrygoat|5 years ago|reply
[+] [-] ekianjo|5 years ago|reply