Recently I enjoyed some "retrofuturistic" development with WASM and CGI. Spin[1], a webserver written in Rust, can execute WASI[2] binaries that speak CGI. You can then deploy it to Fermyon Cloud or your own server and it "just works". It's a wonderful mix of old and new. I used it for PHP (Prolog Home Page): https://github.com/guregu/php
[2]: WASI is a POSIX-ish standard for WASM that gives you all the low level stuff like standard input and output. It includes all the bits and pieces needed for CGI to work.
Kind of feels like the AssemblyScript guys were right - WASI is becoming a standard for WASM modules that foists Unix on Webassembly, including stuff that isn't really portable like symlinks.
That makes sense if you want to compile existing software to WASM but that isn't the case here is it?
CGI was abandoned due to performance problems, but it is surprising scalable on modern OSs, the kernel does a good job of caching the code so it's not really a full round trip to the file system and cold start on each request. Obviously you still have the overhead of an interpreter for many languages, but they also tend to do a fairly good job of caching their byte code. I absolutely wouldn't recommend it for new systems, but for a quick and dirty script it's fine.
The minute you want to do url routing and have middleware, a persistent process makes everything much simpler and keeps everything in one place. I much prefer routing inside a framework than using Mod_Rewrite.
> a persistent process makes everything much simpler ...
I started with PHP and never trusted persistent processes for anything.
What if your persistent process hangs or crashes for whatever reason?
Is your whole site down then, not just the part with the bug?
The only site I ever made in persistent process paradigm was in ASP.NET and it didn't cause too many problems but it was more of an app, not website, with limited number of logged in users and pretty much no anonymous/guest part.
I'm having fun with django + htmx and going to deploy on elastic beanstalk.
It hits that sweetspot of speed of development and responsiveness. You also have the benefit of staying in one language.
Django has so much out of the box that you don't have to decide for yourself after the fact. So much accumulated knowledge of what works too.
Sure go ahead rewrite it if it starts making money and you can afford development time/salary. But these older technologies are easier to fit inside one head and let you get enough speed to take off.
This is also a very sane combination. HTMX is great and Django is also very impressive :) We need tools that work for us not us working to make them work.
I wrote Eksi Sozluk, the most popular Turkish social platform to date, back in 1999 using Delphi as CGI executables[1]. Delphi was the tool that I knew best, and CGI worked for me at the beginning. I almost immediately started having problems with EXE files being constantly locked, making updates impossible (as it was Windows based), which required restarting the server for any kind of update.
CGI executables also caused problems when I switched to an Alpha AXP RISC server which emulated x86, bringing the performance to a crawl. That made me switch to classic ASP, and about 10 years later, to ASP.NET MVC with routing, unit tests, abstractions, jQuery, all the shiny things at the time.
Web didn't stop there of course; there came SPA's, React, Vue and whatnot.
Now, seeing the yearning for CGI in 2023 feels funny. Have we come full circle? :)
> Zero external configuration, other than telling your webserver to enable CGI on your file
This is only true if you've done it before, and know what you're doing. In reality, it looks like a mess of `mod_cgi` configuration, trying different combinations of file permissions, finding the magic `cgi_bin` directory, finding the right obscure log files when there are inevitably errors, wrestling with CORS and other subtleties of HTTP headers, and other complexities that are only easy to navigate if you're already an experienced CGI user.
That being said, I love the philosophy of using CGI for scripts. Instead of using CGI itself, though, I wrote a (single-file, statically-linked) web server called "QuickServ" to bring this philosophy into the twenty-first century. It has all of the upside of CGI, but is much easier to set up and run, especially for beginners.
One of its benefits is that it automatically parses submitted HTTP forms, and converts the form fields to command line arguments. That means it's extremely easy to put existing CLIs on the web with minimal changes other than writing an HTML form front-end.
If you like CGI, I will (shamelessly) ask that you check it out!
I thought about that for few of the personal stuff but in the end I just opted to have a binary that's just http router + abstracted away way to handle requests. So when I need new feature I just add a module and deploy whole (that includes static files) blob. No need to create new service, add it to loadbalancer etc.
I do contract work for a company that has it's pretty customer facing site as a mix of angular/react/react native and .net core apis and the less pretty internal staff site which is vanilla html, js and c++ cgi apis, so I get to work in both on the same day. The internal c++ site has been running for ~25 years. The customer site seems to need to be re-written every time we add a new feature, because all the tech is out of support and there are breaking changes to libraries
I started writing my dissertation project in Perl CGI back in like 2001 or whatever it was. I moved to PHP pretty quickly when I realised I could do the same things in 5 lines that took 70 in Perl. I'd been making websites for about 6 years at that point and PHP just absolutely changed my life. It was, arguably, exactly like this post talks about CGI - put some script in a file, FTP to your host and you're good to go. Web development was simple and exciting - you could get a long way very quickly.
I was talking to a friend the other day who wants to learn some web dev for fun - he had no clue where to start because there's now SO much, so many 'best practice' posts, so much 'you must learn like this'. I know the world changes (and often for the better), but it was a magic time when you could get so far so quickly.
As an aside, it's why I have huge amounts of respect for people like Pieter Levels and his 'just get it done' approach to building things.
To me PHP is still a great tool to do these sort of stuff. Not to do a whole application, it was not what it was designed for, but for simple stuff it's great!
And it's improved a lot PHP, nowadays it's not that bad language...
A lot of that goes away if you avoid Javascript/Node as your first ecosystem. The unnecessary complexity and cargo-culting there is off the charts. It's overwhelming even for the experienced engineers.
I got so jaded dealing with this JS nonsense just to get a simple front-end going for my pet project that I just said "screw it" and started using straight up ES6 in the browser. No libraries, no transpilers, no minifiers.
Perhaps we should start making tutorials that follow that idea from PHP: start with HTML, add an extension (e.g. .eex, .erb) and make it dynamic. This would lower the barrier for beginners.
By the way, the best HTML tutorial I found so far is Scrimba. The idea of having interactive videos where you can pause and edit the code is amazing.
I wrote Perl cgi scripts for a self-serve customer admin interface for an ISP in the early 2000s. As users grew, we started hitting the limits of what 1-2G ram could do. Thank goodness for mod_perl and HTML::Mason, which could REALLY speed up performance. I remain a huge fan of Perl these days.
I used Perl's CGI::Application to transform an Excel spreadsheet of teachers, used for fielding telephone enquiries, into a PostgreSQL-backed web service used by 33 household brands. Everything ran on Apache CGI without a hitch for 15 years.
I built a bunch of "contact us" pages for folks over the years and always used python with the built in CGI library. The python folks thought so little of CGI that they ripped it out of the standard library. I wish I had just used Perl like god had intended CGI scripts to be written.
On that note, I recently revived some nearly 20 year old perl CGI stuff. Had some minor syntax issues (language ambiguities which got fixed), but it was running just fine after having fixed these. There was also some C code with it (some tooling), needed header include fixing + function renaming, otherwise still worked. Small project all in all, not reliant on a lot of external things.
Recently I built a CGI script using Perl for my own pleasure. It was quite easy indeed, just a case of importing Template Toolkit, GD and CGI.pm then printing the result to stdout. Since I'm only using CGI.pm to send the HTTP header it isn't really necessary.
I always felt that the best software should be designed like an onion. Very easy to get started (no need to peel too much to get to the outer layer), but there should always be the possibility to go deeper and peel more layers. Easy to get started is not great (as a general purpose tool) if there is no decent upgrade path.
What replaced CGI scripts is slightly more complex, but also more naturally grows to other use-cases.
CGI was also a security nightmare full of footguns. Not that modern software isn't, but CGI was especially bad.
- Take untrusted input and pass it directly to shell scripts on your filesystem. What could go wrong?
- Executables in cgi-bin were nearly always included in your document root on shared hosting servers.
- File drops and web shells were rampant
- A trivial misconfiguration (not setting +x?) could result in the web server just serving up the source code of your scripts directly to the client. There's your database credentials hard-coded right there at the top of your file now exposed for all to see.
- You had to pay very close attention to resource limits on the server itself to prevent malicious things like fork-bombs
- Apache was a nightmare to configure securely for CGI
In 2023 I don't think CGI is as easy as it was back in the day. The major proxies/servers most people use like nginx only support FCGI which isn't directly compatible with CGI--you need to run helpers like fcgiwrap or other new layers of complexity. The only major servers that still support plain old CGI are apache and lighttpd. IMHO properly configuring apache is a mess.
I think what people really want when they pine for the days of CGI is simple file-based routing for dynamic content. Look around and you can see new frameworks are coming back around to this... or consider plain old PHP.
CGI is slow [ spawns an OS process for every request ]. But FCGI is as fast as any other solution, like compiling the whole app in a single binary and running it as a HTTP server.
CGI is _computationally_ slow, but in practice it's faster than it needs to be for most sites. Some examples of CGI-powered sites i'm aware of include:
Spawning a process is not that slow on modern hardware. CGI is considered slow, because back in the 90's, everything was slow and resource constrained. Slow processors, limited RAM, slow disks for swap, slow internet connections... I remember developing Perl CGI scripts on a Sparc 10 with 32 megs of RAM connected to a T1 that was nearly always maxed out.
On a related note, "modern" cloud-based application development strategies (like AWS Lambda) seem no faster than CGI. We've come full circle. Though now things are more distributed, of course.
It might interest readers to know that the Fossil SCM's primary mode of operation is CGI, and has been since it began life in 2007: https://fossil-scm.org/home
When i first discovered fossil, in December 2007, its ability to run as a CGI was one of its two "killer features" for me, and it's still right at the top of its list of killer SCM features for me.
(Disclaimer: i'm a long-time fossil contributor, but its CGI support pre-dates my joining the project.)
I remember CGI very fondly. I'd still use it to this day in my personal projects if it was not for the fact that I'm running vanilla Caddy 2 that comes without support for it.
Now, CGI and FastCGI are cool, but who else here built CGIs for running on Pre-OSX Macs? I remember AppleScript/AppleEvents based CGI. The Web Server would be a GUI program running on the Mac and the CGI would also be one. They'd communicate to one another sending AppleEvents. I once made good money with a system for running school exams with it.
I still use FCGI. FCGI works a lot like CGI; it loads programs when a request comes in, and the programs deal with the request. But it's much higher performance, because it doesn't have to freshly load the target program each time.
FCGI is an orchestration system. It starts up more worker processes if there are more requests, up to some limit it figures out by itself. If there are few requests, some of the worker processes get an end of file notification and exit. If a worker process crashes, it loads a fresh copy. Little or no human attention required. One of my systems has been up for 1788 days now.
FCGI works fine with Go programs, Python programs, etc. If you use Go, you can get performance reasonably close to what the hardware can do. Until you get to the point that you're renting additional servers as load increases, you don't need more than that for most applications.
You can run FCGI on some cheap shared hosting systems. Why pay more?
I really like Haserl (https://haserl.sourceforge.net/).
It acts as a middleman for CGI scripts and and makes it easy to write them with
a shell or lua while keeping the look a HTTP response with a html body.
> No dependencies or libraries to install, upgrade, maintain, test, and secure.
So you don't actually want this program to do nearly anything useful unless you write it all yourself.
> Zero external configuration, other than telling your webserver to enable CGI on your file.
So now you have to have a webserver external dependency and also configure it. Configuring Apache for CGI securely was no fun unless you were an expert webmaster.
People really put their rose-colored glasses on when they reminisce about this stuff. But there's a reason why we don't use it anymore. We don't need to make up nonsense about why it was so great.
[+] [-] grose|3 years ago|reply
[1]: https://spin.fermyon.dev/
[2]: WASI is a POSIX-ish standard for WASM that gives you all the low level stuff like standard input and output. It includes all the bits and pieces needed for CGI to work.
[+] [-] IshKebab|3 years ago|reply
That makes sense if you want to compile existing software to WASM but that isn't the case here is it?
[+] [-] bmacho|3 years ago|reply
[+] [-] samwillis|3 years ago|reply
The minute you want to do url routing and have middleware, a persistent process makes everything much simpler and keeps everything in one place. I much prefer routing inside a framework than using Mod_Rewrite.
Edit: This article covers what I was thinking of:
"Myths About CGI Scalability"
http://z505.com/cgi-bin/qkcont/qkcont.cgi?p=Myths%20About%20...
Previously discussed:
https://news.ycombinator.com/item?id=32699395
[+] [-] scotty79|3 years ago|reply
I started with PHP and never trusted persistent processes for anything.
What if your persistent process hangs or crashes for whatever reason?
Is your whole site down then, not just the part with the bug?
The only site I ever made in persistent process paradigm was in ASP.NET and it didn't cause too many problems but it was more of an app, not website, with limited number of logged in users and pretty much no anonymous/guest part.
[+] [-] geenat|3 years ago|reply
2023 - Use single binary caddy. It's like 2 lines of config total to remove the extension from your CGI url's.
[+] [-] lucidguppy|3 years ago|reply
It hits that sweetspot of speed of development and responsiveness. You also have the benefit of staying in one language.
Django has so much out of the box that you don't have to decide for yourself after the fact. So much accumulated knowledge of what works too.
Sure go ahead rewrite it if it starts making money and you can afford development time/salary. But these older technologies are easier to fit inside one head and let you get enough speed to take off.
[+] [-] hit8run|3 years ago|reply
[+] [-] sedatk|3 years ago|reply
CGI executables also caused problems when I switched to an Alpha AXP RISC server which emulated x86, bringing the performance to a crawl. That made me switch to classic ASP, and about 10 years later, to ASP.NET MVC with routing, unit tests, abstractions, jQuery, all the shiny things at the time.
Web didn't stop there of course; there came SPA's, React, Vue and whatnot.
Now, seeing the yearning for CGI in 2023 feels funny. Have we come full circle? :)
___
[1] https://github.com/ssg/sozluk-cgi
[+] [-] jstrieb|3 years ago|reply
This is only true if you've done it before, and know what you're doing. In reality, it looks like a mess of `mod_cgi` configuration, trying different combinations of file permissions, finding the magic `cgi_bin` directory, finding the right obscure log files when there are inevitably errors, wrestling with CORS and other subtleties of HTTP headers, and other complexities that are only easy to navigate if you're already an experienced CGI user.
That being said, I love the philosophy of using CGI for scripts. Instead of using CGI itself, though, I wrote a (single-file, statically-linked) web server called "QuickServ" to bring this philosophy into the twenty-first century. It has all of the upside of CGI, but is much easier to set up and run, especially for beginners.
One of its benefits is that it automatically parses submitted HTTP forms, and converts the form fields to command line arguments. That means it's extremely easy to put existing CLIs on the web with minimal changes other than writing an HTML form front-end.
If you like CGI, I will (shamelessly) ask that you check it out!
https://github.com/jstrieb/quickserv
https://news.ycombinator.com/item?id=29002694
[+] [-] ilyt|3 years ago|reply
[+] [-] iso1631|3 years ago|reply
Then in your config file
[+] [-] rgrieselhuber|3 years ago|reply
[+] [-] monkeycantype|3 years ago|reply
[+] [-] iamben|3 years ago|reply
I was talking to a friend the other day who wants to learn some web dev for fun - he had no clue where to start because there's now SO much, so many 'best practice' posts, so much 'you must learn like this'. I know the world changes (and often for the better), but it was a magic time when you could get so far so quickly.
As an aside, it's why I have huge amounts of respect for people like Pieter Levels and his 'just get it done' approach to building things.
[+] [-] alerighi|3 years ago|reply
And it's improved a lot PHP, nowadays it's not that bad language...
[+] [-] papito|3 years ago|reply
I got so jaded dealing with this JS nonsense just to get a simple front-end going for my pet project that I just said "screw it" and started using straight up ES6 in the browser. No libraries, no transpilers, no minifiers.
[+] [-] giovannibonetti|3 years ago|reply
By the way, the best HTML tutorial I found so far is Scrimba. The idea of having interactive videos where you can pause and edit the code is amazing.
[+] [-] endemic|3 years ago|reply
[+] [-] cutler|3 years ago|reply
[+] [-] hestefisk|3 years ago|reply
[+] [-] cutler|3 years ago|reply
[+] [-] papito|3 years ago|reply
mod_perl teaches you quickly about state.
[+] [-] mmcgaha|3 years ago|reply
[+] [-] anthonyhn|3 years ago|reply
IIRC, the issue was that they didn't have a maintainer for the cgi library anymore, and that is why they removed it.
[+] [-] pixelfarmer|3 years ago|reply
On that note, I recently revived some nearly 20 year old perl CGI stuff. Had some minor syntax issues (language ambiguities which got fixed), but it was running just fine after having fixed these. There was also some C code with it (some tooling), needed header include fixing + function renaming, otherwise still worked. Small project all in all, not reliant on a lot of external things.
[+] [-] freyfogle|3 years ago|reply
It's still there, you can still use it. Works great.
[+] [-] HeckFeck|3 years ago|reply
The result is here: https://www.thran.uk/cgi-win/fleg.pl
It is a flag generator affectionately titled flegger.
[+] [-] ilyt|3 years ago|reply
It even does event-based and websockets
[+] [-] nicky0|3 years ago|reply
[+] [-] foldr|3 years ago|reply
[+] [-] tasn|3 years ago|reply
What replaced CGI scripts is slightly more complex, but also more naturally grows to other use-cases.
In short: I personally don't miss CGI. :)
[+] [-] VWWHFSfQ|3 years ago|reply
- Take untrusted input and pass it directly to shell scripts on your filesystem. What could go wrong?
- Executables in cgi-bin were nearly always included in your document root on shared hosting servers.
- File drops and web shells were rampant
- A trivial misconfiguration (not setting +x?) could result in the web server just serving up the source code of your scripts directly to the client. There's your database credentials hard-coded right there at the top of your file now exposed for all to see.
- You had to pay very close attention to resource limits on the server itself to prevent malicious things like fork-bombs
- Apache was a nightmare to configure securely for CGI
The list goes on. I don't miss it at all.
[+] [-] qbasic_forever|3 years ago|reply
I think what people really want when they pine for the days of CGI is simple file-based routing for dynamic content. Look around and you can see new frameworks are coming back around to this... or consider plain old PHP.
[+] [-] habibur|3 years ago|reply
[+] [-] sgbeal|3 years ago|reply
CGI is _computationally_ slow, but in practice it's faster than it needs to be for most sites. Some examples of CGI-powered sites i'm aware of include:
- https://sqlite.org/forum
- https://sqlite.org/wasm (same domain, different sub-site)
- https://fossil-scm.org/home
- https://fossil-scm.org/forum (same domain, separate sub-site)
- https://fossil.wanderinghorse.net
All CGI, all the time, and _plenty_ performant for the purpose.
[+] [-] hawski|3 years ago|reply
It tested in a loop fork/exec and vfork/exec for a dynamic binary and for a static binary with musl.
On an Intel Core i5 6600k:
regular:
fork 13179 ms / 100000 = 131 µs, 7587 per sec
vfork 14423 ms / 100000 = 144 µs, 6933 per sec
static:
fork 3173 ms / 100000 = 31 µs, 31515 per sec
vfork 3833 ms / 100000 = 38 µs, 26089 per sec
[+] [-] icedchai|3 years ago|reply
On a related note, "modern" cloud-based application development strategies (like AWS Lambda) seem no faster than CGI. We've come full circle. Though now things are more distributed, of course.
[+] [-] ilyt|3 years ago|reply
[+] [-] aliqot|3 years ago|reply
Not so uncommon these days tbh
[+] [-] sgbeal|3 years ago|reply
When i first discovered fossil, in December 2007, its ability to run as a CGI was one of its two "killer features" for me, and it's still right at the top of its list of killer SCM features for me.
(Disclaimer: i'm a long-time fossil contributor, but its CGI support pre-dates my joining the project.)
[+] [-] soapdog|3 years ago|reply
Now, CGI and FastCGI are cool, but who else here built CGIs for running on Pre-OSX Macs? I remember AppleScript/AppleEvents based CGI. The Web Server would be a GUI program running on the Mac and the CGI would also be one. They'd communicate to one another sending AppleEvents. I once made good money with a system for running school exams with it.
I miss those days a bit.
[+] [-] kefka_p|3 years ago|reply
Yes, I’m old.
[+] [-] ilyt|3 years ago|reply
projectile vomits profusely
> No dependencies or libraries to install, upgrade, maintain, test, and secure.
Except entire webserver. Just install <your language's preferred web server lib> if you want simple.
> Works with any programming language ever that can read text and print text.
You won't use it with those that can't anyway unless for mental excercise
> Zero external configuration, other than telling your webserver to enable CGI on your file.
Except entire webserver. Just install <your language's preferred web server lib> if you want simple.
Seriously, what kind of nostalgia someone needs to have to have any positive thoughts over cgi-bin?
[+] [-] Animats|3 years ago|reply
FCGI is an orchestration system. It starts up more worker processes if there are more requests, up to some limit it figures out by itself. If there are few requests, some of the worker processes get an end of file notification and exit. If a worker process crashes, it loads a fresh copy. Little or no human attention required. One of my systems has been up for 1788 days now.
FCGI works fine with Go programs, Python programs, etc. If you use Go, you can get performance reasonably close to what the hardware can do. Until you get to the point that you're renting additional servers as load increases, you don't need more than that for most applications.
You can run FCGI on some cheap shared hosting systems. Why pay more?
[+] [-] cassepipe|3 years ago|reply
[+] [-] VWWHFSfQ|3 years ago|reply
So you don't actually want this program to do nearly anything useful unless you write it all yourself.
> Zero external configuration, other than telling your webserver to enable CGI on your file.
So now you have to have a webserver external dependency and also configure it. Configuring Apache for CGI securely was no fun unless you were an expert webmaster.
People really put their rose-colored glasses on when they reminisce about this stuff. But there's a reason why we don't use it anymore. We don't need to make up nonsense about why it was so great.