Ask HN: Production Lisp in 2020?

[dhab]

I have gone through initial chapters of "Practical Common Lisp"[1] and "Loving Common Lisp" [2], and have a bit of intuition on lisp, and the power of macros. Haven't done any projects yet, but was also researching on real issues faced by adopting lisp, and ran into articles about abandoning lisp [3], adopting lisp [4],[5]. Could not find anything more recent; but what they mention in these articles - even the ones that are adopting lisp talk about issues like:

* poor ecosystem of libraries - few gems, most other half-baked

* poor community coordination

* Dependency management limitations with quicklisp

And some specific red flags like:

* poor support for json[6] * poor support for async

* have to restart the server every 20 days because of some memory leak [3]

* hack to tune GC [5]

If you are using lisp in production for non-trivial cases, do these issues still exist? is there a way you can quantify effort is resolving them, and if yes, what is it? and, finally, if you had to re-do your project, would you chose lisp or something else?

[1]: http://www.gigamonkeys.com/book/

[2]: https://leanpub.com/lovinglisp/read#quicklisp

[3]: https://lisp-journey.gitlab.io/blog/why-turtl-switched-from-...

[4]: https://lisp-journey.gitlab.io/blog/why-deftask-chose-common...

[5]: https://www.grammarly.com/blog/engineering/running-lisp-in-p...

[6]: https://stevelosh.com/blog/2018/08/a-road-to-common-lisp/#s4...

[mikelevins]

Yep, I've used Common Lisp in production. Several times. I'm in the middle of writing a new app with Lispworks right now.

Common Lisp is generally the first thing I think of for any new work I undertake. There are exceptions, but usually my first choice is Common Lisp. It has been for a little over thirty years.

I've often worked in other languages--usually because someone wants to pay me to do so. I usually pick Common Lisp, though, when the choice of tools is up to me, unless some specific requirement dictates otherwise.

The objections you list might be an issue to some extent, but not much of one. Certainly not enough to discourage me from using Common Lisp in a pretty wide variety of projects. I've used it for native desktop apps, for web apps, for system programming, for text editors, for interpreters and compilers. When I worked on an experimental OS for Apple, the compiler and runtime system I used were built in Common Lisp.

I'll use something else when someone pays me enough to do it. I'll use something else because it's clearly the best fit for some specific set of requirements. I'll use it when there isn't a suitable Common Lisp implementation for what I want to do. I'll even use something else just because I want to get to know it better.

I've taken various detours along the way into learning and using various other languages. I like several of them quite a bit. I just don't like them as much as Common Lisp.

The pleasure I take in my work is a significant factor in my productivity. Choosing tools that offer me less joy is a cost I prefer not to bear without good reason. That cost often exceeds the advantage I might realize from using some other language. Not always; but often.

There was once a language I liked even better than Common Lisp. Apple initially called it 'Ralph'. It evolved into Dylan, which, in its present form, I don't like as much as Common Lisp. If I or someone else invested the considerable time and effort needed to write a modern version of Ralph, then I might choose it over Common Lisp.

For now, though, I'll stick with the old favorite. It continues to deliver the goods.

[Ingon]

Care to elaborate on how Ralph was different from cl/dylan? Or at least point me to some resources to read more about it? I don’t seem to find anything specific on the internet. Thanks

[mark_l_watson]

I am also working on an app using LispWorks [1]. If you want, please contact me and we can jump on a call and share experiences.

Off topic, but many years ago I had lunch with Larry Tesler (John Koza was also there) and Larry knew of my CL book from the 1980s and pitched me to rewrite it in Dylan. That might have changed my career trajectory.

[1] http://knowledgegraphnavigator.com/

[dang]

I never know about this, and it turns out there have been quite a few rich comments about it over the years—all, unless I've missed one, written by you. Here's a link if anyone wants to look back:

https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

[dialamac]

Do you pretty much exclusively use LispWorks, or do you other tooling. I am curious to hear.

[jasode]

I think Reddit's decision to move from Common Lisp to Python is interesting and their reasoning (ecosystem) is still valid 15 years later.

The historical timeline is especially interesting because Reddit's cofounders Steve Huffman & Aaron Swartz were alumni of Paul Graham's first YC batch and PG is the author of the well-known Lisp essay "Beating the Averages":

- 2001-04 Paul Graham : Lisp essay "Beating the Averages" [1]

- 2005-07-26 Paul Graham : https://groups.google.com/forum/#!topic/comp.lang.lisp/vJmLV...

- 2005-12-05 Steve Huffman : https://redditblog.com/2005/12/05/on-lisp/

- 2005-12-06 Aaron Schwartz : http://www.aaronsw.com/weblog/rewritingreddit

The takeaway from the Reddit case study is this: Yes Lisp macros and language malleability gives it superpowers that other languages don't have (the "Blub Paradox") -- but other languages also have other superpowers (ecosystem/libraries) that can cancel out the power of Lisp macros.

You have to analyze your personal project to predict if the ecosystem matters more than Lisp's elegant syntax powers.

[1] http://www.paulgraham.com/avg.html

[nojito]

Or the simpler reason is that they just knew LISP better than other languages and chose to use it.

Once they needed to grow by adding developers, they decided to rewrite into a language that other people already knew.

The only "Super Power" is the complete understanding of what your endusers want so that you can grow.

[mark_l_watson]

When Alexis Ohanian gave a talk at Google in 2013 I got to talk with him about he and Steve dropping Lisp for Python. He said the Lisp version just kept falling down.

In production using SBCL on a customer project we had good results hiring a SBCL maintainer/developer to work past a few issues. From personal experience, I can assure you that LispWorks and Franz both provide incredibly good support for their customers.

That said, if I had to develop Reddit right now, I would not use Common Lisp either.

[pjc50]

> Blub Paradox") -- but other languages also have other superpowers (ecosystem/libraries)

Indeed. The "blub paradox" is profoundly arrogant - it presumes not only that people are too stupid to see the advantages of Lisp, but that Lisp advocates are smart enough to know that other languages have nothing unique to offer that Lisp doesn't.

[flavio81]

>I think Reddit's decision to move from Common Lisp

That original Reddit system was very simple, yet the Lisp source code was of poor quality (it is currently available on the web). When it was released, the lispers at comp.lang.lisp said they could write a clone in a few days. One of them did it in about 2 or 3 hours (true story).

So it wasn't good code. Moreover they, for some strange reason, chose to develop using a different implementation than the one in the server, and things like that.

Then Reddit got a famous name recommended by PG, who was a Python expert. So he rewrote the thing in Python and obviously, being a Python expert, did a good job.

[snazz]

My understanding was that Steve mostly wrote it in Lisp to impress PG, and then Aaron convinced Steve to rewrite it (although Aaron was kind of a Python fanboy and it's telling that the rewrite only took a weekend).

[andi999]

Do you know of any of this 'superpower'code released somewhere? Like is it possible to study the original viaweb code?

[justinmeiners]

I would like to consider Python a common lisp alternative. But it's one or two orders of magnitude slower and is not really standardized. I think there is a lot more that's attractive than just macros.

[alehander42]

these days many newer languages have macros: lisp is not alone

[jmercouris]

I'm using Lisp in production, our browser and all of our infrastructure is in Lisp https://github.com/atlas-engineer/next.

TO address some of your points;

poor ecosystem of libraries: many great libraries, some incomplete. you must not be afraid to look at the source code to understand and edit

poor community coordination: true, there are few large scale community lisp projects. be the change you want to see

dependency limitations with quicklisp: like what? quicklisp works great. if you are talking about loading several versions of the same library at the same time in a single image, this is being researched and developed

poor support for json: no, there is good support

have to restart the server every 20 days: not familiar, I have had lisp servers running for years now

hack to tune GC: no

if I had to redo my project would I choose Lisp? I would choose lisp, yes. there is no more powerful language than this, and believe me, I have tried many

[dhab]

The post [1] (though old) - hints at no good libs for json (mentions a workaround for one he picks)

  > For me, the most important quality 
  I need in a JSON library is an unambiguous, 
  one-to-one mapping of types. For example: some libraries 
  will deserialize JSON arrays as Lisp lists, 
  and JSON true/false as t/nil. But this means [] and false
  both deserialize to nil, so you can't reliably round trip anything! 

Although this post is from 2018, wondering if it has improved since then.

[1] https://stevelosh.com/blog/2018/08/a-road-to-common-lisp/#s4...

[rataata_jr]

Have you tried Haskell? What was your experience?

[synthc]

Clojure has a good ecosystem and you can fall back to Java's ecosystem if needed. It has excellent support for json, and acceptable async support (I use manifold + aleph, but there are other options). It runs on the JVM which has world class GC.

[ssivark]

I want to understand the complementary picture. Assuming Clojure as a good default, what would be some specific kinds of tasks/problems for which one might wish to avoid Clojure? And what alternatives would you recommend, for those cases? (be it another lisp, or otherwise)

[souenzzo]

I delived a service in 2019 that was written 100% written in clojure. ReactNative app + SPA Web app + Server all sharing the same code.

The developer experience was great. I get subsecond hot-reload on all plataforms (app emulator/app on-device/web/server).

The same UI components was shared between ReactNative and ReactDOM. All App issues are reproducible in Web, then I fix in web and I'm sure that App is working.

Even SSR was possible, with some tweaks.

[virtualwhys]

Got a public repo to share that would show how you pulled this off?

Barring that, as a sibling comment asks, what specific framework and/or libraries did you use to achieve the holy grail of sharing-all-the-things? (thought ReactNative required implementing separate UI for iOS and Android).

Sub second hot reload sounds amazing, btw.

[shoulderfake]

Would love to see a breakdown of your setup for this :)

[deathtrader666]

What framework and libraries did you use?

[sulami]

CircleCI is mostly Clojure, and some Go.

The only problems we are seeing is the significant startup time cost (hence the Go bits, where it matters).

Memory usage also isn't always great, but that's more a JVM problem, as it's not releasing reclaimed memory as quickly as you'd like it to. Fine on servers, annoying on a 16GB laptop.

The Common Lisp problems you outlined don't really apply to Clojure, it's a modern lanaguage and has been stable and versatile for us.

[kudokatz]

Having used Clojure in production many years ago, I was very satisfied. And solid interop with existing Java libraries really brings in a whole ecosystem if needed.

Persistent data structures are really game-changing.

[capableweb]

Is the new frontend also written in ClojureScript? Seems you've closed sourced the frontend (old source is here https://github.com/circleci/frontend) but you used to use CLJS, but maybe new one is written in something else?

[reikonomusha]

Rigetti Computing is a company that builds quantum computers.

We have used Common Lisp in production at Rigetti Computing for 4 years both in cloud deployment as well as for a downloadable SDK. Common Lisp is used to build what is currently the best free optimizing compiler for quantum computing [1, 2] as well as one of the most flexible quantum computer simulators [3]. They’re both open source, so you can actually see for yourself how we version, deploy as binaries, and deploy in Docker containers.

> If you are using lisp in production for non-trivial cases, do these issues still exist?

Before getting into the specifics, any large, non-trivial product that customers use will need tweaking. Especially languages that have runtimes, including languages like C. This is usually a result of leaky abstractions that just have to be dealt with in The Real World (TM).

> * poor ecosystem of libraries - few gems, most other half-baked

You have libraries for most things you need. Some are lacking, like in scientific computing. But all your usual stuff to glue things together is there and works.

Some people confuse “half-baked” with “no commits for 5 years”. Lisp libraries are older than many in vogue languages, and Lisp doesn’t change, so “5 years no commits” shouldn’t be worrisome.

At Rigetti, the biggest weight we had to pull that we wouldn’t have needed to do in, say, Python, is bind to a bunch of FORTRAN code, specifically LAPACK. That was a pain because you need to do FFI with complex double floats.

* poor community coordination

The community all agrees on one thing: Lisp is standardized.

Lisp lacks a sort of “centralized community” like Go or Clojure. I like to think that Common Lisp is more “democratic”. We have a “constitution” (the Common Lisp standard) that we all agree on and we all make our own contributions atop that.

With that said, #lisp on Freenode is attentive. SBCL, for example, responds to bug reports almost immediately. Several library ecosystems are kept up to date, etc.

> * Dependency management limitations with quicklisp

With Quicklisp, I personally realized a lot of other dependency management solutions are overblown and don’t actually solve problems. Maybe QL gets away with it because the ecosystem is smaller than others.

Xach, the creator of QL, does a lot of community diligence for free to ensure things work.

For production code, I’ve had 0 issues.

> * poor support for json

We use YASON [4] and it works fine.

> * poor support for async

This is true now but wasn’t true years ago. Lisp implementers have not been interested in implementing a runtime supporting async programming. Little in the Common Lisp language itself prohibits it. Before the world of threads and SMP, Lisp implementers did implement async runtimes.

Why aren’t Lisp implementers interested in async? I’m not sure. Maybe they see it as a passing fad?

> * have to restart the server every 20 days because of some memory leak

Memory leaks can happen but they’re usually the programmer’s fault for not understanding symbol interning, weak pointers, or finalizers. Writing long-running applications does require know-how.

> * hack to tune GC

I don’t consider it hacking. In SBCL we tune the GC to our application. This seems reasonable; different programs have different allocation patterns. But you wouldn’t need to do this out of the box.

> is there a way you can quantify effort is resolving them, and if yes, what is it?

Big applications that are customer products require care and feeding. The benefits of writing in Lisp in the first place largely outweigh some of the extra work needed to productionize it.

I will say that writing production Common Lisp code is greatly benefited by having at least one experienced Lisper on the team. If a team would, for some reason, write an application in Lisp but not really have a good understanding of the language, then productionizing and hardening will be difficult.

> and, finally, if you had to re-do your project, would you chose lisp or something else?

Common Lisp allows something that never existed before (a completely automatic, optimizing quantum compiler) to exist. If Lisp weren’t there, that wouldn’t exist.

My parting advice is this: The biggest difficulty in writing an application is to actually write it. It takes a lot of hard work to get requirements, listen to feedback, articulate a direction, etc. In Common Lisp, SBCL or LispWorks or whatever else, dumping an executable is so easy and you can have a deployable “production app” in 5 minutes. So the good news is that you won’t be held up trying to figure out how to dockerize your Python app.

I definitely agree that before embarking, doing an objective and relevant comparative analysis would be good (we did this at Rigetti), but ultimately you just need to sit down and write.

Almost every programming problem is solvable these days with enough glue and elbow grease. I wouldn’t be too worried.

[1] https://arxiv.org/abs/2003.13961

[2] https://github.com/rigetti/quilc

[3] https://github.com/rigetti/qvm

[4] https://github.com/phmarek/yason

[mikelevins]

Ping me if you need another experienced Lisper. ;-)

[7thaccount]

How complete are your LAPACK bindings?

[dhab]

Thank you for your detailed insight

[thom]

Perhaps you didn't mention it for a reason, but I am sure I won't be the only one to bring up Clojure in this thread, and others will argue for and against more eloquently. Obviously it lacks much of what makes people love Common Lisp, but it's more Lisp than not Lisp, and specifically addresses all of your points via the JVM and its ecosystem.

[dan-robertson]

I’ve not used lisp in production. But I sometimes wish I did. In particular, whenever there’s an issue with a production process (think a batch job rather than some kind of online processing), I wish I could attach a debugger, inspect some variables, and invoke a restart. I don’t know how of any language that makes it more convenient to do that sort of issue-resolution.

I don’t really care about libraries. I think generally it’s not very hard to write an ffi binding to a C library and most of the time you don’t need a library anyway. Most of the time you can just write your own library from scratch as you probably only need a few functions.

But maybe this is more a statement of the sort of programs I work on and if you worked on different systems you would want to use lots of external libraries that you don’t control.

I think the biggest issue with CL is how tightly attached it is to an outdated standard. It makes it harder to have things like good Unix support if your interface is through pathnames and streams.

Other thing which may be annoying about CL are the lack of a good way to consistently go from source code to a binary and that compiler optimisations can be unpredictable

[justinmeiners]

> consistently go from source code to a binary

Can you talk more about this?

[mark_l_watson]

First, thanks for linking my Living Common Lisp book. Common Lisp is my preferred language, but only for projects where I need/want an environment that support rapid experimentation of new ideas. If I know how to implement something and I want to deploy at scale, then I might not choose Common Lisp.

I am a happy LispWorks customer but I also use SBCL (efficient code, easy to package standalone apps) and Clozure Common Lisp (fast compiler for development).

[TurboHaskal]

Can't speak for SBCL (I use it exclusively for development and haven't deployed anything non trivial and user facing with it) but I have used Clozure CL (CCL) successfully to develop back end services without horror stories regarding the GC. If you end up needing huge heap sizes, in theory with ABCL you get access to state of the art GC algorithm implementations and (IMHO) a much nicer, less dogmatic language than Clojure.

I recently switched to LispWorks after struggling a bit with CCL's Cocoa bridge and I wish I had done that sooner. A much younger, idealistic me would scream at the thought of it but sometimes you just need to throw some money at a problem. CAPI is really a killer feature.

[varjag]

We use CL for production (embedded/distributed system). CCL and Lispworks on low end ARM32.

We do use JSON but mainly for the product's Websocket API towards higher level SCADA system. As we have control over both ends there isn't much trouble really.

The project has soft realtime component also implemented with Lisp, and at least Lispworks exposes a substantial degree of control over GC. We however didn't need to tune anything thus far.

The system has perhaps a dozen direct dependencies to the libraries, luckily few of them posed any challenges in use.

[spauldo]

If you don't mind my asking, what company is this? I'm a Lisp hobbyist working for a SCADA integrator, and while I'm not in the job market right now I might be in a few years.

[foobar-xk]

Apple has been using Clojure in production for years, and is hiring: https://jobs.apple.com/en-us/details/200000986/apple-media-p...

Excerpt from the description: "These are the people who power the App Store, Apple TV, Apple Music, Apple Podcasts, and Apple Books. And they do it on a massive scale, meeting Apple’s high expectations with high performance to deliver a huge variety of entertainment in over 35 languages to more than 150 countries."

[dutchblacksmith]

I use lispworks, the CAPI works well on allmost all platforms. They know what they are doing. Books: common lisp recipes, PAIP, ANSI COMMON LISP. The few libraries you need are very good.And you need Quicklisp. I stopped using C, C++, pascal, fortran, javascript. Altough they all are just fine. You still have to learn the craft of programming. Just go for it.

[nicklaf]

I'm going to second the recommendation of Common Lisp Recipes. The others too, of course (especially PAIP).

[Grue3]

Not really production because the traffic is low, but I run http://ichi.moe on SBCL (Hunchentoot web server) and I never had to restart it because of memory leak or a crash. I usually hot-swap the updates and the same server process stays up for months. It also heavily uses JSON and XML and the libraries for them (jsown and cxml) are perfectly adequate.

[lukego]

I'm starting a new project in Lisp in 2020. That's rolling my own CAD tools for learning hardware development.

Lisp makes sense because it's a comfortable environment for interacting with my data. I'll be running the Lisp code "offline" so I'm not worried about runtime aspects like GC pauses and most of my I/O will be generating data for external tools (e.g. KiCad) in their own esoteric formats.

Coming back to Lisp it's nice to see that all my favourite software is still available and working fine, and there seem to be more active open source Lisp developers than ever. (Far fewer than Javascript, for example, but that's not my yardstick.)

[vindarel]

I am running a web app in production, with live reloads, no issues so far.

> * poor ecosystem of libraries - few gems, most other half-baked

I actually had the choice for my libraries: DB interface, email sender, HTML template, data structure helpers… these ones with good documentation.

I have observed and explored the ecosystem for the last 3 years and I still find hidden gems. There are way more libraries than we can see at a quick glance.

We do a selection on https://github.com/CodyReichert/awesome-cl

> * poor community coordination

It looks like the oldest ones don't coordinate much and the youngers do (and push the established players, such as the CL foundation).

> * poor support for async

Maybe not for a full async runtime (cf Turtl feedback), but there is a good choice for all sort of async stuff (bordeaux-threads, lparallel, lfarm, channel, SBCL built-ins etc)

[deckeraa]

I use Clojure and ClojureScript in production. The developer experience is a real pleasure: there is good library support, libraries are typically easy to use, and you can always interop with the underlying JVM/js if the need arises.

Clojure also has the core.async library. This can be used along with Reagent to build React-based apps; core.async allows you to do what Redux does but in fewer lines of code.

https://github.com/deckeraa/OpenStainer is an example of building apps this way.

[juskrey]

Clojure is a production Lisp in 2020

[_ix]

Our health benefits startup has been using Clojure in production for some time. Efforts are ongoing to migrate the last bits from other JVM languages to Clojure, but ClojureScript is already ubiquitous.

[ashtonkem]

I used to work in Common Lisp (SBCL).

The ecosystem problem meant something different to me than most. Sure, when new tech came out (Kafka to pick a random example) there wouldn’t be an open source library ready for me. But I had stable infrastructure so I didn’t really care much about that.

The ecosystem problem for me meant that if something went wrong I was on my own. It’s a pretty unfortunate feeling to search for a bug and find nothing on Stack Overflow.

[kazinator]

> It’s a pretty unfortunate feeling to search for a bug and find nothing on Stack Overflow.

Par for the course in embedded. So isn't going to tell you why the kernel driver for the ABC123 chip is locking up on the XYZ15 eval board from Such-and-Such vendor.

You debug it yourself, whipping out JTAG debuggers and oscilloscopes, if necessary.

Because I work in embedded, I don't understand the whole ecosystem fuss.

I mean, joy of joys, you have the darned source to almost everything nowadays. That makes everything so easy, that all else is a footnote.