top | item 13979002

How to write Common Lisp in 2017 – an initiation manual

476 points| macco | 9 years ago |articulate-lisp.com | reply

254 comments

order
[+] hydandata|9 years ago|reply
I started a big project at work using Common Lisp in 2017 and could not be happier. Sure, most nice features have trickled down to other languages, but they are rarely as nicely integrated. And Lisp still has many advantages that are not found elsewhere: Unmatched stability, on-demand performance, tunable compiler, CLOS, condition system, and Macros to name a few. It has its warts too but which language does not?

I found lack of high quality library documentation a bit annoying, but a non-issue, there were tests and/or examples included in practically all of the libraries I have used so far.

Lastly, this rarely gets brought up, but I think Common Lisp has some of the best books available out of any programming language. The fact that it is so stable means that most of material, and code, from the end of 80's and 90's is quite relevant today, and new stuff is being written.

The biggest downside is that it makes JavaScript and Python revolting to work with. But I can still enjoy SML for example.

[+] yarrel|9 years ago|reply
I'd replace the first few steps with "Install Roswell" -

https://github.com/roswell/roswell

Roswell will install a Lisp and QuickLisp for you, and give you a single point of entry to install libraries, create and run code, and launch en editor (Emacs with Slime of course).

I can't recommend it highly enough (I'm nothing to do with the project, just a very happy user).

[+] iak8god|9 years ago|reply
Neat. Does Roswell do all this without modifying your current lisp/emacs/slime/quicklisp if you already have them set up the way you like but want to play around with it?

And is there something like Roswell for Clojure?

[+] Scarblac|9 years ago|reply
I'm used to languages like Python, that have a number of files that are modules, and to start a program you run one of them as an entry point.

C programs consist of a lot of files that are compiled and linked into a binary executable.

Whenever I've tried to learn CL, I couldn't really wrap my head around what the eventual program would be. You build an in-memory state by adding things to it, later dump it to a binary. How do you get an overview of what there is?

I'm just too used to my files, perhaps. Or I'm missing something.

[+] brudgers|9 years ago|reply
I mentioned elsewhere I'm learning Common Lisp. I'm also learning Python by translating some Common Lisp code into Python and part of that is to make sure I understand what the Common Lisp is doing. As an understatement, that's meant building some very unPythonic abstractions (yay, me).

Anyway, I think there is a fundamental design difference between Common Lisp and other 'first class' programming languages: Common Lisp was designed as a way to use computers because a computer user would sit down at a Lisp Machine (which was the future when Common Lisp was designed) and use Common Lisp to do ordinary computer stuff like store their recipes and manage appointments and copy files between directories. It's reflected in the :cl-user package not being called :cl-programmer and the logic of typing (in-package :tps-report) on Saturday at the usual time.

Most other languages don't have this idea...In Unix users don't fool around with stdio.h. In Python, there's no clean way of switching between applications at the REPL -- or rather interpreter -- because of how Python handles the hard job of naming things (like most languages, there's a bit of punting on third down with the catchall epithet 'unpythonic').

This makes it hard to get one's head around Common Lisp, but it explains why a person might leave a REPL running for a week and make better progress because of it.

[+] mikelevins|9 years ago|reply
As other people have said, you can use Lisp the same way you describe using Python or the same way you describe using C.

You also have a third option: you can use Lisp as a sort of interactive command-line calculator-plus-kitchen-sink. Leave it running and teach it to do odd jobs for you. If you accumulate your odd jobs into a file then you'll have them for later. You could of course save the state of the running Lisp, but a source file is better in that it keeps a nice tidy record of the source code of your hacks.

Serious programs written in Lisp are mostly organized pretty much like serious programs written in other languages: the program is factored into a collection of source files that, in the best case, reflects a logical decomposition of the functionality. There's some form of system loader that compiles and loads the sources in the right order and, if desired, dumps the result into an executable program.

Nowadays most people use ASDF for system loading and dumping, I think, but it's not hard to write your own system loader, and I still know people who prefer to do it that way. Just as an example, CCL still uses its own homegrown loader to build.

If your program is like a Python script then it's a Lisp source file that you pass to the Lisp kernel, in just the same way you would do it in Python.

If it's a compiled executable, then your sources were compiled into the Lisp and dumped to that executable, and it works pretty much like a C program. The main difference is that there is no distinguished main(); instead, Lisps generally treat the Lisp's own repl as the default main function, and the image-dumping tools offer you the option to substitute the main function of your choice.

Things are a little different, but only a little, and it's pretty easy to learn how they work.

[+] rayiner|9 years ago|reply
In a compiled system like SBCL, things work much as they do in C, except the process is more "programmable."

To recap: when you load a (dynamically-linked) C program in Linux or OS X, the OS will create a new process, and map the dynamic linker (ld-linux.so or dyld) into that process's memory. It then transfers control to the dynamic linker. The dynamic linker then loads the rest of the program into memory (from ".so" or ".dylib" or ".dll" files) and links everything together (i.e. resolves symbol addresses) before passing control to the application entry point.

A compiled Lisp system like SBCL works much the same way, except it uses its own linking/compilation machinery. A new process is bootstrapped by running a C program, which loads the Lisp image into memory. The image is compiled code/data containing the Lisp runtime, compiler, and standard library. After the Lisp image is loaded, you've got two things to work with: COMPILE and LOAD. COMPILE takes Lisp source code and compiles it into a FASL (compiled code and data, equivalent to a ".o" file). LOAD functions much like dyld or ld-linux, and loads compiled code and data into the running process. Indeed, in some implementations like ECL, LOAD is just built on dlopen(). The "eventual program" is ultimately built by LOAD-ing the FASL files comprising the program.

In practice, you don't do this manually. Instead, you use something like ASDF: https://common-lisp.net/project/asdf/asdf.html#Defining-syst.... ASDF functions much like 'make' in that you've got a system definition listing the files belonging to your program. Whenever you call ASDF from the REPL, it'll look at what source files need to be (re)-built, compile them to FASLs, and (re)-load them into the running process. In contrast with C (but similarly to Python), Lisp allows code to be loaded into a running image. In the Lisp workflow, you don't restart the program you're writing each time you recompile (although you can, if you want). Instead, you keep it running, and load or re-load code into it as you work on it.

[+] lispm|9 years ago|reply
Common Lisp implementations do whatever you want.

> I'm used to languages like Python, that have a number of files that are modules, and to start a program you run one of them as an entry point.

You can do that. start a Lisp with a file to load at start, it can then load the dependencies.

> C programs consist of a lot of files that are compiled and linked into a binary executable.

You can do that, too. That's typical when you create a 'system' declaration which describes your software. ASDF would be a tool for that. You then compile the software, load it and dump an image. Some implementations have a more elaborate way to create applications or can create loadable libraries, which you can integrate into other applications.

> You build an in-memory state by adding things to it, later dump it to a binary. How do you get an overview of what there is?

Typically you would work with files. Write the code in files, evaluate the code from there and build the software from time to time as a whole. If you use SBCL, then you get tons of compile time information, type checks, efficiency hints, warnings, ....

[+] _ph_|9 years ago|reply
You can build your Common Lisp programs from a set of files like any other programming language. "asdf" is the most common system which provides loading complex lisp system. I actually use a makefile to build my Common Lisp based executables from my set of files, no neeed to build up an in-memory state.

The difference with Common Lisp is, you are not limited to this build model, but while your whole code is loaded, you can keep redefining functions for development. This creates the minimal cycle of editing and testing any single function in your program. This is a big asset in development, but in no way means you have to build your systems that way. Actually the contrary, I can only recommend to restart your lisp image frequently to ensure the system loads and builds from file rather than depending on the image state.

[+] jlarocco|9 years ago|reply
I don't know where you got that impression, but CL development is no different than Python development.

Code goes in .lisp files, and then you either load it into the REPL or pass it to the compiler and tell it what the entry point is.

I suppose it's technically possible to develop an entire application at the REPL and then dump to a binary, but I don't think anybody does that any more than they do it in Python.

[+] reikonomusha|9 years ago|reply
Lisp offers a lot of ways to check what there is. You can inspect, search, look at source code, docs, assembly code. All of these things are dynamically inspectable. Tools like Emacs and SLIME make it easier.

You can still look at the files that get loaded. Lisp organizes itself around systems (libraries) and packages (namespaces). It's good to check what packages a system has, then you can check out the symbols provided by a package.

Lisp isn't totally wild-west in the concept of the Lisp image. There is organization to good code.

[+] kunabi|9 years ago|reply
I've been working on a CL project for a couple of years. Was my first big stab at using CL for something other than a toy. Sbcl is a nice choice, but far from the only option. It has many tradeoffs. CL is not without its frustrations. Documentation that has not aged well. A community that can be less than welcoming. (in contrast to say the Racket community) Inconsistencies, e.g. first, nth, elt, getf, aref... However portability appears to be a strong point vs on the scheme scene. Single binary compilation on SBCL/LW/ACL/CCL are great. Found GC to sbcl to be lacking on large garbage rates. Tended to promote garbage to a tenure that prevented it from being removed. It would max out 32GB of memory, even with explicit gc's enabled between files. Where as the other implementations would stay below 4GB.

So ymmv.

Performance benchmarks using cl-bench really highlighted some strong points http://zeniv.linux.org.uk/~ober/clb AWS Cloudtrail parser. https://github.com/kunabi/kunabi

[+] azrazalea|9 years ago|reply
This talks about GC settings that worked for them in SBCL for production, not sure if you already tried the same kind of stuff but may be an interesting read http://tech.grammarly.com/blog/posts/Running-Lisp-in-Product....

If you can make an easy to run example of the garbage collector not working correctly, the folks in #sbcl on freenode are very responsive and have commit access to fix it.

[+] pnathan|9 years ago|reply
Hi, HN! I made this! Ask me any questions you like. I'll try to respond as the workday progresses and I wait for deploys to complete!

[email protected] if you want to email me instead. (or @p_nathan on Twitter, if that's your thing).

[+] znpy|9 years ago|reply
Some notes based on my (brief) experience toying with Common Lisp:

* Why hasn't anyone made a more eye-frendly version of the Common Lisp Hyper Spec ? Having good, easily-browsable documentation is a core-problem.

* The relation between the various native data-types were quite unclear to me.

* dealing with the external world was quite a mess. Many project/libraries implementing only half of something and then got abandoned.

* some libraries had a compatibility matrix... with common lisp implementations. that seemed weird to me.

[+] AlexCoventry|9 years ago|reply
Can someone point me at an argument for why I'd want to write CL in 2017, given all the great alternatives available now?
[+] pnathan|9 years ago|reply
It's a seamless dynamic programming language, which can, with care, be given excellent performance and a high level of abstraction. In my opinion, it's miles better than the other dynamic languages out there, by nearly every factor. It rewards investment and development very well; it's a tool for mastery, not for quick and easy starting.

If you're looking for statically typed languages, it's not going to win there. But my experience writing a lot of Perl and Python, with a certain amount of helping with Clojure and Ruby experience, strongly indicates that Common Lisp is very very competitive there outside of the 'library' front.

[+] baggers|9 years ago|reply
There are some other practical answers here so I'll take a different angle.

Fun! CL is a language to play in, after a day of wrangling Java & ObjC issues I love settling down to just play in an environment that lets blast some code out and play with ideas. Of course this applies to other languages too and this is dependent on your interests, so the case I want to put out there is:

Even if a language isn't suitable for your current business needs, see if it gives you joy. Languages have trades offs to meet their goals, evaluate languages for pleasure too.

Also come visit #lispgames on freenode sometime..most of us are procrastinating making engines but it's always nice to have new folks around.

[+] agentultra|9 years ago|reply
I'll take a stab at this glib comment:

1. You've inherited an application written in Common Lisp

2. Common Lisp has features you find desirable that aren't available in another system

3. You like Common Lisp

4. Common Lisp helps you get the thing you're trying to do, done

[+] phlakaton|9 years ago|reply
This is a question that has been answered many times over the years. Search for "why common lisp" and you'll get a long list of them.

The answers now, IMO, are not substantially different than they were, say, ten years ago. The big differences, I think, are the new competitors in the LISP world: the rise of Clojure, and the rebranding and further development of PLT Scheme as Racket.

[+] HeinzPanzer|9 years ago|reply
Well then you are giving us a fantasy scenario. Because in reality, the alternatives are not great.
[+] agentultra|9 years ago|reply
If you're so inclined I'd make it a "living document" that gets updated as the state-of-the-art evolves. Writing CL in 2017 is not likely to change rapidly in the next decade but even compared to what writing CL was like 8 years ago it has changed enough.

Nice job.

[+] pnathan|9 years ago|reply
Yes, send in PRs if things significantly change! It's a stable ecosystem, but things do improve over time. Things like examples would be quite useful.
[+] mikelevins|9 years ago|reply
For people using macs, it's probably worthwhile to mention CCL's IDE, which you can easily build from within the CCL sources using (require :cocoa-application), or which you can get for free from the Mac App Store (it's called "Clozure CL").

It's a little bit bare bones and a little bit perpetually unfinished, but it works and it gives you a Lisp-aware environment for editingand running Lisp code, and even has a few handy tools.

[+] sigjuice|9 years ago|reply
Is there something like paredit available for CCL's IDE?
[+] edem|9 years ago|reply
How does Common Lisp compares to Racket nowadays? I've seen a lot of activity but I can't decide which one to try out. I only have time for one of them ATM.
[+] spdegabrielle|9 years ago|reply
1. Racket is a multi-paradigm programming language. It has Java-style class/object system, a CLOS-like object system(swindle) and a prototype object system (like self and JavaScript). 2. The macro system is arguably the most sophisticated available. 3. Functionsl programming! including '(purely)Functional Data Structures'. 4. Parallelism (futures) 5. concurrency 6. Contracts 7. Typed Racket 8. Pattern matching 9. Modules 10. Units 12. Pattern Matching 13. Exception, continuations 14. Reflection 15. arguably the most sophisticated tools for creating full languages and DSLs 16. Datalog language (like in datomic) 17.an amazing IDE - it has a tool to debug macros! (but you can also use Emacs) 18. Typed Racket 19. Functional pictures (pict) 20. OpenGL 21 cross platform (win, mac, Unix, Linux) it will even run on a raspberry pi 22. Amazing community 23. Math library (and plot library) 24. Scribble/@-expressions (like markdown but much more powerful) 26. Quite a few libraries
[+] aidenn0|9 years ago|reply
I think you will be happy with either one. It's targeted mainly toward people familiar with them, but here's a recent comparison:

http://fare.livejournal.com/188429.html

Note that a lot of what Faré likes about racket is somewhat intrinsic in there being one implementation, and a big part of why racket branded itself away from scheme (it was formerly PLT-Scheme).

In lisp there are still a lot of very different implementations in use, so if you want to "grow down" you either have to be non-portable or do a lot more work.

I love Common Lisp, so I will say "learn Common Lisp" but you'll probably be just as happy if you flip a coin (and I would recommend doing so rather than debating much as learning the "wrong" one now is probably better than learning the "right" one in the future).

[+] peatmoss|9 years ago|reply
My personal, subjective, feeling is that the Racket offers an easier entry path (docs, libraries and library discovery, relative non-cruftiness, package management via raco, IDE, etc.). I also like that Racket is a little more biased toward FP than CL. I'm told the macro system is ahead of CL's, though I can't say I deeply grok macros.

The Racket community always feels friendly and welcoming to beginners, which is something the CL community hasn't always been.

I do like that CL has a standard with multiple implementations. That said, the standard feels old, and you can quickly run into libraries that were built with less-than-universal compatibility. Things like tail recursion in CL exist commonly in implementations, but not in the spec.

I've tried to carve out a bit of hobby time for lisps over the past few years. I started with CL and fought with the tooling and I could see the power, but I never felt great about my abilities with it. I tried Clojure and periodically use it as a stand-in for Java, and in that sense it is good. But these days, I've been playing in Racket, and it feels like the lisp I wish I'd started with.

[+] gravypod|9 years ago|reply
If this could have a "start" page and a "next" button that will take me from topic to topic in order I'd enjoy that.
[+] ntdef|9 years ago|reply
Ah yes this is exactly what I needed. I was recently trying to start a CL project but I had trouble wading through all the outdated material, especially with regards to including external packages. Thanks for putting this together!
[+] brudgers|9 years ago|reply
I'm going through a similar process. My caution is that 'package' has a very technical meaning in Common Lisp that is at odds with how 'package' is used in other languages (and a bit at odds with how the author uses it in their tutorial).

A package in Common Lisp is a set of interned symbols. In Common Lisp, systems are more in keeping with an ordinary understanding of packages...but combined with the idea of a build system just for fun.

ASDF is a way for managing systems (but it is worth keeping in mind that Common Lisp does not have any 'official' understanding of systems). ASDF is pretty much a de facto standard by consensus.

Quicklisp is a 'package manager' in the sense that it will go out and fetch a dependency from a repository. But what it fetches is a system: it is usually not a package in Common Lisp's technical sense.

From the Quicklisp FAQ:

How is Quicklisp related to ASDF?

Quicklisp has an archive of project files and metadata about project relationships. It can download a project and its dependencies. ASDF is used to actually compile and load the project and its dependencies.

ASDF is a little like make and Quicklisp is a little like a Linux package manager.

On the other hand, Common Lisp is very stable around ASDF and SLIME and QuickLisp. ASDF was started in 2002. Quicklisp in 2005. SLIME in 2003.

[+] GreyZephyr|9 years ago|reply
Wondering if anyone has any experience using lisp for machine learning? I'm aware of mgl[0], but it seems to be abandoned. The lack any wrappers for tensor flow or caffe is also a bit surprising to me. The cliki page [1] is also unhelpful and out of date. Is machine learning on lisp dead or are there projects out there that I'm just not aware of?

[0] https://github.com/melisgl/mgl

[1] http://www.cliki.net/machine%20learning

[+] juanre|9 years ago|reply
I wrote in Common Lisp the star map generation software at the core of my startup, http://greaterskies.com, and could not be happier. But now that it's getting off the ground I wonder whether it may adversely impact my chances of being acquired. Are there any known examples of recent CL-based startups?
[+] na85|9 years ago|reply
If emacs is an obstacle to Common Lisp in 2017, maybe what's needed is a Lisp-interaction plugin for vi(m) (or whatever it is that vim uses in lieu of emacs modes). I don't get the hype for modal editing but you can't argue with the data clearly showing emacs users are in the minority.
[+] isaiahg|9 years ago|reply
I used to have a problem with emacs. But just a few days ago I finally sat down to finally learn it and now, 2 days later, can no longer imagine life without it.
[+] pnathan|9 years ago|reply
Quoting this dead (and useful) comment for others who don't browse showdead:

```

l04m33 3 hours ago [dead] [-]

I wrote a new CL plugin for Vim, which only depends on +channel and supports most SLIME features: https://github.com/l04m33/vlime It's quite new, and I'd be grateful if you could try it out and provide some feedback.

```

[+] l04m33|9 years ago|reply
I wrote a new CL plugin for Vim, which only depends on +channel and supports most SLIME features: https://github.com/l04m33/vlime

It's quite new, and I'd be grateful if you could try it out and provide some feedback.

[+] kahrkunne|9 years ago|reply
Honestly I don't think Emacs is an obstacle to people who use Vim. It's an obstacle to people who use IDEs
[+] macco|9 years ago|reply
If somebody is not comfortable to use emacs (I am), there is a atom plugin for use with CL: https://atom.io/packages/atom-slime

It doesn't replace emacs, but it works as a first Lisp ide.

[+] s_kilk|9 years ago|reply
While people are directing their attention here:

Last year I looked into Common Lisp for a while, but got turned off when I found that there's no distinction between the empty list and boolean false (or nil, in CL-speak).

I found this kinda weird and vaguely off-putting. I don't want to write code to handle the diffence between, say, an empty array and false or null in deserialized JSON data.

Can anyone comment on whether this comes up as an actual issue in practice?

[+] aidenn0|9 years ago|reply
It comes up in practice but is easilly avoided with the mapping of:

    null -> :null [] -> #() and false -> nil
Mapping arrays to lists instead of arrays seems wrong to me, but is what most json libraries do by default. Fortunately most of them allow you to change that.
[+] fiddlerwoaroof|9 years ago|reply
So, I really dislike truthiness and falsiness in other languages but somehow, in Lisp, it just seems to work. One thing that helps is that, in places where the distinction between null and false matters, other features of lisp help keep them distinct: so, for example, getting a value from a hash table returns nil if the value is not found and, consequently, you can't distinguish a missing key from a stored null. GETHASH solves this by returning multiple values: the first is the value stored (if there is one) while the second indicates whether or not the key was found in the hash table. Similarly, optional arguments [declared like (defun foo (&optional bar))] default to null but, if you want to no whether a value was actually passed, you can change the argument declaration to account for this: (defun foo (&optional (bar nil bar-p))) In this case, when bar isn't passed, bar-p will b nil but when bar is passed, it will be t.