top | item 8224469

Dylan: the harsh realities of the market

172 points| kryptiskt | 11 years ago |logicaltypes.blogspot.com | reply

118 comments

order
[+] BruceM|11 years ago|reply
I'm the Bruce that he mentioned in the post.

For better or worse, I've been pushing Dylan forward heavily over the last few years and am effectively the primary maintainer.

Over the last couple of years, we've made a lot of progress. We've completely revived the documentation from 1990s era FrameMaker files and have it published via a pretty modern system. We've converted from SVN to Git and moved to GitHub. We've done 4 actual releases. We've improved our platform portability. We've provided some basic debugging integration with LLDB. We've fixed some long standing issues in the compiler and tool chain. We've improved the GC integration on all platforms.

But there's a lot to do. We need to fix our Unicode support. We need to improve the type system in at least minor ways if not major ways. We need to improve how parse failures are handled as the errors are not always friendly. We need more libraries. Some of this is really easy, some isn't. But for pretty much everything, there are bite-sized pieces of work that could be done in a couple of hours/week that would lead to significant gains.

I've wanted to just flat out use Dylan for something and have built some small prototypes with it and while they've worked out well enough, the actual projects themselves didn't go anywhere (unrelated to the use of Dylan).

I think this blog post was triggered by a comment that I'd made publicly yesterday that I'm feeling rather discouraged at this point. There was also a private email that I sent to 19 people who have been involved with Dylan recently, but the author of this post didn't get that email.

I view Dylan, not as a language from the past, but as a stepping ladder towards building a better language for the future. We don't have to get bogged down in a lot of the minutiae involved in creating a new language as a lot of the work has been done. We get to focus on things at a different level and those things are just as important. People bring up Goo often when Dylan comes up. Goo is interesting, but the implementation is nothing close to being industrial enough to survive an encounter with the real world.

I came to Dylan because I saw the mess that Scala and other languages were. I didn't like where they were going and following some people on Twitter like https://twitter.com/milessabin and others seems to show that I'm not alone.

And that's why I'll probably keep at it with Dylan. I want a better future and I'm going to keep trying to build it.

[+] klibertp|11 years ago|reply
Thank you for your work and please, please keep working on Dylan.

A couple of years back I wanted to learn Dylan, but it really looked rather old and unfriendly. I read about Dylan module system and object system and thought it's really interesting. My first impression was that it's somewhat similar to Racket units system and to CLOS.

I hoped for Dylan to be modernized enough to make it easier for me to learn it. I forgot about it as I learned a dozen or so new (or old) exciting languages in the meantime, but seeing improvements in Dylan - especially in the docs - makes me interested in it once again.

> there are bite-sized pieces of work that could be done in a couple of hours/week that would lead to significant gains

How to get involved with it? Are there issues on Github, is there a mailing list? Free time is scarce, but if I finally decide to learn Dylan I can help with its development as well.

Again, please don't stop working on Dylan. It's only natural for such a language to have a small user base, it's nothing to be worried about. Just keep working on it and sooner or later it will become (more) popular. That's my experience with various niche languages to date, at least.

[+] vijucat|11 years ago|reply
Bruce, not sure if you are interested in commercial success, but assuming you are, one suggestion is : whatever you may feel about Scala, I believe that Martin Odersky had the right strategy when he noted that Object Oriented Programming did well despite it's flaws because people could build real, working things with OO languages. This probably holds true even more for PHP.

So my suggestion is to do the same for your language : build a cool, new mp3 player that reminds folks of the days when WinAmp was so awesome. Or build a scalable server-side framework that makes WhatsApp-scale chat easy as pie. Make it easy for folks to achieve commercial success, and your language will thrive, too.

[+] justinpombrio|11 years ago|reply
It sounds like you care not only about Dylan but also about advancing programming language construction in general. Have you considered writing, e.g., blog posts about design decisions you make as you work on Dylan? Maybe it would help other people learn from your work.
[+] bsaul|11 years ago|reply
Since you seem very qualified to critize a PL, could you elaborate on the problems you see with scala ?

I recently had to study the option of starting a real project with this language, but after looking at it, it didn't feel like an elegant and well thought out language. More like a monster language people tried to stuff as many features as they could into.

But that was just a first glance impression. So i'm really wondering what someone like you think of it.

[+] georgegeorges|11 years ago|reply
If you're going to rant, you might want to actually have a clear point to make.

> college kids on comp.lang.lisp asking for the answers for problem-set 3 on last night's homework

Surely not during the Naggum days. CLL was a hostile wasteland.

> That is the lesson of perl and python and all these other languages. They're not good for anything. They suck. And they suck in libraries and syntax and semantics and weirdness-factor and everything.

What? How have you not heard of CPAN? There is not a single language in the world that can touch Perl's libraries. I'm not sure why you feel the need to toss either Perl or Python under the bus to make some petty point about Dylan's lack of popularity. Python replaced Scheme at MIT. It's time to move on. I know I have.

You have to have your head pretty far up your own ass to not see how much Common Lisp sucks. It's a language designed by committee, and it looks like it.

I've used Erlang too. For everything Erlang does well, there are countless areas that make you want to bang your head against the desk.

Languages don't matter. Platforms matter. APIs matter. Playing nicely with the rest of the world fucking matters. Common Lisp wouldn't.

[+] chrismonsanto|11 years ago|reply
> What? How have you not heard of CPAN? There is not a single language in the world that can touch Perl's libraries.

That's a really outdated meme. CPAN is small, pretty much every language you hear about regularly has a larger package space than CPAN. Even Go's package space is bigger than CPAN's. JS and Java each have package spaces roughly 3x CPAN's size.

http://www.modulecounts.com/

[+] klibertp|11 years ago|reply
> Languages don't matter.

Bullshit.

Languages DO matter. Language features matter, too. Even syntax matters, although not that much.

It's easy to get trapped in "languages don't matter" attitude if you know a couple languages. The perspective changes drastically with mastery - being able to use a language to its fullest, as opposed to just using it - and with dozens more of learned languages.

A language is your baseline, a starting point for building things. How you build API, how you interact with outside world, how you accomplish common tasks depends on a language and language features. And it gets even more important as you climb the abstraction ladder. For example, at language level Scheme offers call/cc. You may say it doesn't matter, as it's just a language feature. But take a look at Racket continuation-based web server - it's an impressive piece of work which solves a certain problem really well and is almost impossible to implement in some other languages (you don't need continuations specifically to implement this, there are other features which would enable it). At a language level Lua offers coroutines, which in itself is not very impressive. But take a look at OpenResty: you can write code which looks perfectly normal and synchronous (no callback etc.) while still fitting inside of async by nature Nginx.

Anyway, thing you mention, like APIs, do matter too, but language features are equally important. Don't dismiss them because you superficially know a few languages; rather learn some of them in depth and build real world things with them and I'm 97% sure you'll see how language features matter.

[+] ohyes|11 years ago|reply
> You have to have your head pretty far up your own ass to not see how much Common Lisp sucks. It's a language designed by committee, and it looks like it.

Erh, no. You don't seem to have any historical insight into how common lisp came to be. (Or maybe you do have historical insight and are just being willfully ignorant).

It is a language designed by a set of companies and institutions compromising 25 years ago. Companies and institutions that had radically different operating systems and hardware, it wasn't a matter of writing software for a machine that ran either 'Windows or Unix.' You had to support the lisp machine vendors and a variety of mainframes/minicomputers. Emphasis on 'Common.' The theoretical possibility was that you could write a program on a MacIvory and then run it on your Symbolics Machine, ThinkingMachines machine, Unix Machine, Windows machine, PDP-11, Dandelion, DEC Alpha etc...

Go write a C (or Python for that matter) program that lets you access the file-system port-ably on a huge variety of operating systems using only the C standard. I suspect it will be difficult.

Anyway, you couldn't be any more wrong. You've limited the 'entire world' to essentially two similar platforms, one of which (Windows) isn't really that well supported.

The whole point is that it tried really, really hard to satisfy the entire lisp world, which included a number the larger players who were put out of business by Moore's law and cheap generic hardware rapidly outpacing expensive special purpose hardware.

All this said, Common Lisp (and Scheme, and it's implementations) have features that the general programming community is still rediscovering 30 years later. Dismissing it as something that sucks because the people involved didn't try hard enough is incredibly myopic.

tl;dr: Nu-uh.

[+] eudox|11 years ago|reply
> It's a language designed by committee, and it looks like it.

This means nothing. When a design succeeds, we say it was designed by the community; when it fails, by committee. You can produce examples for any language: In Common Lisp the committee that designed it and the community of previous Lisp users that backed it. For Python, the community of users and Guidos' mailing list pals. There is nothing inherently superior or inferior about design by community or committee, respectively.

[+] lispm|11 years ago|reply
> You have to have your head pretty far up your own ass to not see how much Common Lisp sucks. It's a language designed by committee, and it looks like it.

The original design was by four people. Common Lisp does not look worse or better than the Lisp's it was designed to improve on (Maclisp, Lisp Machine Lisp, NIL, ...) and the Lisps it made mostly obsolete (Portable Standard Lisp, Interlisp, ...).

[+] mindstab|11 years ago|reply
> Languages don't matter. Platforms matter. APIs matter. Playing nicely with the rest of the world fucking matters. Common Lisp wouldn't.

this. playing with CL inside CL is fun and nice. As soon as you try to interface with the outside world, and actually do almost anything you run into trouble. I mean an FS api that was designed before we solidified on hierarchical filesystems? Lisp does not play nice or well or easily with the outside environment it is running on. (in my experience)

it can be an ugly language, but if it has all the libraries, then you can get so much more done so much quicker with so much work done for you, people will go there.

Again, some little progress has started with things like quicklisp finally starting to provide a better (or first proper) package management system for CL like most newer languages have all had, but still. little and late. But given time and work maybe resuscitateable. We'll see

[+] Dolimiter|11 years ago|reply
"Languages don't matter. Platforms matter. APIs matter. Playing nicely with the rest of the world fucking matters."

Nice, I like that.

Reminds me of the Discourse peeps, who rejected eg PHP, and wrote it in their pet language, ignoring that no-one cares about language, and finding the problem that their platform is so difficult to install and use, that no-one uses their software.

[+] sitkack|11 years ago|reply
Read the whole post, everything sucks. It is Salinger + PK Dick + Vonnegut + Allan Watts.
[+] eudox|11 years ago|reply
The article hits rather close to home, as someone who, for better or worse, is committed to improving Common Lisp's ecosystem. I often feel this existential dread of "Is it worth it? Whose lives will it change? Will I spend years labouring in obscurity for nothing?". And while the answer to all those questions is probably not what I'd like to hear, I still do it.

This is why ecosystems are, for most people, more valuable than intrinsic language features: Tribalism along the lines of "We have X thousand people backing us up"/"X thousand devs can't go wrong". People don't care about monads or macros, they care about feeling like they're part of a large community.

[+] pnathan|11 years ago|reply
As someone also invested in Common Lisp, it.... hit really close to home.
[+] seanmcdirmid|11 years ago|reply
Its even worse for the PL designer inventing something new without even a small ecosystem in place; I ask myself everyday "is it worth it?"
[+] bainsfather|11 years ago|reply
Hi eudox, I would love to use CL for my projects - I've read (most of) Norvig's and Graham's books, written small programs etc. I really like most of the language, especially the 'break into debugger on error'.

But, when I came to write CL for a project that needed gui, web scraping, using APIs etc (i.e. I wanted to connect it to the outside world) I was stumped. Unlike in e.g. python, there was no nice gui library that I could just use. Now maybe people will reply telling me that something exists, but compared to python, the difference in ecosystem is massive. (possibly some commercial CL environment has this, but I am unwilling to pay for something that is closed).

I really want to use CL for more than just toy programs, but I don't/can't.

So, when you are asking yourself "Is it worth it? Whose lives will it change?" - the answer is mine, but only if you think you are able to fix this.

Good luck.

[+] rdtsc|11 years ago|reply
> These languages are defined, right there in the dictionary.

Erlang: see 'career wrecker.'

Please. Someone, wreck my career some more.

Unlike Dylan Erlang was created by a company for a purpose with very clear goals and it did and still excels at meeting those goals, and nothing out there gets close to the qualities it has. Not everyone needs those qualities, but sometimes nothing will do. Erlang is at the core of many solid industrial applications -- mobile to internet gateways, message queues, trading systems, large databases, handling millions of concurrent connection and billions of messages per day for WhatsApp.

What does Dylan do? This is the second time I heard about Dylan. I've played with Mercury, Prolog, Nimrod, Curry (Haskell + Logic programming) and other rather obscure languages but haven't heard about Dylan much.

Some languages just don't make it, sometimes it is just luck. However I don't like the disparaging and angry remarks thrown around at other languages and ecosystems. That does nothing to promote Dylan it only pushes people away.

[+] Tomte|11 years ago|reply
> What does Dylan do?

I'm totally unfair here, because I really don't know much about Dylan, but my view maybe explains a bit why Dylan does not even "win" with people like me, who really like those strange languages like Lisp or ML.

To me, Dylan was never about creating something new and great. It was taking Common Lisp and "fixing" its syntax.

Unfortunately, I can't stand Lisp without prefix notation and lots of parentheses. And people who hate prefix notation and lots of parentheses aren't interested too much in Common Lisp.

The resulting set of people interested in Dylan was... well, I've heard of Bruce. Andreas Bogk was doing some heavy advocacy in Germany, to the point where people just didn't want to listen anymore.

Let's say about ten people in the world cared. And I just don't see how that set could grow by orders of magnitude.

[+] lispm|11 years ago|reply
Dylan mostly died before practical applications were written in it. I doubt there were more than ten serious applications written in it (besides its own IDE and compilers).
[+] nostrademons|11 years ago|reply
Dylan was my favorite language back in college. I remember following PG's essays to Lisp, and then Lisp to all the newer dialects like Dylan or Goo. Dylan had it all: a metaobject protocol, generic functions, optional static typing, infix macros. I even got started working on an Eclipse plugin for it, which I ended up shelving after like 3 weeks when I lost interest.

Unfortunately, there are large network effects to programming languages, and the stuff that really makes you productive - libraries and tooling - Dylan just lacked. It wasn't practical to write anything larger than an ICFP contest entry in it. So I went from Dylan to Python, which lacks many of the really cool language features and is a lot slower, but at least comes with so many batteries included that you can whip up a prototype for anything really quickly.

[+] wisty|11 years ago|reply
I don't really think it's purely tooling that's the issue. If it were so easy to use Dylan, then the tooling would be easy to write.

I think it's performance.

"Scripting languages" tear compiled languages apart when it comes to iterating. Write a "hello world" web app page in Python and Scala, then see which one takes 30 seconds and about a gig of RAM to actually display in the browser.

On the other hand, if you needed performance scaled across millions of users, you needed C or C++.

The big languages are C and C++ and Fortran (if you need performance), a long laundry list of "scripting" languages, and Java. Java is the only one that actually gets a spot because of its tooling, most other languages only got tooling because people loved the languages.

There's a lot of compiled languages coming out (or re-emerging) these days (Scala, Haskell, Swift, Julia) with funky features that require compilation that don't get you closer to the metal. I suspect that's largely because computers now have the RAM and CPU power to actually make them fun to use.

[+] Fice|11 years ago|reply
You could think of it as putting a low-pass filter on some of the good ideas from the ’60s and ’70s, as computing spread out much, much faster than educating unsophisticated people can happen. In the last 25 years or so, we actually got something like a pop culture ... So I think the lack of a real computer science today, and the lack of real software engineering today, is partly due to this pop culture.

A Conversation with Alan Kay, ACM Queue, 2004, https://queue.acm.org/detail.cfm?id=1039523

[+] djur|11 years ago|reply
The history of mass computing involves numerous 'bottleneck' events where an increase in usage was bought with a curtailment of hardware and software quality. You had the first blast of cheap minis available outside military and academic environments. Then you had cheap home computers with limited BASIC implementations. That was followed by web apps scripted with half-assed '90s JavaScript and served from cramped mass hosting servers. Then you had smartphones -- iOS using a somewhat spruced-up but constrained version of the decades-old NeXTSTEP, and Android using an outdated and pared-down version of Java.

This is pretty much what "Worse Is Better" is about. Cheap, readily-available software that runs on cheap, readily-available hardware is always going to have a huge head start.

[+] loup-vaillant|11 years ago|reply
> Algebraic types? Dependent types? You'll never see them. They're too ... research-y. They stink of academe, which is: they stink of uselessness-to-industry.

One may think that because closures are finally entering the mainstream (after what, 5 decades?), we have hope for those things to come as well.

But then I saw Swift. Built-in support for an Option type, so one can avoid null pointer exceptions. At the same time, this languages manages to recognize the extreme usefulness of algebraic data types, without using them in their full generality. Like, why bother with a generic feature when we can settle for an ad-hoc one?

I'd give much to know what went so deeply wrong in our industry that we keep making such basic mistakes.

[+] cwzwarich|11 years ago|reply
> But then I saw Swift. Built-in support for an Option type, so one can avoid null pointer exceptions. At the same time, this languages manages to recognize the extreme usefulness of algebraic data types, without using them in their full generality. Like, why bother with a generic feature when we can settle for an ad-hoc one?

Swift has the syntax to define arbitrary algebraic datatypes, even if it doesn't yet work in the beta versions of the compiler.

[+] agentultra|11 years ago|reply
> I'd give much to know what went so deeply wrong in our industry that we keep making such basic mistakes.

I don't think we can all agree on what counts as progress. Some saw Exceptions as the advance in error handling we need while Go reverts back to error codes. People still think Go is superior for different reasons. I think both suck and prefer conditions and restarts as in Common Lisp.

It is rather difficult to build a programming language from a set of axioms we can all agree on.

[+] AnimalMuppet|11 years ago|reply
> I'd give much to know what went so deeply wrong in our industry that we keep making such basic mistakes.

The vast bulk of the industry is much more practical than theoretical. They don't care about your theory about how languages ought to be built. They care about solving the problems that are actually hindering programmers who are trying to write programs.

"But", I hear you say, "null pointer exceptions are one of those problems!" True. "And algebraic types can fix that!" Also true. But here's the thing: (Almost) Nobody cares. Nobody thinks that algebraic types are a price worth paying to fix null pointers.

Do not automatically assume that you are right, and that everybody else is too stupid to see it. Instead, try to expand your mind far enough to see that they may have a better grasp of the trade-offs that confront them than you do. They think your solution doesn't work in their world. Bemoaning their stupidity is the lazy way out. Instead, try to find out why they think that.

I think the problem is not with the industry. The problem is with your expectations of the industry.

[+] chc|11 years ago|reply
Swift has discriminated unions, and IIRC Optional is one example. It's defined as something like:

  enum Optional<T> {
    case None
    case Some(T)
  }
[+] peterkelly|11 years ago|reply
> closures are finally entering the mainstream (after what, 5 decades?)

8 decades, to be precise. They predate electronic computing - Church published his original paper in 1932.

It still amazes me that it's only in the past few years that widely-used languages have begun adopting them.

[+] rayiner|11 years ago|reply
Dylan is also, unfortunately, an example of "worse is better" in action. The extant Dylan implementations were incredibly ambitious. CMU's d2c built on the experience with CMUCL. Harlequin Dylan (AKA OpenDylan) was "Dylan all the way down" with a sophisticated native code compiler and an IDE written in Dylan. Multithreading, generational GC, etc.

And what filled the dynamic language niche? Interpreted languages like Ruby and Python that have yet to achieve 1970's levels of implementation sophistication. But simplicity made them agile and portable and allowed resources to be spent on libraries.

[+] enduser|11 years ago|reply
I appreciate this article by someone who is serious about contributing useful solutions to the world--not just the social aspect of programming--and appreciates a language that empowers him to develop those useful solutions as readily as possible.

When I'm not programming I like to get some distance from my work and hang out with people who have diverse interests. When I'm serious about programming I use Common Lisp. When I'm serious about connecting with other people I use English. Many people seem to confound these pursuits and end up with languages that compromise weakly between talking to people and talking to computers.

For me, programming is about solving business problems ASAP in a manner that is amenable to a long series of minor improvements over many years. Having a stable language standard with language improvements happening as add-on libraries is a huge win. My old code keeps working, so I can stay focused on improvements instead of bailing water.

Also Lisp has the seemingly magical property of being one of the easiest languages to read, understand, and reason about by programmers who have the aptitude to learn it--and it scares the pants off of people who don't. With all of the "expert programmer" pretenders out there it's helpful as an employer to have something that separates the serious programmers from the pretenders.

[+] tempodox|11 years ago|reply
I fully agree. Maybe it's just me, but that sounds much more hands-on to me than the typical HN comment.
[+] nickbauman|11 years ago|reply
If you want to gauge a language look at one thing and one thing only. Who are these people using it and what are they using it for? The answer to that question is more important than anything else on the list after that. Amen.
[+] xplothar|11 years ago|reply
Good one. Used this heuristic to select Erlang and very happy since that time :-)
[+] davidw|11 years ago|reply
There are a lot of things that strike me as being wrong about this essay. Most importantly, to understand the success or failure, you have to think about it, and its economics, like any other product on the market. Granted, programming languages have different characteristics as products than lighthouses or tv shows or telephones, but they do share things with them that we can learn from. This is a somewhat dated attempt of my own to share a bit of that thinking:

http://www.welton.it/articles/programming_language_economics

A few other things:

* Like rdtsc says, Erlang does not fit the mold in a lot of ways. It had a large corporate sponsor from the get-go, which was good for it in some ways (money for developers), and perhaps bad in others: lots of production code early on means it's not possible to change stuff that is less than optimal.

* As per my article, new/small/unpopular languages need a niche, a beachhead if they are to gain traction. You can't create a new language and platform from scratch with an ecosystem as big as Java's (indeed, piggybacking on the JVM is a popular strategy because of this), so you'd better have one thing where you absolutely kick ass. Erlang has this in spades, for instance. Ruby had Rails. Tcl had Tk and a few other "killer apps". PHP was way easier to get started with than mod_perl, back in the day. I don't see this for Dylan, particularly, but then I don't know much about it, so maybe it's there somewhere, and BruceM will figure it out and the language will gain a following.

[+] ZenoArrow|11 years ago|reply
I don't know that much about Dylan, but isn't there some consensus that Julia is the "new Dylan"?
[+] RichardFord|11 years ago|reply
I really like Dylan, but it has a huge problem that never gets discussed - verbosity. The syntax is entirely too verbose for a modern language. My suggestion would be to first complete the Intellij plugin and then actually change the syntax so it can compete with the scripting languages.
[+] pjmlp|11 years ago|reply
Very interesting read.

I have a soft spot for Dylan from magazine articles back in the Newton days.

Here we have a Lisp like language, with a more approachable syntax for the average Joe/Jane developers, AOT compilation and Apple kills it.

I think it is also important to bring out the paper from Erik Meijer about using Visual Basic to hook the typical enterprise developer into FP (via LINQ).

http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.118....

[+] AndrewKemendo|11 years ago|reply
Something I never quite understood about languages was why they are restricted to one or another domain. For example the author writes:

Algebraic types? Dependent types? You'll never see them. They're too ... research-y

Why can't those features be baked into C++ or Java?

[+] peterkelly|11 years ago|reply
I think C++ is full now.
[+] theseoafs|11 years ago|reply
Well as far as I know neither language is clever enough to allow arbitrary proofs to be expressed at compile-time. (I hear that C++'s template system is Turing-complete, which may or may not be true, but even if it is, that doesn't mean that dependent types could be expressed in a way that's easy to write/read.) So you'd have to essentially rewrite a big part of the compiler if you were to "bake dependent types" into C++ or Java, which is of course easier said than done. Not to mention that dependent types are extraordinarily complicated and have only been implemented in research systems.
[+] judk|11 years ago|reply
Java has a fixed assembly-language level syntax. Look to Scala or F#
[+] yarrel|11 years ago|reply
I love Dylan. We should all be using it instead of C++ and Java. Particularly C++. Each time I type "friend" in C++ or have to use the STL, I miss Dylan.
[+] cschep|11 years ago|reply
just say fuck! we're all thinking it!