> Detecting an error when creating a record is much better than when using it later on, as during creation you have the context needed to debug the problem.
This is a great insight no matter what language or framework you're operating in. Laziness has its virtues, but invariants, validity checks, run-time type checks, etc. should all be performed as early (and often) as possible - it's much easier to debug issues when the are proximate to an object or structure being created or modified, then when that data type is being used much later.
Yes, and this is where statically typed languages shine (in my opinion). I like programming in a style that makes heavy use of the type system to enforce this.
For example when writing an api endpoint to create a task I would typically deserialise the json into a CreateTaskRequest. If the object is created without exceptions I can be sure it is valid. CreateTaskRequest implements the ToTask interface. The service layer takes only objects of this interface and converts into a Task object that gets persisted. The persisted Task then gets converted into a TaskResponse so only valid JSON comes back out.
Lots of classes and interfaces but they are all small and with a single purpose.
> invariants, validity checks, run-time type checks, etc. should all be performed as early (and often) as possible
Having worked on numerous large-scale distributed systems, I strongly disagree (and I would think the author would too).
There is a tradeoff to strictness, which is coupling. You want to validate and type check as much as necessary—but not more. If you are over-coupled in a distributed system, it can make it difficult to change your system incrementally, as one must also do with distributed systems. A canonical example of this is closed enums, which don’t tolerate unknown values. Good luck adding a new enum value if every part of your system is strictly validating that. Each part of the system should validate what it needs in order to guarantee that that system can operate correctly, and no more.
> One of the coolest parts of our codebase is the new general purpose language at its foundation. Though the semantics of the language are substantially different than Clojure, it’s defined entirely within Clojure using macros to express the differing behavior. It compiles directly to bytecode using the ASM library. The rest of our system is built using both this language and vanilla Clojure, interoperating seamlessly.
This is what I imagine experienced clojure developers can squeeze out of a language like clojure. I would venture that they can train a junior programmer in a couple of weeks, and make them productive very fast.
I guess they could make it work in any language but judging by the description clojure is indeed a great fit, due to the macro capabilities, flexibility and solid runtime via JVM.
Yeah, it's nice to see such thoughtful adoption of language facilities clearly oriented towards creating a successful and maintainable codebase.
The Clojure team has always done a nice job expressing the rationale for specific language features, and these rationales often lean towards solving problems that system designers historically faced.
Oftentimes, I think folks think of modern languages in regard to their syntax, tooling, ergonomics, etc; however, to me, the more interesting benefit in adopting a modern language is in how its inbuilt features address design problems that earlier generation languages exposed.
For a real world example of what I'm talking about, you can google "clojure expression problem" and find compelling articles about how Clojure solves this with protocols.
Providing a toolkit for attacking categories of problems inherently gets people focused on the fact that these problems exist in the first place when they may not even recognize them otherwise, and regardless of the choice of language, it leads to better design oriented thinking in the context of larger and more complex systems.
> would venture that they can train a junior programmer in a couple of weeks, and make them productive very fast
I've got some experience with functional languages, and a good amount with functional features in OO languages. Started learning Clojure this week, and was pleasantly surprised with how quickly I could get working projects up and running. Less upfront learning curve than Elixir, which was unexpected.
What I heard from colleagues that work with Clojure is that it is a horrible language where the default way of writing code is an imperative programming style where contexts are passed around and updated. Far from the concepts of functional programming.
While I've used all of them, some of their listed libraries are a little dated IMO. Of course this true of almost all mature codebases.
For the sake of the less experienced, I'd point them to these substitutions in particular:
Compojure: Reitit (which they used in the front end too, so migration maybe in progress) would be my preference for backend routing.
Component: Integrant takes Component's ideas, but prefers the flexibility of multimethods operating on plain data to the typed records approach for defining systems.
Schema: Once very popular, but superseded by clojure.spec (bigger in scope) and to a lesser degree Malli for data schemas.
Potemkin: Avoid, handy for some internal code organisation purposes but hostile to tooling and debugging IMO.
In a lage codebase, the cost of switching is huge, and if older libraries work well, there is often no clear incentive to do so.
Clojure in general being very stable and backwards-compatible makes it even more easier to just continue using older libraries. So what if the library is "dated"? If it works well, why not continue using it?
we have a clojure codebase that's about 100k lines. Honestly I'm kinda fed up with it. Certain 3rd party libs we've used have been abandoned. We wrote our own libs for a major framework and it is failing behind.
Too many "I'm very clever" functions that are hard to understand and also have subtle bugs.
Seconded. I work in a clojure codebase that were trying to get out of. There's just dead libraries everywhere and stuff that maintained by one person that gets no updates at all. That or we just end up making functional "wrappers" around Java libraries and at that point we might as well just write straight Java.
Also yea everyone wants to be so damn smart having macros within macros within macros that no one knows what the original intent of the code is anymore.
The repl based development I find also breeds a really bad mentality of forgoing building a deployment process and instead people just repl in and make a bunch of changes and prod rarely matches whats checked into github.
I develop and maintain a 60k line Clojure+ClojureScript codebase by myself, so I can definitely confirm that Clojure does allow for smaller teams to maintain larger codebases :-)
I also fully agree with this:
> Detecting an error when creating a record is much better than when using it later on, as during creation you have the context needed to debug the problem.
I made it a rule to perform integrity checks as early as possible: when creating, accepting or transforming data. I use spec (but schema would work just as well here), and have lots of pre/post conditions in my code.
I tend to settle on "simpler" solutions. I use mount, rather than component, because it uses the namespace hierarchy and requires less code and management. I use very few macros, and try to use simpler tools rather than more complex ones.
I noticed that these days roughly 30-40% of the code I write deals with integrity checks, anomalies and anomaly handling.
> I noticed that these days roughly 30-40% of the code I write deals with integrity checks, anomalies and anomaly handling.
Love to hear more about this.
My approach has been to convert errors into data and have behavior that deals with conveying these errors in different ways, but it's always felt too complex for the task. I've just got a lot of stuff dealing with handling errors in the different environments (jvm / browser / nodejs / rn) and across async and non-async code.
I don't really like 'Component'. I seems very clunky and we had a lot of issues with it and a lot incidental complexity in our codebase (now converted to Java). It was the first real system that did these sort of things but if I start a project now, I much rather use Integrant or Clip.
I haven't used Clip a lot yet but my next project is defiantly going to be with Clip.
For validation I have started to use Malli (https://github.com/metosin/malli). I really liked the idea behind clojure.spec but some of the implementation was a bit clunky and a bit too 'Rich Hickey', not sure how to describe it otherwise. Schema was again the first serious attempt at a library like that for Clojure so it was really nice when it came out. Malli sort of combines what is great about both.
Really a great article! It's extremely rare to find people who truly understand Clojure/Lisp willing to share to this detail.
Having said that, the article also exposes some flaws of Clojure that I also found.
1. Clojure overly obsesses with expression and non-procedural style coding. While in reality many large scale Clojure repo deploy their own macro to bring procedural style back (like letlocal in the article).
2. The builtin abstraction tools are almost always too simple to be useful.
But those are not big deals IMO.
At the same time, I also wonder if this article mention any feature of Clojure that is truly unique to Clojure compared to other Lisp/Scheme languages. I wonder if the article will still make sense if we simply substitute all "Clojure" to "Racket" (obviously I know ecosystem is not comparable).
I'm also curious. It seems like the website has been around for 2 years and hasn't changed much. If they wrote 250k lines in two years, that is around 340 lines a day.
That seems like a rather large project to build before putting it in the hands of customers.
It does seem odd. Stealth product. Bold claims. 3 blog posts one of which is a funding announcement and the other two having nothing to do with the product. I am intrigued but also a bit skeptical.
I've heard from investors that it is similar to Darklang. It certainly has the same goals, but dunno if it's the same approach in any way. Will be interesting to see
The team is impressive, I was wondering what all this new internal language, 400 macros, etc., could be put towards, thinking they were stuck in over-engineering. But after seeing that promise for their app, I changed my mind. Something that’s capable of making you 100 times more productive probably does need that level of development.
Taken literally the claim isn’t true - a Turing complete type system can enforce any constraint that a Turing complete programming language can. But your everyday type systems typically can’t express concepts like “a list of at least 3 elements” or “a number which is a power of 2”.
If your impression is that this is like sugary unit tests: It is not. You can run specs during development, while running a in-editor REPL, code gets evaluated while you type it so to speak.
It is way more expressive than a type system and it is opt-in, but it doesn't give the same guarantees obviously. It is also not meant to be a type system but rather a tool to express the shape of your data. It is used for obvious things like validation but also for documentation (over time) and generative testing among other things.
Thank you for sharing this detailed rundown. One of the challenges that I faced as a new Clojure developer was understanding how all the parts fit together (and in fact, what parts I should care about in the first place). Example: is learning Component worth it? (yes)
Extremely glad to see this resource published! Thank you to the Red Planet team for releasing this!
> Our codebase consists of 250k lines of Clojure split evenly between source and test code. It’s one of the largest Clojure codebases in the world.
Aren't there many Clojure projects out there, or is the language generally not used for large projects? I have a JavaScript frontend written in Vue that is 150k+ lines of code, without any tests. Which would be 25k lines of code more than they have, disregarding their tests.
I'm not saying that more code is better, I just found it odd that 250k was considered one of the largest Clojure codebases in the world
I jumped to the comments first, and my initial thought was, “what would Nathan Marz think of all of this...and what is he investing in these days?”
I was honestly skeptical, reading all of this clojure-macro criticism (constructive for the most part), but trust was restored when I saw the author. Well done, sir! Excited to see the details.
Why hasn't Clojure provide any java.util.function interfaces integration? for language a that's suppose to "embrace the host" to have "ergonomic" access to the host libs because it itself lacks an ecosystem this seems weird.
this is wonderful - I've been curious about continuations and program state as a language construct. Not sure I understand the redplanet - type system et
[+] [-] bradleybuda|4 years ago|reply
This is a great insight no matter what language or framework you're operating in. Laziness has its virtues, but invariants, validity checks, run-time type checks, etc. should all be performed as early (and often) as possible - it's much easier to debug issues when the are proximate to an object or structure being created or modified, then when that data type is being used much later.
[+] [-] vbsteven|4 years ago|reply
For example when writing an api endpoint to create a task I would typically deserialise the json into a CreateTaskRequest. If the object is created without exceptions I can be sure it is valid. CreateTaskRequest implements the ToTask interface. The service layer takes only objects of this interface and converts into a Task object that gets persisted. The persisted Task then gets converted into a TaskResponse so only valid JSON comes back out.
Lots of classes and interfaces but they are all small and with a single purpose.
[+] [-] cle|4 years ago|reply
Having worked on numerous large-scale distributed systems, I strongly disagree (and I would think the author would too).
There is a tradeoff to strictness, which is coupling. You want to validate and type check as much as necessary—but not more. If you are over-coupled in a distributed system, it can make it difficult to change your system incrementally, as one must also do with distributed systems. A canonical example of this is closed enums, which don’t tolerate unknown values. Good luck adding a new enum value if every part of your system is strictly validating that. Each part of the system should validate what it needs in order to guarantee that that system can operate correctly, and no more.
[+] [-] chromanoid|4 years ago|reply
Actually this sounds quite horrible.
[+] [-] mping|4 years ago|reply
I guess they could make it work in any language but judging by the description clojure is indeed a great fit, due to the macro capabilities, flexibility and solid runtime via JVM.
[+] [-] tmountain|4 years ago|reply
The Clojure team has always done a nice job expressing the rationale for specific language features, and these rationales often lean towards solving problems that system designers historically faced.
Oftentimes, I think folks think of modern languages in regard to their syntax, tooling, ergonomics, etc; however, to me, the more interesting benefit in adopting a modern language is in how its inbuilt features address design problems that earlier generation languages exposed.
For a real world example of what I'm talking about, you can google "clojure expression problem" and find compelling articles about how Clojure solves this with protocols.
Providing a toolkit for attacking categories of problems inherently gets people focused on the fact that these problems exist in the first place when they may not even recognize them otherwise, and regardless of the choice of language, it leads to better design oriented thinking in the context of larger and more complex systems.
[+] [-] trutannus|4 years ago|reply
I've got some experience with functional languages, and a good amount with functional features in OO languages. Started learning Clojure this week, and was pleasantly surprised with how quickly I could get working projects up and running. Less upfront learning curve than Elixir, which was unexpected.
[+] [-] cliftonk|4 years ago|reply
[+] [-] AtNightWeCode|4 years ago|reply
[+] [-] coltnz|4 years ago|reply
For the sake of the less experienced, I'd point them to these substitutions in particular:
Compojure: Reitit (which they used in the front end too, so migration maybe in progress) would be my preference for backend routing.
Component: Integrant takes Component's ideas, but prefers the flexibility of multimethods operating on plain data to the typed records approach for defining systems.
Schema: Once very popular, but superseded by clojure.spec (bigger in scope) and to a lesser degree Malli for data schemas.
Potemkin: Avoid, handy for some internal code organisation purposes but hostile to tooling and debugging IMO.
[+] [-] jwr|4 years ago|reply
Clojure in general being very stable and backwards-compatible makes it even more easier to just continue using older libraries. So what if the library is "dated"? If it works well, why not continue using it?
(I speak from my own experience)
[+] [-] fnord77|4 years ago|reply
Too many "I'm very clever" functions that are hard to understand and also have subtle bugs.
[+] [-] coneill|4 years ago|reply
Also yea everyone wants to be so damn smart having macros within macros within macros that no one knows what the original intent of the code is anymore.
The repl based development I find also breeds a really bad mentality of forgoing building a deployment process and instead people just repl in and make a bunch of changes and prod rarely matches whats checked into github.
[+] [-] dragandj|4 years ago|reply
[+] [-] Zelphyr|4 years ago|reply
[+] [-] raspasov|4 years ago|reply
[+] [-] ashes-of-sol|4 years ago|reply
When you say you have too many "I'm clever" functions, do you mean within code your team wrote, or in the ecosystem at large?
[+] [-] oh-4-fucks-sake|4 years ago|reply
[+] [-] jwr|4 years ago|reply
I also fully agree with this:
> Detecting an error when creating a record is much better than when using it later on, as during creation you have the context needed to debug the problem.
I made it a rule to perform integrity checks as early as possible: when creating, accepting or transforming data. I use spec (but schema would work just as well here), and have lots of pre/post conditions in my code.
I tend to settle on "simpler" solutions. I use mount, rather than component, because it uses the namespace hierarchy and requires less code and management. I use very few macros, and try to use simpler tools rather than more complex ones.
I noticed that these days roughly 30-40% of the code I write deals with integrity checks, anomalies and anomaly handling.
[+] [-] heyzk|4 years ago|reply
Love to hear more about this.
My approach has been to convert errors into data and have behavior that deals with conveying these errors in different ways, but it's always felt too complex for the task. I've just got a lot of stuff dealing with handling errors in the different environments (jvm / browser / nodejs / rn) and across async and non-async code.
[+] [-] nickik|4 years ago|reply
https://github.com/weavejester/integrant
https://github.com/juxt/clip
I haven't used Clip a lot yet but my next project is defiantly going to be with Clip.
For validation I have started to use Malli (https://github.com/metosin/malli). I really liked the idea behind clojure.spec but some of the implementation was a bit clunky and a bit too 'Rich Hickey', not sure how to describe it otherwise. Schema was again the first serious attempt at a library like that for Clojure so it was really nice when it came out. Malli sort of combines what is great about both.
[+] [-] mnming|4 years ago|reply
Having said that, the article also exposes some flaws of Clojure that I also found.
1. Clojure overly obsesses with expression and non-procedural style coding. While in reality many large scale Clojure repo deploy their own macro to bring procedural style back (like letlocal in the article).
2. The builtin abstraction tools are almost always too simple to be useful.
But those are not big deals IMO.
At the same time, I also wonder if this article mention any feature of Clojure that is truly unique to Clojure compared to other Lisp/Scheme languages. I wonder if the article will still make sense if we simply substitute all "Clojure" to "Racket" (obviously I know ecosystem is not comparable).
[+] [-] chairhairair|4 years ago|reply
[+] [-] dj_gitmo|4 years ago|reply
[+] [-] mberning|4 years ago|reply
[+] [-] yamrzou|4 years ago|reply
[+] [-] pbiggar|4 years ago|reply
[+] [-] karmasimida|4 years ago|reply
[+] [-] twobitshifter|4 years ago|reply
[+] [-] mbrodersen|4 years ago|reply
[+] [-] kulig|4 years ago|reply
[deleted]
[+] [-] misiti3780|4 years ago|reply
I have a lot of experience with Python/Javascript now, and spent many years in C/C++/Objective C and Java. Also have some Go.
[+] [-] pkd|4 years ago|reply
If you're into interactive kata-style problems, there's 4Clojure [3].
Also join the Clojurians's Slack [4] for the community.
[1] https://www.braveclojure.com/clojure-for-the-brave-and-true/ [2] https://www.oreilly.com/library/view/living-clojure/97814919... [3] https://www.4clojure.com/ [4] https://clojurians.slack.com/
[+] [-] raspasov|4 years ago|reply
One of the best "why Clojure" talks that I've seen, especially for somebody coming from an OOP perspective, like myself.
[+] [-] sremani|4 years ago|reply
[+] [-] ifFxhF938|4 years ago|reply
[+] [-] geospeck|4 years ago|reply
Can someone please explain this to a novice like me?
[+] [-] jahewson|4 years ago|reply
[+] [-] dgb23|4 years ago|reply
https://clojure.org/about/spec
If your impression is that this is like sugary unit tests: It is not. You can run specs during development, while running a in-editor REPL, code gets evaluated while you type it so to speak.
It is way more expressive than a type system and it is opt-in, but it doesn't give the same guarantees obviously. It is also not meant to be a type system but rather a tool to express the shape of your data. It is used for obvious things like validation but also for documentation (over time) and generative testing among other things.
[+] [-] amackera|4 years ago|reply
Extremely glad to see this resource published! Thank you to the Red Planet team for releasing this!
[+] [-] Xevi|4 years ago|reply
Aren't there many Clojure projects out there, or is the language generally not used for large projects? I have a JavaScript frontend written in Vue that is 150k+ lines of code, without any tests. Which would be 25k lines of code more than they have, disregarding their tests.
I'm not saying that more code is better, I just found it odd that 250k was considered one of the largest Clojure codebases in the world
[+] [-] dmcclurg|4 years ago|reply
I was honestly skeptical, reading all of this clojure-macro criticism (constructive for the most part), but trust was restored when I saw the author. Well done, sir! Excited to see the details.
[+] [-] ithrow|4 years ago|reply
[+] [-] jb1991|4 years ago|reply
[+] [-] logistark|4 years ago|reply
[+] [-] logistark|4 years ago|reply
[+] [-] unknown|4 years ago|reply
[deleted]
[+] [-] swamiji|4 years ago|reply