I've been building with electric for several months now and it's hands down incredible! Why? I think most importantly because it allows for functional composition across the network (client/server–distributed systems more generally) while also ensuring payloads are as small as possible (only what functions in each env access) which translates to FAST.
IMO this is a paradigm shift similar to GC. No one wants to manage their garbage. No one wants to manage the network. It's grunt work. Let the compiler take care of it.
I've naively rewritten a re-frame (react/redux in js) app that started getting prohibitively slow due to a bunch of deserialization on the client in electric and the performance is outstanding! TBH I'm taking a break from additional UI work on the app to do research because I realized that with electric I can build something much more amazing than I thought possible before.
Caveat: the future will not be served on a silver platter. it's early days and there are some quirks
This seems similar in approach to Lamdera for Elm. Just curious, did you take any inspiration from that project or arrive at this solution independently?
Practicalities aside (and there might be many issues and even be show-stoppers), the approach is a breath of fresh air in what appears like a stagnating swamp. How to orchestrate overall client/server computations is obviously a vital importance if not the most important design choice in the modern tech landscape. Yet while strong opinions abound, frameworks that help us to think more concretely and rationally about the tradeoffs and break the redundancy of infinite alternative possibilities are not common.
If I understand the idea correctly effectively here you have a platform where alternative choices are simple rearrangements of code.
Not to be a downer, but while I think in theory this sounds interesting, and treating frontend and backend differences as a DAG that is compiled for the different environments seems cool, I'm skeptical that it would actually be productive in a large application.
One thing I'd be particularly concerned about is debugging. One of the things that's on the roadmap is "developer experience", and I don't see any particular debug docs or helpers, so it tells me that if there is a bug somewhere debugging it is going to be a pain, even more so because I won't easily be able to see whether the problem is my client or server code as they live together and are compiled into separate things. As a critique the fact that developer experience is on the roadmap rather than thought about from the beginning is a little off-putting, but maybe that's not what was meant here.
Another thing that gives me hesitation is that I can't see how I would extend the setup to support offline capable applications. This is increasingly becoming a big thing for web+desktop apps, does this easily support that?
Lastly, the docs seem to point to use a homegrown DOM library, rather than one of the major players. How easily would this work with a library like Reagent, Helix, or UIx?
Re. abstraction weight - You're right, it's true that the program is less inspectable than before, and that it adds more distance between programmer and metal. We're aware of the issue and have already begun – we landed async stack traces a while ago (which works about the same as in React.js).
And yet. It's also true that HLL (high level languages) like C, Java, Python all faced the same issue. The benefit is a huge gain in terms of ability to reason about ever larger programs. Java gave us garbage collection. Not everyone liked that; C++ developers want to manage their own memory. But at what cost?
Our thinking is: first get the semantics right. Then polish the experience (today we are here). Then add observability tools. Then make it fast. And if we've screwed up the semantics, then the project will fail for the reasons you describe.
The saving grace is we're seeing 10x LOC reduction (18k to 2k) in rebuilding Electric's sister project, Hyperfiddle (a spreadsheet like tool for robust UI development), as well as massive gains in performance.
Re. offline-first - we think the model can be extended to that but this is a blog post for another day
> As a critique the fact that developer experience is on the roadmap rather than thought about from the beginning is a little off-putting
Implementing it is on the roadmap but that's no evidence that it hasn't been thought about extensively and for something like this you really -have- to think things out up front (it's more like a math proof than a CRUD app in a lot of ways) or you're going to find yourself with holes in both feet and a codebase you have to throw away and restart from scratch.
From what I can see, they've been laser focused on getting the core to be correct by design, which will have required repeatedly gutting parts of it as conceptual mistakes are shaken out, so if you -wrote- the DX code as you were going rather than simply keeping in mind that the codebase needs to be amenable to it -being- written (and periodically thinking through how you'd do so as a sort of mental dry run) I'd expect the DX parts to have to be thrown away entirely, repeatedly, which is a motivation sink and in the best case scenario would've substantially slowed down getting the core right.
But yeah, the key thing is "not yet implemented" and "hasn't been thought about at all" are very different things and I'd strongly suspect that my read of what was meant is more likely to be the right one given the nature of the project.
(please take the above as "an explanation of why I suspect their actions are fine and their chosen trade-offs make sense" rather than "let me tell you why you're wrong" because you do have a point that if you don't keep DX in mind at all in the early stages the end result tend to be unhappy-making, and, well, let's just say I speak from repeated experience of doing that that to myself ;)
Our DOM module is only 300 LOC - it's bare metal DOM point writes + Electric (reactive language) + macros for JSX-y syntax. When the programming language itself is reactive, DOM rendering falls out for free.
Mechanically, Electric is comparable to Solid.js except the reactive engine is general purpose, not coupled to DOM rendering, which is a special case of incremental view maintenance.
Electric's reactive engine is https://github.com/leonoel/missionary ; Electric can be seen as a Clojure to Missionary compiler (if you ignore the distribution stuff - which btw is optional). Missionary is cross-platform functional effect and streaming system that today is tested on JVM, Node and Browser platforms. Missionary was created by Léo Noel, who is also the architect of Electric.
I've been building with this for 2+ months. Debugging is fine (not great but when is it great? Especially in Clojure). It prints stacktraces in each env. Haven't found debugging harder than traditional client/server dev. But complexity has decreased thanks to it so less bugs.
Agreeing on the debugging concerns. Adding another layer of abstraction on top of everything else isn't going to help anyone. Creating an abstraction that fits so well that you don't have to worry about the layer underneath is extremely hard. That might not show during initial development, but will come up in the maintenance phase for sure.
A bit unsure where the observation of more offline capabilities comes from. I have the impression that things are moving in the opposite direction: Everything needs to be online.
Regarding offline I tested the photon-starter-app repo a bit and it looks like it already supports disconnections but without optimistic updates:
Here's what I did
- killed the network
- added a bunch of items to the todo list, which disappeared from the input field but didn't appear in the list
- switched network back on, items appeared in the list
I'm curious to see how adding optimistic updates to the mix would work
You can already kill the boilerplate with code generation and metaprogramming. Once you have done that, you figure out why that is not always a good idea and how to best deal with the exceptions that your particular use case brings to the table.
I don't know much about this kind of architecture, but wouldn't it be problematic from an infosec standpoint? It seems like it would be very easy to accidentally leak information from the server to the client in a subtle manner if the developer makes a mistake somewhere. And in ways which would not be obvious from reading the code unless you carefully consider the implications of each e/server and e/client with respect to the mentioned graph analysis?
Are there any mitigations in place to prevent this, except the obvious "be careful and don't write bad code"? Or have I misunderstood things completely and it's a non-issue?
(founder here) You're right that accidents can happen, and if we need to build in safeguards to prevent accidents (i.e. tag a value as server-only) – that should be trivial. To be clear, your backend still exists and has the same security model as a REST Backend-for-Frontend. The client/server markers denote at compile time where a computation (function) must occur. A scope binding like `password` will only transfer if the programmer asked it to by using it from a client region. That would be like putting passwords in a REST payload – it's a programming bug and should be caught in review. But you're right, providing safeguards is a great idea.
Code is precompiled so you could add all your safeguards during compilation. Should not be too hard to enforce fo only precompiled code to run instead of exposing eval. However, I don't believe this is implemented at this stage.
I tried to get hands-on experience with this hyper fiddle stuff a few months ago but I think it was still in closed beta.
I would imagine you can inspect the network traffic. I would hope it would be readable so that you could have a good idea of what's going on and errors would be more obvious.
Hey Dustin, could you share a few details of how you are funding your projects? I know that you have been working in hyperfiddle for a long while but not sure if you have any revenue from it yet.
Your projects are very “futuristic” and the kind of thing I imagine may be hard to get financial backing for, because of the novelty and other considerations… so makes me curious how you balance innovation with monetization in your work.
Personally I think it is ok to say that I am one of the people backing Dustin financially. The last project I backed in this space was Light Table/Eve who did some interesting work in the space. I think Dustin's work shows promise too.
“The Electric compiler performs deep graph analysis of your unified frontend/backend program to automatically determine the optimal network cut, and then compile it into separate client and server target programs that cooperate and anticipate each other's needs.”
This make it sound like client and server segments are determined on the fly based on prevailing conditions, but then the sample code has forms wrapped with e/client and e/server. What do you mean when you say you determine “the optimal network cut” and how do you reconcile this with the expressly marked client and server code?
By the way I’ve been following this project for a while and am very excited about it, kudos on getting to this point.
Great question. A trivial example, expr (e/client (inc (e/server (e/client x)))) should simplify to just (p/client (inc x)) and eliminate the pointless (and perf-damaging) server round trip.
It gets interesting when you introduce lambdas, revealing that this optimization is non-local – you don't know if the transfer is superfluous until you know where the function is called from, and consider the caller and callee together.
The good news is that the DAG (Circuit really) contains everything there is to know about the data flow, so insert compilers research here and poof, insanely fast app.
Looks very awesome. Will definitely give it a try. Hopefully it doesn't end up like Om.next. Om.next felt like an amazing quasi-revolutionary thing when David announced it. But it was so poorly documented and hard to get it right, almost nobody used it in production. I don't know for a fact, but it felt like it complicated Circle CI's frontend to the point that they'd decided to abandon Clojurescript altogether. It's sad, because Om.Next really did have some cool ideas. Fulcro project fixed many problems, but it was too late, the momentum for Om.Next was already wavering.
Seems like e/client and e/server are for when you'd like to force sections of code to run on one side or the other. Otherwise the compiler decides what lives where?
Could someone who understands the similar blurring of lines between client and server in the F# world, through Fable or more broadly in Blazor/Bolero, comment on how this approach compares?
I may be mistaken but I think Electric Clojure renders HTMX unnecessary. Where HTMX still works by a developer specifying the network calls (albeit in a declarative manner), Electric Clojure hands over the same responsibility to the compiler.
This looks cool. If I think one step ahead, this reduces enough incidental complexity its the kind of thing that, when combined with AI, could produce the holy grail of an AI writing most of the code for a full CRUD app.
[+] [-] den1k|3 years ago|reply
IMO this is a paradigm shift similar to GC. No one wants to manage their garbage. No one wants to manage the network. It's grunt work. Let the compiler take care of it.
I've naively rewritten a re-frame (react/redux in js) app that started getting prohibitively slow due to a bunch of deserialization on the client in electric and the performance is outstanding! TBH I'm taking a break from additional UI work on the app to do research because I realized that with electric I can build something much more amazing than I thought possible before.
Caveat: the future will not be served on a silver platter. it's early days and there are some quirks
[+] [-] yenda|3 years ago|reply
[+] [-] dustingetz|3 years ago|reply
Demo videos: https://hyperfiddle.notion.site/Electric-Clojure-progress-De...
Prior HN discussions addressing many of the FAQs: https://news.ycombinator.com/item?id=28630209 https://news.ycombinator.com/item?id=31217448
Some notes on the network layer: https://www.reddit.com/r/Clojure/comments/vizdcc/hyperfiddle...
I'll try to address some FAQs in another post
[+] [-] ordinaryradical|3 years ago|reply
[+] [-] college_physics|3 years ago|reply
If I understand the idea correctly effectively here you have a platform where alternative choices are simple rearrangements of code.
[+] [-] dustingetz|3 years ago|reply
[+] [-] yladiz|3 years ago|reply
One thing I'd be particularly concerned about is debugging. One of the things that's on the roadmap is "developer experience", and I don't see any particular debug docs or helpers, so it tells me that if there is a bug somewhere debugging it is going to be a pain, even more so because I won't easily be able to see whether the problem is my client or server code as they live together and are compiled into separate things. As a critique the fact that developer experience is on the roadmap rather than thought about from the beginning is a little off-putting, but maybe that's not what was meant here.
Another thing that gives me hesitation is that I can't see how I would extend the setup to support offline capable applications. This is increasingly becoming a big thing for web+desktop apps, does this easily support that?
Lastly, the docs seem to point to use a homegrown DOM library, rather than one of the major players. How easily would this work with a library like Reagent, Helix, or UIx?
[+] [-] dustingetz|3 years ago|reply
And yet. It's also true that HLL (high level languages) like C, Java, Python all faced the same issue. The benefit is a huge gain in terms of ability to reason about ever larger programs. Java gave us garbage collection. Not everyone liked that; C++ developers want to manage their own memory. But at what cost?
Our thinking is: first get the semantics right. Then polish the experience (today we are here). Then add observability tools. Then make it fast. And if we've screwed up the semantics, then the project will fail for the reasons you describe.
The saving grace is we're seeing 10x LOC reduction (18k to 2k) in rebuilding Electric's sister project, Hyperfiddle (a spreadsheet like tool for robust UI development), as well as massive gains in performance.
Re. offline-first - we think the model can be extended to that but this is a blog post for another day
[+] [-] mst|3 years ago|reply
Implementing it is on the roadmap but that's no evidence that it hasn't been thought about extensively and for something like this you really -have- to think things out up front (it's more like a math proof than a CRUD app in a lot of ways) or you're going to find yourself with holes in both feet and a codebase you have to throw away and restart from scratch.
From what I can see, they've been laser focused on getting the core to be correct by design, which will have required repeatedly gutting parts of it as conceptual mistakes are shaken out, so if you -wrote- the DX code as you were going rather than simply keeping in mind that the codebase needs to be amenable to it -being- written (and periodically thinking through how you'd do so as a sort of mental dry run) I'd expect the DX parts to have to be thrown away entirely, repeatedly, which is a motivation sink and in the best case scenario would've substantially slowed down getting the core right.
But yeah, the key thing is "not yet implemented" and "hasn't been thought about at all" are very different things and I'd strongly suspect that my read of what was meant is more likely to be the right one given the nature of the project.
(please take the above as "an explanation of why I suspect their actions are fine and their chosen trade-offs make sense" rather than "let me tell you why you're wrong" because you do have a point that if you don't keep DX in mind at all in the early stages the end result tend to be unhappy-making, and, well, let's just say I speak from repeated experience of doing that that to myself ;)
[+] [-] dustingetz|3 years ago|reply
Here's a demo of React.js interop via Reagent, the wrapper is 25 LOC: https://gist.github.com/dustingetz/9854d23037b55bfab3845539f...
Our DOM module is only 300 LOC - it's bare metal DOM point writes + Electric (reactive language) + macros for JSX-y syntax. When the programming language itself is reactive, DOM rendering falls out for free.
Mechanically, Electric is comparable to Solid.js except the reactive engine is general purpose, not coupled to DOM rendering, which is a special case of incremental view maintenance.
Electric's reactive engine is https://github.com/leonoel/missionary ; Electric can be seen as a Clojure to Missionary compiler (if you ignore the distribution stuff - which btw is optional). Missionary is cross-platform functional effect and streaming system that today is tested on JVM, Node and Browser platforms. Missionary was created by Léo Noel, who is also the architect of Electric.
[+] [-] den1k|3 years ago|reply
[+] [-] scary-size|3 years ago|reply
A bit unsure where the observation of more offline capabilities comes from. I have the impression that things are moving in the opposite direction: Everything needs to be online.
[+] [-] yenda|3 years ago|reply
Here's what I did - killed the network - added a bunch of items to the todo list, which disappeared from the input field but didn't appear in the list - switched network back on, items appeared in the list
I'm curious to see how adding optimistic updates to the mix would work
[+] [-] kimi|3 years ago|reply
Not sure if it is too late, because Elixir's LiveView seem to have gotten important because it works around the same issue.
But it's time to push the wooden stake through the heart of the SPA (and the phone app too, but that's for another day).
[+] [-] incrudible|3 years ago|reply
[+] [-] i_cannot_hack|3 years ago|reply
Are there any mitigations in place to prevent this, except the obvious "be careful and don't write bad code"? Or have I misunderstood things completely and it's a non-issue?
[+] [-] adlpz|3 years ago|reply
It seems like you're doing SSR but not exactly.
You return the wrong prop in a getServerSideProps and boom, your stripe keys are dumped in a nice JSON for all to see.
[+] [-] dustingetz|3 years ago|reply
[+] [-] den1k|3 years ago|reply
[+] [-] prettyStandard|3 years ago|reply
I would imagine you can inspect the network traffic. I would hope it would be readable so that you could have a good idea of what's going on and errors would be more obvious.
[+] [-] emmanueloga_|3 years ago|reply
Your projects are very “futuristic” and the kind of thing I imagine may be hard to get financial backing for, because of the novelty and other considerations… so makes me curious how you balance innovation with monetization in your work.
Thank you!
[+] [-] zubairq|3 years ago|reply
[+] [-] eduction|3 years ago|reply
This make it sound like client and server segments are determined on the fly based on prevailing conditions, but then the sample code has forms wrapped with e/client and e/server. What do you mean when you say you determine “the optimal network cut” and how do you reconcile this with the expressly marked client and server code?
By the way I’ve been following this project for a while and am very excited about it, kudos on getting to this point.
[+] [-] dustingetz|3 years ago|reply
It gets interesting when you introduce lambdas, revealing that this optimization is non-local – you don't know if the transfer is superfluous until you know where the function is called from, and consider the caller and callee together.
The good news is that the DAG (Circuit really) contains everything there is to know about the data flow, so insert compilers research here and poof, insanely fast app.
[+] [-] beders|3 years ago|reply
Congratulations on the release! A great showcase for the flexibility of Lisp.
Others invent whole programming languages around a paradigm, while Lisp just marches on with s-expressions. Well done!
[+] [-] iLemming|3 years ago|reply
[+] [-] patates|3 years ago|reply
But on the first example, I see a lot of e/client and e/server calls. Am I misunderstanding this? If so, how is this different from meteor.js?
[+] [-] popinman322|3 years ago|reply
[+] [-] nih0|3 years ago|reply
[+] [-] den1k|3 years ago|reply
[+] [-] samsquire|3 years ago|reply
The idea is that we write what should be done and it is scheduled onto different systems and communication is inserted.
https://github.com/samsquire/structured-logging-2-system
I really like the idea of being capable of switching where something runs on the server or the client by changing a boundary.
[+] [-] waldrews|3 years ago|reply
[+] [-] the-alchemist|3 years ago|reply
[+] [-] butternoodle|3 years ago|reply
[+] [-] gfodor|3 years ago|reply
[+] [-] dustingetz|3 years ago|reply
[+] [-] tr14|3 years ago|reply