Livebook: A collaborative and interactive code notebook for Elixir

[dnautics]

"connect to an existing node" is going to be amazing.

I imagine two interesting uses for this.

1. producing reports. Connect a livebook session to your system in prod, run a livebook (which pulls data from DB eg), then get back a report. User engagement, cloud resources consumed, p90s, or an incident report...

2. manual database intervention. Suppose you need to manually make a batch of changes to the DB. It would be very cool if you could run a livebook inside of a sandboxed checkout (a temporary transaction) and write your db changes, then write your code that verifies that the db is in a state that you want it to be in. If you mess up your code, then you just terminate the transaction without committing. Then when you're happy with your notebook, you run it once with a commit at the end, and then save your notebook immutably in your records so that you can keep track of what you've done.

I think for intervening in prod, it would be very cool if someone could figure out how to extend erlang distribution over websockets so you don't have to figure out complicated mechanisms (open up ssh port, forward encrypted epmd, e.g.) to get into your system in prod

[qbasic_forever]

Be careful, websockets are tricky to secure and get right. It's easy to do something like forget to check the origin and now you've opened up a whole class of cross-site websocket scripting attacks. I love the idea of easy access and nice UI to prod machines, but put it behind a secure proxy or VPN layer and not facing the public internet.

[nonbirithm]

I wish nREPL would be standardized to be non-Clojure-specific as with LSP, so you could connect to a program written in any language with an nREPL server and poke at it. This is basically the Elixir version of that.

[devoutsalsa]

We use Elixir at work. I attach to a running sapp all the time. It’s great!

[namelosw]

Wow, this is great.

The fact is can connect to existing nodes also makes it can also be great for operation and administration - instead of building an awkward admin panel, you can just having a collection of scripts.

The Elixir community is insanely productive. Also, the Elixir system is impressively pretty balanced - productivity-wise like Rails, concurrent-wise like Erlang, and continuously brings novel yet practical ideas to life in a robust manner!

[CTmystery]

Awesome work! Towards the end, José hits on a couple points that Jupyter does not do well:

  - Minute 25: Saved notebook is readable (markdown)
  - Minute 23: Live collaborate on a single notebook
  - Minute 22: Visual indication that cell is stale after an upstream cell changes

[srush]

I've been playing around with some of these ideas for Python notebooks.

https://github.com/srush/streambook

This is a proof of concept that combines Jupytext for markdown readable notebooks with Streamlit for in-order execution+caching. More difficult to get some of the more advanced features of Pluto / Livebook due to Python state management.

Curious to hear any thoughts.

[losvedir]

I'm pretty excited about this. I'm admit I was skeptical when I saw a similar LiveView based notebook a couple months ago, and wondered why not just make a kernel for Jupyter.

But with the idea of real-time collaboration I get it. And with the momentum of Jose and team behind it, I think there's real promise here, especially once they figure out widgets and charts, which they have an open issue for.

I program in Elixir at work, so I usually have an iex shell open, but lately I've been using a LiveBook and it's great.

And I love the new Mix.install in Elixir 1.12 demonstrated here. Using a hex package in an adhoc way was a real shortcoming in Elixir before. Shows the power of having this thing developed by the language team.

[filmor]

There are already Jupyter kernels, IElixir and ierl (which I wrote). The latter supports Erlang, Elixir and LFE right now and supports connecting to another node as well.

https://github.com/filmor/ierl

[AlchemistCamp]

I'll admit that I was initially skeptical about Elixir for anything ML related when I first picked up the pragprog book on Elixir and Genetic Algorithms.

Since that time, things have moved quite a bit. Between the Erlang VM JIT, then Nx, and now this, it's starting to look hopeful!

I'm looking forward to digging into this.

[cancan]

Love how prolific the elixir community and especially José is. Also, if you are into liveview/elixir, definitely dive into the codebase. There is a small operational-transform lib built in there too.

[mstipetic]

These guys are amazingly productive, while also running a commercial company

[fastball]

This is software that directly helps their business, so it's not really productivity outside of their business endeavors.

[out_of_protocol]

Readable `.livemd` format looks lovely (regular markdown without extra elements)

http://lon-screenshots.s3.amazonaws.com/2021-04-18_Screensho...

[Pandabob]

Is it just me or is Elixir being talked/written about more in the past couple of months?

[cultofmetatron]

A bunch of startups including my own bet big on elixir 2 years ago and are starting to launch and gain prominence. the first wave of early adopters are coming out with battle stories to share and the conclusion is that elixir is the real deal.

[lawik]

It is growing and having good adoption. Not a huge language but it also has a fair share of novel features (being built on Erlang) that aren't commonly seen with other langs. So I think it might have a disproportionate visibility compared to size. Lots of work happening in the community, lots of ambitious ideas and releases.

[SatvikBeri]

There were 300 elixir stories in 2020, an average of roughly .8 per day. In 2021 there have been 89 elixir stories for 108 days, so about the same. (Source: hnsearch)

[cpursley]

Years.

[transfire]

All this cool Elixir news. I'm really having a hard time deciding between Elixir and Julia.

(For programs that I must have compiled to a distributable executable, I've pretty much settled on Crystal. I find it more productive and pleasent than Go or Rust.)

[dnautics]

As someone who started on julia first and then moved to elixir:

If you're doing computational/numerical stuff, stick with julia, for now. You really can't do that elixir, not today, and maybe not for at least another year or two. It will likely never be as good as julia since computation is not first class in the vm it runs on.

If you're doing or planning or doing anything with web, or orchestration, go with elixir.

[cultofmetatron]

if you want to build a saas, go with elixir. Its got a great batteries included web framework thats built to scale and an emerging suit of ml and linear algebra software coming out.

If you want to experiment with ML and scientific programming where you do a lot of matrix processing, go with julia. thats literally what its made for.

[thruflo22]

It’s not quite the same story but https://github.com/spawnfest/bakeware builds elixir releases into distributable executables.

[neolog]

https://github.com/JuliaLang/PackageCompiler.jl

[rkss]

Would like to know how Livebook achieve collaborate-editing. Does it use algorithms like CRDT or OT ?

[lawik]

OT. Using quill js on the frontend and a subset of that delta format handled on the backend. From conversation with Jose on Twitter.

[hades32]

Super nice! It's only missing auto-complete and it'll be light-years ahead of other tools on terms of usability

[chrismccord]

There’s already a PR for it with a video of autocomplete in action :)

https://github.com/elixir-nx/livebook/pull/208

[seanhandley]

So this is like Jupyter notebooks for Elixir?

[lawik]

With google docs style collaborative editing, can connect to an existing cluster to run code in it. You can run one instance of the Livebook application and have different livebook docs run for different Elixir projects you happen to be working on.

Not sure what parts are in Jupyter but I've heard collab editing isn't.

Pretty hype about trying this for some living docs and teaching. Will see if I can do anything neat with it. Already tried it some.

[matreyes]

If you compare the work done in both projects (github insights), you will feel the power of the Beam, Elixir, Phoenix and LiveView.

[rajandatta]

Yes - it would seem so. The support for custom run times is interesting and seems like an improvement. Not sure Jupyter notebooks do that - certainly not for all supported kernels.

[arthurcolle]

How to use dependencies within a livebook?

I just tried to use the enum_type (version 1.1.3) by just installing it inside the cloned livebook repo, but I'm getting the error that EnumType isn't loaded.

Why is this?

[aleDsz]

I'm using this tool to teach the basics of Elixir on my livestream. I think there's a lot of good usecases for Livebook, not only machine learning and stuff like that.

[ch4s3]

I'm not sure how I'm going to use this yet but I think there are going to be some great opportunities for prototyping here.

[cocco]

Using it everyfay. Greate tool.

[cocco]

Using it everyday. Great tool.

[spicyramen]

If I'm a data scientists with a bunch of existing Python notebooks can I use this product?

[astrowilson]

Not yet. It only supports Elixir for now (there is an issue for supporting other languages) and the notebook formats are different, so someone would also need to write a nbconvert to Markdown (.livemd).