From Python to Elixir Machine Learning

[bongobingo1]

Something not mentioned is Explorer (https://hexdocs.pm/explorer/Explorer.html) which is a very nice dataframe library backed by Rusts Polars https://www.pola.rs/ with great integrations for Livebook https://livebook.dev/.

[el_oni]

I found explorer quite frustrating. I've used polars in python and loved it, but I brought in some financial data and couldn't strip off "£" from the start of a string so I could go on to cast it to a number.

As far as I could tell I would have to bring that data into elixir, do the text processing and put it back into explorer, which to me defeated the whole point of a dataframe library.

I imagine it's good for precleaned data, using it with the built in datasets has been fine

[thibaut_barrere]

Don't miss the interesting book (which I just bought) in beta here:

https://pragprog.com/titles/smelixir/machine-learning-in-eli...

[ricketycricket]

Recommendation seconded. I'm only to chapter three due to time constraints, but it is well written and approachable for someone lacking ML experience.

[pawelduda]

I use Elixir for a long time and wouldn't recommend it for ML over Python. Yes, maybe Elixir has some advantages and solutions for pain points because of VM architecture and other subjective reasons but doesn't come close to what Python offers in terms of tooling, support, community. Also, if you juggle more complex data in Elixir, it's something that you need to get used to coming from Python, it will be much much different and therefore harder to grok for someone not used to the style.

You can always make Elixir app talk to Python ML backend and get the best of both worlds if you desire.

[bcardarella]

Nx solves all of the issues you're citing. That is how you do ML in Elixir

[dpflan]

Have you used Numerical Elixir (Nx)?

[pawelduda]

I can no longer edit parent post so to further elaborate on tooling: even if Elixir was comparable to Python (let's assume it's true as of today), there is a lot going on in AI, so that won't necessarily be true tomorrow. There is no doubt Elixir will always be slower to catch up to recent development because of smaller community. This also applies to how quickly issues will be resolved as they appear. It's likely OK if you're toying with a hobby ML project on your own, but not something you'd want in a startup with stakeholders who have expectations in regards to timeline, and so on...

I like Elixir for web development otherwise, it is a much more stable domain so above doesn't apply (although I've seen some claim otherwise, which is telling how much more of an issue it would be for niche ML use case).

I'd be very happy to be proven wrong by some case studies of how companies leveraged Elixir in real ML projects and concluded it is superior to Python.

[nesarkvechnep]

So you wouldn’t recommend it because it’s different than Python?

[jswny]

What ML tooling is better in Python?

[matt_daemon]

I wish there wouldn’t be such a song and dance about “moving away from Python”. There’s nothing wrong with creating ML tools in Elixir, but it’s always Python is slow, Python has no concurrency support, blah blah

[thibaut_barrere]

I come from Ruby but the reactions can be similar, happy to give my data point.

The thing is Elixir is really good at an increasing number of things.

If you need to write a HTTP proxy in the middle of your application, since Elixir processes & incoming HTTP workers are cheap, you do not need to go evented: it just works.

If you need to have reactive web apps with automated changes pushed to the client, it's the same: there is no need to external tools (e.g. any cable) at certain scale.

If you need to do some scripting, there is `Mix.install/2` for single-file dependencies description & use.

If you start crawling too much web pages or process to many APIs, the concurrency support kicks in and there is less need to scale (or later), turning into fewer machines, fewer ops problems (or delayed) etc.

And now you start being able to use MachineLearning, deploy the same type of code on GPU, embed Machine Learning models right in the middle of your web app without much work, etc, which in turns makes it a nice platform for apps / SaaS.

Elixir really is becoming a Swiss-army knife which scales easily :-)

[OJFord]

"Everything is working fantastically with our python ML project but we're rewriting it in Elixir anyway" would be a weird article wouldn't it?

[sodapopcan]

This article in particular doesn't feel like there's any song and dance. The very first line is directed at people already using Elixir who are looking to stay in Elixir-land while getting deeper into ML.

[substation13]

Those are real issues though.

[whalesalad]

Python is slow and concurrency is not great.

[throwawaymaths]

lots of negativity in here from people who have invested a lot of effort learning python, despite that there's also people who complain a lot about ML/DS in python. Is it an expression of sunk cost fallacy? If you don't support people who are trying to make a difference, nothing will change.

[mrdoops]

It's a weird mental block too - learning a language like Elixir, being so well designed and documented, is as comfortable as it gets.

Devs we hire without direct Elixir experience pick it up really quick (within a couple weeks). The energy needed to "get good" with Elixir is really not much considering it provides veritable super powers on the backend and introduces a whole category of concurrency concepts that are not easy to grasp elsewhere.

[paulluuk]

Show me I can save a considerable amount of time training a 70b parameter huggingface model by switching to Elixir, then we can talk about me learning a new language.

[josevalim]

The training costs and performance at that scale are mostly driven by hardware and I personally don't think language will make a large difference (I may be wrong).

Most of the Elixir Nx efforts are on inference, especially on how you can embed and scale it using concurrent and distributed patterns (see this post/video [1]). It may not be what you are looking for but we have more folks deploying than training models, so maybe they will find incentives to give Elixir a try. :)

[1]: https://news.livebook.dev/distributed2-machine-learning-note...

[sodapopcan]

Seems like you didn't read even the first line as this article is directed at people already using Elixir who are currently doing ML stuff with Python.

[madspindel]

Instead of moving to Elixir I believe it makes more sense to wait/move to Mojo when it's ready:

https://www.modular.com/mojo

[weatherlight]

Elixir (NX) is ready now, Mojo might be, at some time in the future.... also Mojo won't have the BEAM or ergonomics around Functional Programming.

[mrdoops]

Mojo is pretty cool but its a difficult road to take - especially to implement the breadth of accrued functionality of a 20 year old language like Python.

Elixir & Python are not an apples to apples comparison - there are fundamental differences in the programming model (functional, immutability, etc) and runtime (preemptive scheduling + OTP) that is the reason it has distinct advantages not available without heavy cost trade-offs elsewhere.

Either way once Mojo is production ready Elixir will be able to use it as well like it does Rust, Zig, or Python.

[pjmlp]

Given how Swift for Tensorflow went, how openai was equally pushing for it, I would rather take a "wait and see" approach to Mojo.

[throwawaymaths]

Since it aims to be transparently compatible with python Mojo will very likely suffer from the worst part of python: packaging

Last week heard a story about an ML dev that would literally rebuild his system every week because python would break it

[ilikehurdles]

No source code, no thanks.

[mkranjec]

At first I thought what does Nx (nrwl/nx) has to do with machine learning, then I figured context is about Elixir lib named Nx.

Naming things is hard.

[dsiegel2275]

Agreed, naming things is hard. That and off by one errors are the three hardest things in software development, IMO.

[ranjanprj]

I just ran a 2006 Java code to detect and read car license plate, and it ran in the first run on Java 17. The code is 17 years old and runs just fine without any issues. I think Java folks messed up AI/ML space due to licensing. But I still think it's best PL for AI/ML.

[frakt0x90]

What makes it the best language for AI and ML? I've never heard that take so just curious your thoughts. It's nice that legacy code still runs but that's usually not people's concern in ML.

[realusername]

You would be happy in Elixir as well, the language is pretty much frozen and very small bug fixes and features are added.

It's definitely not a fast moving language.