Micrograd.jl

[anon389r58r58]

Almost feels like a fallacy of Julia at this point, on the one hand Julia really needs a stable, high-performance AD-engine, but on the other hand it seems to be fairly easy to get a minimal AD-package off the ground.

And so the perennial cycle continues and another Julia AD-package emerges, and ignores all/most previous work in order to claim novelty.

Without a claim for a complete list: ReverseDiff.jl, ForwardDiff.jl, Zygote.jl, Enzyme.jl, Tangent.jl, Diffractor.jl, and many more whose name has disappeared in the short history of Julia...

[0cf8612b2e1e]

I do not think this is meant to be a “real” library, but a didactic exercise inspired by Andrej Kaparthy‘s Python implementation.

[bdjsiqoocwk]

> Almost feels like a fallacy of Julia at this point, on the one hand Julia really needs a stable, high-performance AD-engine, but on the other hand it seems to be fairly easy to get a minimal AD-package off the ground.

Nothing wrong with this. Very successful languages like Python and JavaScript has tons of reimplementations of the same things because those languages are _easy_. This isn't a loss for the language it's a victory. If a lot of people can use it that's good.

As a consequence there will be lots of garbage libs, like there are lots of garbage libs in Python and JavaScript.

[xyproto]

Why did Julia select a package naming convention that makes every project name look like a filename?

[eigenspace]

It makes a julia package name very recognizable and easily searchable. It's actually something I really miss when I'm trying to look up packages in other languages.

[NeuroCoder]

I thought that was weird too but then I realized it was on of the most useful tools for searching stuff online and getting exactly what I wanted.

[stackghost]

Presumably for similar reasons to JavaScript, with names like Next.js, etc.

[infogulch]

Imagine searching for Plots.jl, Symbolics.jl, CUDA.jl if they didn't have the ".jl". I wish more package ecosystems used a convention like this.

[xiaodai]

I kinda gave up on Julia for deep learning since it’s so buggy. I am using PyTorch now. Not great but at least it works!

[barbarr]

I didn't find it too buggy personally, in fact it has an unexpected level of composability between libraries that I found exciting. Stuff "just works". But I felt it lacked performance in practical areas such as file I/O and one-off development in notebooks (e.g. plotting results), which is really important in the initial stages of model development.

(I also remember getting frustrated by frequent uninterruptible kernel hangs in Jupyter, but that might have been a skill issue on my part. But it was definitely a friction I don't encounter with python. When I was developing in Julia I remember feeling anxiety/dread about hitting enter on new cells, double and triple checking my code lest I initiate an uninterruptible error and have to restart my kernel and lose all my compilation progress, meaning I'll have to wait a long time again to run code and generate new graphs.)

[pkage]

Same here. I started my PhD with the full intention of doing most of my research with Julia (via Flux[0]), and while things worked well enough there were a few things which made it challenging:

- Lack of multi-GPU support,

- some other weird bugs related to autograd which i never fully figured out,

- and the killer one: none of my coauthors used Julia, so I decided to just go with PyTorch.

PyTorch has been just fine, and it's nice to not have to reinvent to wheel for every new model architecture.

[0] https://fluxml.ai/

[ssivark]

I think it would be more useful to list concrete bugs/complaints that the Julia devs could address. Blanket/vague claims like "Julia for deep learning [...] so buggy" is unfalsifiable and un-addressable. It promotes gossip with tribal dynamics rather than helping ecosystems improve and helping people pick the right tools for their needs. This is even more so with pile-on second hand claims (though the above comment might be first-hand, but potentially out-of-date).

Also, it's now pretty easy to call Python from Julia (and vice versa) [1]. I haven't used it for deep learning, but I've been using it to implement my algorithms in Julia while making use of Jax-based libraries from Python so it's certainly quite smooth and ergonomic.

[1] https://juliapy.github.io/PythonCall.jl/

[moelf]

what is so buggy, Julia the language or the deep learning libraries in Julia? in either case it would be good to have some examples.

[catgary]

Yeah I was pretty enthusiastic about Julia for a year or two, even using it professionally. But honestly, JAX gives you (almost) everything Julia promises and its automatic differentiation is incredibly robust. As Python itself becomes a pretty reasonable language (the static typing improvements in 3.12, the promise of a JIT compiler) and JAX develops (it now has support for dynamic shape and AOT compilation) I can’t see why I’d ever go back.

The Julia repl is incredibly nice though, I do miss that.

[enkursigilo]

Can you elaborate a bit more?

[thetwentyone]

Odd that the author excluded ForwardDiff.jl and Zygote.jl, both of which get a lot of mileage in the Julia AD world. Nonetheless, awesome tutorial and great to see more Julia content like this!

[fithisux]

Another testament to the awesomeness of Julia

[huqedato]

Julia is a splendid, high performance language. And the most overlooked. Such a huge pity and shame that the entire current AI ecosystem is build on Python/Pytorch. Python - not a real programming language, let alone is interpreted... such a huge loss of performance besides Julia.

[Y_Y]

I recognize the use of "not a real language" as traditional hyperbole[0]. I have my own gripes with python, even though it pays the bills, but this is just going to set off a load of people and is probably bad for discussion quality.

Ironically it's very hard to write actual low-level parallel code (like CUDA) through python, there's really no choice but to call out to Fortran and C libraries for the likes of pytorch.

[0] https://en.wikipedia.org/wiki/Real_Programmers_Don't_Use_Pas...

[brrrrrm]

It’s all about the kernels tho. The language doesn’t matter much. For the things that matter, everything is a dispatch to some cuda graph

I’m not really a fan of this convergence but the old school imperative CPU way of thinking about things is dead in this space

[xaellison]

as a major julia enthusiast I gotta say this is not how you get people to check it out buddy

[FranzFerdiNaN]

> not a real programming language

Really? Why do you feel the need to say this? Not liking Python, sure, but this kind of comments is just stupid elitism. What's next, the only REAL programmers are the ones that make their own punch cards?

[atoav]

Python is not a real programing language? That must come as a shocking revalation to the many thousand people running it successfully in production. /s

As someone who programs C/C++/Python/Rust/JS you had me curious in the first half of the post. But that comment makes me wonder about the quality of the rest of what you're saying.