[yauneyz]

My professor has talked about this. He thinks that the real gem of the deep learning revolution is the ability to take the derivative of arbitrary code and use that to optimize. Deep learning is just one application of that, but there are tons more.

[SleekEagle]

That's part of why Julia is so exciting! Building it specifically to be a differentiable programming language opens so many doors ...

[mountainriver]

Julia wasn’t really built specifically to be differentiable, it was just built in a way that you have access to the IR, which is what zygote does. Enzyme AD is the most exciting to me because any LLVM language can be differentiable

[melony]

I am just happy that the previously siloed fields of operations research and various control theory sub-disciplines are now incentivized to pool their research together thanks to the funding in ML. Also many expensive and proprietary optimization software in industry are finally getting some competition.

[SleekEagle]

Hm I didn't know different areas of control theory were siloed. Learning about control theory in graduate school was awesome and it seems like a field that would benefit from ML a lot. I know they use RL agents for control networks for e.g. cartpole, but I would've thought it would be more widespread! Do you think the development of Differentiable Programming (i.e. the observation of more generality beyond pure ML/DL) was really the missing piece?

Also, just curious, what are your studies in?

[potbelly83]

How do you differentiate a string? Enum?

[6gvONxR4sf7o]

The answer to that is a huge part of the NLP field. The current answer is that you break down the string into constituent parts and map each of them into a high dimensional space. “cat” becomes a large vector whose position is continuous and therefore differentiable. “the cat” probably becomes a pair of vectors.

[geysersam]

Not all functions are differentiable.

Sometimes there are other better ways to describe "how does changing x affect y". Derivatives are powerful but they are not the only possible description of such relationships.

I'm very excited for what other things future "compilers" will be able to do to programs besides differentiation. That's just the beginning.

[titanomachy]

If you were dealing with e.g. English words rather than arbitrary strings, one approach would be to treat each word as a point in n-dimensional space. Then you can use continuous (and differentiable) functions to output into that space.

[bmc7505]

https://en.wikipedia.org/wiki/Brzozowski_derivative

[adgjlsfhk1]

generally you consider them to be piecewise constant.