I'm an undergrad student, and I'm nervous about picking between Tensorflow+Keras over PyTorch.
It looks like many more companies are hiring for TensorFlow, and there's a wealth of information out there on learning ML with it. In addition, it just got the 2.0 update.
But, PyTorch is preferred nearly every single time when I see the discussion come up on HN and Google searches. I'm having a hard time deciding what to dedicate my time to.
I posted this link but now the title has somehow changed. I do not know what is the policy on HN. But the title saying "[video]" might give a wrong impression that this points to a one hour long video. The link points to a tutorial which embeds an entirely optional two minute video that introduces the main content contained in five web pages.
One thing I've noticed is that it's quite hard to have vibrant discussions about DL because it is all either so simple or it is dauntingly complicated/unpredictable. Mostly my DL conversations end up being about frameworks. Anyone else experience this?
Also the number of DL submissions on HN seems surprisingly low given the applicability of the technology.
It's pretty easy when you're talking to people who understand the fundamentals of deep learning, but that understanding isn't very common even on HN. I think that's because the real-world, valuable usecases of DL are not very accessible:
(a) DL is pretty complicated in a way that's unfamiliar to most software engineers. You are consistently working with Tensors that have a couple more dimensions than people are used to holding in their heads (i.e. images mean you are typically working with 4D Tensors).
(b) You learn from academic papers, not blogs. It's a new workflow for many software people and intimidating to some (although the papers are usually closer to blog posts than rigorous academic papers).
(c) It's very difficult to learn deep learning on your own without it getting pretty expensive. Advanced uses pretty much require GPUs/TPUs and that's either a big upfront purchase or a serious per-experiment cost.
(d) Deep Learning is not a single field. It is CV, NLP, RL, speech recognition and probably others I'm forgetting about. They overlap, but it further reduces the number of people you can have informed discussions with because being knowledgeable about computer vision does not mean you are able to have a vibrant discussion about NLP.
These are “hands on” in the sense that you can replicate the results just by pasting in the same code. It’s kind of like a tutorial notebook in essay form.
In my experience the best place to have informal ai discussions is Twitter. The community is shockingly helpful. Follow @jonathanfly, @roadrunning01, @pbaylies and whoever pops up in the stuff they post. Roadrunning in particular posts tweets of the form “here’s some research; here’s the code” often with an interactive notebook.
Keep in mind that tutorials will always make it look easy compared to debugging actual production code. If you look through tensorflow tutorials, they also look very easy, especially with TF2.
That said, I've experimented with pytorch and I agree that it is really nice to work with.
Disclaimer: I work at Google and do use tensorflow, though I don't work on the tensorflow team.
Very good that Pytorch emerged as a serious contender to TF. While TF still provided more production grade tools (TFX, TensorRT, TF serving), Pytorch continue to evolve and hope soon we have a more complete ecosystem
I really like JAX as well: https://github.com/google/jax.
It's younger than PyTorch and TF, but feels cleaner and more expressive. It has a very nice autodiff implementation (based on https://github.com/HIPS/autograd) and performance is comparable to TF in my experience.
Is there a drop in replacement for TensorBoard? It’s probably the biggest thing keeping me using tensorflow. Ideally the api of the pytorch equivalent would be about the same too.
Anyone have thoughts on TF2.0 vs pytorch? Over on Twitter people seem to be pretty hyped about TF2.0, but when I tried learning it it just felt... not very fun. I need to give it a fair shot though.
Anyone know somewhere that has a good overview of the various ML and DL model types and what they are good for? I've been looking for a survey paper or book or just a glossary of ML.
When you hear autoregressive model, think “predicting a sequence”. These are good for text to speech since you can say “given some text, generate a spectrogram.” GPT-2 is probably the most impressive example of autoregressive techniques (I think).
GANs, and especially stylegan, are good for generating high quality images up to 1024x1024. These take about 5 weeks to train and $1k of GCE credits. The dataset size is around 70k photos for FFHQ. Mode collapse is a concern, which is when the discriminator wins the game and the generator fails to generate anything that can fool it. Stylegan has some built in techniques to combat this. IMLEs recently showed that mode collapse can be solved without gans at all.
Hmm.. what else... I’ll update this as I think of stuff. Any questions?
EDIT: Regarding IMLE vs GAN, here are some resources:
For comparing images, I believe they use the standard VGG perceptual loss metric that StyleGAN uses. (See section 3.5 of https://arxiv.org/pdf/1811.12373.pdf)
It seems to me that the main disadvantage of IMLE is that you might not get any latent directions that you get with StyleGAN. E.g. I'm not sure you could "make a photograph smile" the way you can with StyleGAN. But in the paper, they show that you can at least interpolate between two latents in much the same way, and the interpolations look pretty solid.
Not as far I know. It does have max-margin loss [1], which is pretty much all you need to implement a neural ranking model, apart from data iterators, and training loops.
Does no one build their own ml algos anymore? I don't understand the need for pytorch and tensor flow. I honestly thought tensor flow was nothing but a teaching thing for undergrads
This type of reasoning can be extended to any high-level tool. " Does no one writes there own OS. I don't understand the need for Linux or windows. I honestly thought windows or linux was nothing but a tool for undergrads to use Excel or host a WordPress site". And this is not a caricature of your argument. There is a lot of stuff under the hood that Tensorflow or Pytorch implement for a programmer. So much so that people have written wrapper for using TF or Pytorch to even further abstract the working of the library. Implementing deeplearning architecture is less of a science and more of a "let me try this or that" and iterating ideas quickly if of the utmost importance. Also, I can implement a neural network in C (CUDA) although not the auto diff part, but I could if given time to research) but if I started implementing my own library, it would take an order (or even more) of magnitude more time to do the stuff I do daily. We don't need to reinvent the wheel here guys.
Do you also write your own automatic differentiation tools? Using libraries like TF and PyTorch makes sense if you use neural networks because they provide automatic differentiation (who wants to write out their gradients by hand?) and standard neural network components.
Edit: If your algorithm is not using neural networks, then libraries like TF may or may not be a good fit, it depends on the algorithm.
Writing custom low-level code can still make sense in those cases.
Not all of us need to build their own ML algos. Just in the same way that not all of us need to build their sorting libraries or data structures. Some people are specialized in this to develop and do research. While other software engineers just want something they can use without much hassle and just a superficial understanding.
They're frameworks which implement high performance tools commonly used in ml problems like tensor operations, automatic differentiation, various gradient descent optimisers, and also neural network building blocks
I am sure you could write stuff like Diffentiable Processors or the like from scratch with numpy but if you respect yourself and your time, you won’t. Complicated architectures are orders of magnitude harder than writing feed forward networks from scratch. For example, see the Merlin paper.
[+] [-] rayalez|6 years ago|reply
https://www.youtube.com/playlist?list=PLZbbT5o_s2xrfNyHZsM6u...
They explain things incredibly well, videos are easy to understand, engaging, and to the point. Highly recommend it to everyone!
I've also heard that Udacity has some good courses, but I can't vouch for those yet.
[+] [-] p1esk|6 years ago|reply
[+] [-] TruckingThrow|6 years ago|reply
I'm having to learn this framework for a course assignment, and I feel a lot better about it now than I did after going through the OP.
Thanks for sharing!
[+] [-] M5x7wI3CmbEem1O|6 years ago|reply
I'm an undergrad student, and I'm nervous about picking between Tensorflow+Keras over PyTorch.
It looks like many more companies are hiring for TensorFlow, and there's a wealth of information out there on learning ML with it. In addition, it just got the 2.0 update.
But, PyTorch is preferred nearly every single time when I see the discussion come up on HN and Google searches. I'm having a hard time deciding what to dedicate my time to.
[+] [-] vyuh|6 years ago|reply
[+] [-] jdoliner|6 years ago|reply
[+] [-] hoseja|6 years ago|reply
[+] [-] amelius|6 years ago|reply
Also the number of DL submissions on HN seems surprisingly low given the applicability of the technology.
[+] [-] solidasparagus|6 years ago|reply
(a) DL is pretty complicated in a way that's unfamiliar to most software engineers. You are consistently working with Tensors that have a couple more dimensions than people are used to holding in their heads (i.e. images mean you are typically working with 4D Tensors).
(b) You learn from academic papers, not blogs. It's a new workflow for many software people and intimidating to some (although the papers are usually closer to blog posts than rigorous academic papers).
(c) It's very difficult to learn deep learning on your own without it getting pretty expensive. Advanced uses pretty much require GPUs/TPUs and that's either a big upfront purchase or a serious per-experiment cost.
(d) Deep Learning is not a single field. It is CV, NLP, RL, speech recognition and probably others I'm forgetting about. They overlap, but it further reduces the number of people you can have informed discussions with because being knowledgeable about computer vision does not mean you are able to have a vibrant discussion about NLP.
[+] [-] sillysaurusx|6 years ago|reply
https://www.gwern.net/GPT-2
https://www.gwern.net/Faces
These are “hands on” in the sense that you can replicate the results just by pasting in the same code. It’s kind of like a tutorial notebook in essay form.
Speaking of tutorial notebooks, pbaylies’ stylegan-encoder is quite good and you can run it on colab: https://colab.research.google.com/github/pbaylies/stylegan-e...
(Set runtime to GPU up in the menu.)
https://github.com/pbaylies/stylegan-encoder
In my experience the best place to have informal ai discussions is Twitter. The community is shockingly helpful. Follow @jonathanfly, @roadrunning01, @pbaylies and whoever pops up in the stuff they post. Roadrunning in particular posts tweets of the form “here’s some research; here’s the code” often with an interactive notebook.
[+] [-] abledon|6 years ago|reply
[+] [-] ftufek|6 years ago|reply
That said, I've experimented with pytorch and I agree that it is really nice to work with.
Disclaimer: I work at Google and do use tensorflow, though I don't work on the tensorflow team.
[+] [-] shmageggy|6 years ago|reply
[+] [-] spicyramen|6 years ago|reply
[+] [-] ibab|6 years ago|reply
[+] [-] sillysaurusx|6 years ago|reply
I answered my own comment before posting it. But in case it’s helpful to anyone else, I’ll put the answer here: yes, TensorBoardX. Looks like it’s very easy to use: https://tensorboardx.readthedocs.io/en/latest/tutorial.html
Anyone have thoughts on TF2.0 vs pytorch? Over on Twitter people seem to be pretty hyped about TF2.0, but when I tried learning it it just felt... not very fun. I need to give it a fair shot though.
[+] [-] mathusuthan|6 years ago|reply
[+] [-] emilfihlman|6 years ago|reply
[+] [-] faizshah|6 years ago|reply
[+] [-] sillysaurusx|6 years ago|reply
GANs, and especially stylegan, are good for generating high quality images up to 1024x1024. These take about 5 weeks to train and $1k of GCE credits. The dataset size is around 70k photos for FFHQ. Mode collapse is a concern, which is when the discriminator wins the game and the generator fails to generate anything that can fool it. Stylegan has some built in techniques to combat this. IMLEs recently showed that mode collapse can be solved without gans at all.
Hmm.. what else... I’ll update this as I think of stuff. Any questions?
EDIT: Regarding IMLE vs GAN, here are some resources:
Mode collapse solved (original claim): https://twitter.com/KL_Div/status/1168913453744103426
Overview of mode collapse, why it occurs, and how to solve it with IMLE: https://people.eecs.berkeley.edu/~ke.li/papers/imle_slides.p...
Paper + code: https://people.eecs.berkeley.edu/~ke.li/projects/imle/scene_...
Some simple code for reproducing IMLE from scratch (I haven't seen this referenced many other places; stumbled onto it by accident): https://people.eecs.berkeley.edu/~ke.li/projects/imle/
Super resolution with IMLE: https://people.eecs.berkeley.edu/~ke.li/projects/imle/superr...
For comparing images, I believe they use the standard VGG perceptual loss metric that StyleGAN uses. (See section 3.5 of https://arxiv.org/pdf/1811.12373.pdf)
It seems to me that the main disadvantage of IMLE is that you might not get any latent directions that you get with StyleGAN. E.g. I'm not sure you could "make a photograph smile" the way you can with StyleGAN. But in the paper, they show that you can at least interpolate between two latents in much the same way, and the interpolations look pretty solid.
[+] [-] BillFranklin|6 years ago|reply
[+] [-] geraltofrivia|6 years ago|reply
[1] (https://pytorch.org/docs/stable/nn.html?highlight=margin%20l...)
[+] [-] theemathas|6 years ago|reply
[+] [-] williom|6 years ago|reply
[deleted]
[+] [-] __Asturias__|6 years ago|reply
[+] [-] tsm212|6 years ago|reply
[+] [-] ibab|6 years ago|reply
Edit: If your algorithm is not using neural networks, then libraries like TF may or may not be a good fit, it depends on the algorithm. Writing custom low-level code can still make sense in those cases.
[+] [-] kiloreux|6 years ago|reply
[+] [-] TruckingThrow|6 years ago|reply
What I have to say is this: please don't build your own.
[+] [-] falkaer|6 years ago|reply
[+] [-] ghaff|6 years ago|reply
[+] [-] halfarmbandit|6 years ago|reply
I am sure you could write stuff like Diffentiable Processors or the like from scratch with numpy but if you respect yourself and your time, you won’t. Complicated architectures are orders of magnitude harder than writing feed forward networks from scratch. For example, see the Merlin paper.
[+] [-] mrfox321|6 years ago|reply
[+] [-] et2o|6 years ago|reply
[+] [-] mlevental|6 years ago|reply