I've used tensorflow in 2016-2018 and switched to Pytorch a few months ago. My theory is that TF is used mainly because it is supported by Google, even if it is really badly designed for practitioners. In fact, TF was not meant to be public but was at first some internal tool at Google. This would explain why people at Google itself started to develop Keras or Sonnet (deepmind) at some point.
For anyone interested in research in deep learning with a python-numpy like environment, I can only recommend to switch.
We used to be a TensorFlow shop. Last year we started playing around with Pytorch, only for R&D projects at first... and it felt like a breath of fresh air. Whereas TensorFlow always felt like it was designed and engineered from day one for massive scalability and deployment flexibility (at the expense of easy/fast development), Pytorch felt instantly like it was designed for easy/fast development by AI researchers who need to experiment and iterate through models as quickly as possible with as little hassle as possible. Almost overnight, we stopped tinkering and experimenting with TensorFlow.
Earlier this year, after a blog post announcing the production-friendly features of Pytorch 1.0[a], we decided to switch our production systems from TensorFlow to Pytorch. So far we're happy with the decision.
But why? I do use Tensorflow form the 0.x release and I'm still using it right now, migrating all the codebase to the 2.x version.
I can do everything in both pytorch and tensorflow and when I have to define really efficient input pipelines (tf.data is a great thing), parallelize and distribute the training and export a trained model to production... With tensorflow everything is easier.
Moreover, pytorch in 1.x will have static-graph too, exactly like Tensorflow.
Both frameworks are converging to something really similar. I don't see a reason to switch (right now)
Keras wasn't developed by Google, it was adopted by TF.
> My theory is that TF is used mainly because it is supported by Google, even if it is really badly designed for practitioners.
This is not correct. TF is mainly used because it's designed for industrial-strength machine learning. It provides primitives with an eye to scale from the outset (because it's Google).
It's probably true that prototyping / research was not the main audience. That's exactly why Keras was adopted, as well as features like tf.eager, to abstract away the underlying computation graphs and make it easy for people to try different things.
Well-designed primitives / abstractions are important; Tensorflow does this well.
I am probably an outlier (I have experience with both compiler development and lazy languages) but I found the graph model to be very natural. Much of the struggles I saw people having conceptually I had already had years before.
The reason I bet on Tensorflow originally I admit was it was built by Google. But another really strong reason is it had a fully integrated toolchain from research to production. That toolchain has gotten much stronger with the continued investment in tf.data and tensorflow-serving. There was a binary alternative to CSVs (TFRecords). Also given some of our needs with some pretty esoteric components the tf community is actually quite deep.
I had only minor python experience before starting with tensorflow so I literally did not want a python-numpy environment. I am still sad that the world has converged on python as the language of choice but that ship seems to have sailed.
I will say tf.while is something I won't miss, as is tf.cond. I didn't find either really difficult, but it was more typing than was necessary. If there is something I really think should be invested in it is documentation. What should be in a SessionRunHook and what is the best way to write them? If you dig through some of google's repos that use TF estimators you find a lot of useful hooks as well as utility functions to help build them. But learning how to use them was more painful than it should have been.
All that said I am not a researcher. I would suspect that some of my workflow overlaps with researchers as I have found myself implementing a number of papers. I also need the ability to rapidly move a model from training to production.
> My theory is that TF is used mainly because it is supported by Google, even if it is really badly designed for practitioners.
I use TensorFlow via Keras because there are a lot of concise examples, Q&As and tutorials on it for different use cases while Pytorch looks a little bit like a magic black box you can't use before you take a course and study it as a whole. I believe this is a major point for many. Also the first Pytorch [official] tutorial I've found mentioned an nVidia GPU as a prerequisite and I don't have a GPU, just old built-in Intel Graphics with Core 2 Duo and TensorFlow seems having no problems with this.
> Tensorflow 2.0 will be a major milestone for the most popular machine learning framework: lots of changes are coming, and all with the aim of making ML accessible to everyone. These changes, however, requires for the old users to completely re-learn how to use the framework
I never felt like I was not re-learning tensorflow! A constant series of breackage, deprecations, new apis etc.
I think that’s a property of Google though, because there’s never been a google service for which I’ve read the documentation and not been like “I do not understand what’s going on at all” for a non-trivial amount of time.
And all their examples for Python are just like “run this magic-code-ridden python file, congrats you did the tutorial”.
I feel it's a bit unfair to criticise changes in a product at the leading edge of the most dynamic field of software. It's also slightly ungrateful for such a valuable addition to OSS–and I say that even though I tend to consider this argument overused.
But mostly I don't believe such criticism has much of chance of changing anything: The creators are already the most qualified you could have working on it, and it's unlikely they intentionally made mistakes in the early rounds of API design. Maybe it would have helped to spend a few more weeks on specifications rather than coding but, again, I wouldn't feel very comfortable second-guessing Google's project management.
I think the most likely outcome if such complaints start to pile up would be future projects just remaining proprietary, either longer, of completely.
Not trying to start a language war, but I would like the old or the new api better if it had static types. Right now it's thousands of functions that can take thousands of different objects, and I would never know what to use except the few I randomly learned in a tutorial.
But the API is strictly statically typed in reality.
The core is written in C++ and every operation requires a well-known type in input. As you can see from the error raised in this example: https://pgaleone.eu/tensorflow/go/2017/05/29/understanding-t...
And no.. it's not simply a Swift wrapper for tensorflow, it's literally Tensorflow (or generally speaking: differentiable calculation graphs) built into the Swift language.
This is a great development for both new and experienced users. Most importantly, it seems to be an effort to introduce a canonical way to build a given network in Tensorflow, as opposed to the previous era where any given TF repo might implement the same network in one of several completely different ways (tf.contrib.slim, tf.layers, etc). Hopefully, defining a standard way to build models will also help accelerate the standardization of train/test scripts and data pipelines, putting to rest the era of models being deeply infected with the execution structure built around them.
Overall, it looks like Pytorch is forcing TF devs to focus more on users and usability, and I'm excited to see how they continue to spur each other's growth.
> Support for more platforms and languages, and improved compatibility and parity between these components via standardization on exchange formats and alignment of APIs.
Does this mean we can get a pure C API for training and running? Right now, IIRC you can only train in Python and C++ which ignores a large part of the programming community. If not a pure C FFI API, what is the approach being taken?
How about making it more stable? Kind of tired to close 10 issues a day on github because tensorflow developers decided to rename a variable. Seriously looking on pytorch.
Tangential, but as someone in the southern hemisphere it's frustrating when people use seasons for timelines. "Q2 2019" is immediately more understandable instead of having to do some mental gymnastics to convert it.
However, migrating to a keras-like approach, you can work thinking about "objects with variables inside" and then the graph will be built for you by the abstraction introduced by `tf.keras.Model`.
However, for automatic differentiation, graph is always required (as you can see from the example that uses eager)
A very common compiler architecture for functional languages is to use a combinatory language (dataflow like graphs) as the IR, and have a more usable frontend language that has variables and binders.
I think they should really consider making a clean JavaScript/typescript api first — use that and test for awhile - then port that design to python for a 2.0 of the python api...
The worst thing about tensorflow is being locked into having to use python for graph description and training (in my opinion) ...
By the sake of god, please try to add also a more friendly build system too. Bazel is a disaster that would almost make maven looks lightweight, npm reliable and autotools userfriendly.
[+] [-] CretinDesAlpes|7 years ago|reply
For anyone interested in research in deep learning with a python-numpy like environment, I can only recommend to switch.
[+] [-] cs702|7 years ago|reply
We used to be a TensorFlow shop. Last year we started playing around with Pytorch, only for R&D projects at first... and it felt like a breath of fresh air. Whereas TensorFlow always felt like it was designed and engineered from day one for massive scalability and deployment flexibility (at the expense of easy/fast development), Pytorch felt instantly like it was designed for easy/fast development by AI researchers who need to experiment and iterate through models as quickly as possible with as little hassle as possible. Almost overnight, we stopped tinkering and experimenting with TensorFlow.
Earlier this year, after a blog post announcing the production-friendly features of Pytorch 1.0[a], we decided to switch our production systems from TensorFlow to Pytorch. So far we're happy with the decision.
[a] https://pytorch.org/2018/05/02/road-to-1.0.html
[+] [-] me2too|7 years ago|reply
I can do everything in both pytorch and tensorflow and when I have to define really efficient input pipelines (tf.data is a great thing), parallelize and distribute the training and export a trained model to production... With tensorflow everything is easier.
Moreover, pytorch in 1.x will have static-graph too, exactly like Tensorflow.
Both frameworks are converging to something really similar. I don't see a reason to switch (right now)
[+] [-] kajecounterhack|7 years ago|reply
> My theory is that TF is used mainly because it is supported by Google, even if it is really badly designed for practitioners.
This is not correct. TF is mainly used because it's designed for industrial-strength machine learning. It provides primitives with an eye to scale from the outset (because it's Google).
It's probably true that prototyping / research was not the main audience. That's exactly why Keras was adopted, as well as features like tf.eager, to abstract away the underlying computation graphs and make it easy for people to try different things.
Well-designed primitives / abstractions are important; Tensorflow does this well.
[+] [-] sseveran|7 years ago|reply
The reason I bet on Tensorflow originally I admit was it was built by Google. But another really strong reason is it had a fully integrated toolchain from research to production. That toolchain has gotten much stronger with the continued investment in tf.data and tensorflow-serving. There was a binary alternative to CSVs (TFRecords). Also given some of our needs with some pretty esoteric components the tf community is actually quite deep.
I had only minor python experience before starting with tensorflow so I literally did not want a python-numpy environment. I am still sad that the world has converged on python as the language of choice but that ship seems to have sailed.
I will say tf.while is something I won't miss, as is tf.cond. I didn't find either really difficult, but it was more typing than was necessary. If there is something I really think should be invested in it is documentation. What should be in a SessionRunHook and what is the best way to write them? If you dig through some of google's repos that use TF estimators you find a lot of useful hooks as well as utility functions to help build them. But learning how to use them was more painful than it should have been.
All that said I am not a researcher. I would suspect that some of my workflow overlaps with researchers as I have found myself implementing a number of papers. I also need the ability to rapidly move a model from training to production.
[+] [-] adamnemecek|7 years ago|reply
Sounds like every Google developed technology. Angular, Go, Android, the list is infinite.
[+] [-] qwerty456127|7 years ago|reply
I use TensorFlow via Keras because there are a lot of concise examples, Q&As and tutorials on it for different use cases while Pytorch looks a little bit like a magic black box you can't use before you take a course and study it as a whole. I believe this is a major point for many. Also the first Pytorch [official] tutorial I've found mentioned an nVidia GPU as a prerequisite and I don't have a GPU, just old built-in Intel Graphics with Core 2 Duo and TensorFlow seems having no problems with this.
[+] [-] solomatov|7 years ago|reply
Sonnet is based on TF
[+] [-] LeanderK|7 years ago|reply
I never felt like I was not re-learning tensorflow! A constant series of breackage, deprecations, new apis etc.
[+] [-] FridgeSeal|7 years ago|reply
I think that’s a property of Google though, because there’s never been a google service for which I’ve read the documentation and not been like “I do not understand what’s going on at all” for a non-trivial amount of time.
And all their examples for Python are just like “run this magic-code-ridden python file, congrats you did the tutorial”.
[+] [-] sebazzz|7 years ago|reply
[+] [-] matt4077|7 years ago|reply
But mostly I don't believe such criticism has much of chance of changing anything: The creators are already the most qualified you could have working on it, and it's unlikely they intentionally made mistakes in the early rounds of API design. Maybe it would have helped to spend a few more weeks on specifications rather than coding but, again, I wouldn't feel very comfortable second-guessing Google's project management.
I think the most likely outcome if such complaints start to pile up would be future projects just remaining proprietary, either longer, of completely.
[+] [-] maaaats|7 years ago|reply
[+] [-] me2too|7 years ago|reply
[+] [-] halflings|7 years ago|reply
https://www.tensorflow.org/swift/
And no.. it's not simply a Swift wrapper for tensorflow, it's literally Tensorflow (or generally speaking: differentiable calculation graphs) built into the Swift language.
[+] [-] lostmsu|7 years ago|reply
[+] [-] friendshaver|7 years ago|reply
Overall, it looks like Pytorch is forcing TF devs to focus more on users and usability, and I'm excited to see how they continue to spur each other's growth.
[+] [-] p1esk|7 years ago|reply
I'm pretty sure there will be plenty of incompatible API changes in every 2.* release, and a completely new "canonical" way introduced in TF 3.0.
[+] [-] kodablah|7 years ago|reply
Does this mean we can get a pure C API for training and running? Right now, IIRC you can only train in Python and C++ which ignores a large part of the programming community. If not a pure C FFI API, what is the approach being taken?
[+] [-] microtonal|7 years ago|reply
(Of course, you could also write Tensorflow protobuf directly, but that would be tedious.)
[+] [-] nine_k|7 years ago|reply
[+] [-] nshm|7 years ago|reply
[+] [-] juanuys|7 years ago|reply
https://github.com/tensorflow/tensorflow/issues/14248
[+] [-] beefsack|7 years ago|reply
[+] [-] qnsi|7 years ago|reply
[+] [-] sytelus|7 years ago|reply
[+] [-] davmar|7 years ago|reply
should i still use those outdates TF slim models that are for some reason kept in models/inception?
please, tensorflow team, make this stuff easier for us non-Phds.
[+] [-] etaioinshrdlu|7 years ago|reply
[+] [-] unknown|7 years ago|reply
[deleted]
[+] [-] amelius|7 years ago|reply
As a programmer, I don't want to think in terms of dataflow graphs. That is what compilers are for!
[+] [-] me2too|7 years ago|reply
However, migrating to a keras-like approach, you can work thinking about "objects with variables inside" and then the graph will be built for you by the abstraction introduced by `tf.keras.Model`.
However, for automatic differentiation, graph is always required (as you can see from the example that uses eager)
[+] [-] CMCDragonkai|7 years ago|reply
[+] [-] breatheoften|7 years ago|reply
The worst thing about tensorflow is being locked into having to use python for graph description and training (in my opinion) ...
[+] [-] mark_l_watson|7 years ago|reply
[+] [-] ericfriday|7 years ago|reply
[+] [-] adev_|7 years ago|reply
[+] [-] solomatov|7 years ago|reply
[+] [-] Judgmentality|7 years ago|reply
[+] [-] xvilka|7 years ago|reply
[+] [-] tmulc18|7 years ago|reply
[+] [-] shimnasuresh|7 years ago|reply
[deleted]