AMD isn't even mentioned in this article, and I recently read that nobody uses AMD for machine learning. Is this only down to the proprietary Cuda having more momentum than OpenCL? I've used both, albeit not very much, and my impression was that Cuda was a bit easier to write but OpenCL didn't have any deal breakers. I know AMD cards were the most popular for crypto currencies, so it's not like they're always the slowest. Granted, I only have experience with each company's consumer cards, and I think Nvidia holds their GeForce cards back more than AMD does their Radeon cards concerning compute performance.
Why is Nvidia this dominant, and why are so many researchers using a proprietary API like Cuda when there's open alternatives?
Related to this, are anybody using Intel's Xeon Phi? Tianhe-2 [0], the worlds fastest supercomputer since June 2013 use them as co-processors, but I rarely if ever hear about them in other projects.
I have a little experience in GPU programming so I can address your first question.
;TLDR It is mostly due to momentum.
The difference in momentum stems from a difference in design paradigms. OpenCL is just a library that you can link to can compile using GCC. Because the code you're writing is C++ syntactically you're often jumping through a lot more hoops. Nvidia uses their own compiler, so when you're writing cuda programs you have something that looks a lot like C++, however they have the freedom to extend the syntax to ease development.
The marketing arm of Nvidia, combined with the ease of use helped get Cuda the defacto research platform. Once that ball was rolling, it becomes easier to find cuda teachers and cuda examples so it's naturally become more prolific.
Torch, Caffe, and all the other deep learning frameworks are waiting for OpenCL patches. OpenCL support would make a great contribution!
I know Theano has had OpenCL support in the oven for quite a while now.
But it's hard to justify investing heavily into OpenCL when all the serious deep learning labs have already sunk $X0,000 into buying NVidia Titan cards by now. To these people (who are the stakeholders), OpenCL wouldn't add any benefit.
> Why is Nvidia this dominant, and why are so many researchers using a proprietary API like Cuda when there's open alternatives?
CUDA came out first and was really good, IMO. OpenCL is also really good, but doesn't get as much love from NVIDIA. NVIDIA really hypes CUDA and when they add features, they add them to CUDA first (and OCL never in some cases).
Why does NVidia dominate GPU computing? Where is AMD in this space?
I was under the impression that AMD's strongest R&D in the past few years has been directed at hybrid consumer desktops, surely this should translate directly into their server processing products.
IMO it's all due to the developer ecosystem. NVIDIA had a big head start with CUDA. AMD effectively embraced CUDA and transformed the best of CUDA into OpenCL (jointly with NVIDIA, Apple, via Khronos). NVIDIA's OpenCL support lags the standard, while they keep improving CUDA. NVIDIA probably feels commitment to their early adopters and doesn't want to divide up their engineering capability.
Meanwhile NVIDIA also seems to have really good marketing behind their GPU software stack. Lots of people who use CUDA have no idea that OpenCL exists and no idea that porting between CUDA and OCL is not super difficult. (It's been years since I've looked at CUDA it might have diverged more significantly since).
AMD's stability (on linux at least) is pretty poor. I don't know if it hurts adoption much, but it can't help.
Also, AMD absolutely dominates on integer workloads which is why it had been used so much for cryptocoin hashing machines prior to the widespread availability of ASICs for most algorithms. On floating point workloads, NVIDIA might do better than AMD, and those seem much more common in science than integer math.
Best quote:
"dealing with fglrx is murder. I have never installed an AMD graphics driver without repeated trial and error."
My own impression (using AMD consumer cards for OpenCL on Linux) is that AMD just do not give a damn - their documentation is way out of date, their support in the forums is poor/nonexistent, their installers are just broken most of the time, etc etc.
| hybrid consumer desktops, surely this should translate directly into their server processing products.
You mean their APUs? They're taking the worst of both worlds (AMD CPUs and integrated graphics) and selling them as a package. They're pointless, imo...
I've been ignoring GPUs for years, waiting for open standards, public documentation, and open-source drivers. I'm really disappointed that so much work is still being poured into these mysterious proprietary architectures. We wouldn't tolerate this anywhere else; why are GPUs allowed to be an exception?
Because ideology doesn't get work done. This is an example of making the perfect the enemy of the good. I assume you've been ignoring GPUs for years because you don't work in a problem space where you can leverage their power. Lots of other people however are in those spaces getting work done.
We use CST microwave studio (https://www.cst.com/Products/CSTMWS) for a project. Calculation time on a Xeon E5-2680 was about 50-60 hours per simulation, together with a Tesla K80 it was reduced to about 4-5 hours. Pretty impressive.
Adoption rate isn't very high, since most applications still don't use that power.
NVidia has been great for this. Back in 2009 they gave me 2 Teslas (C2060) and 2 Quadro FX 5800 cards. Then, I was able to get some results from those to get an NVidia Professor Partnership grant of $25000 (as a post doc). They had a large part in launching my engineering career doing HPC.
nVidia's Tesla processors and high-end GPUs are really stellar. We recently bought a machine with a Tesla K20c with some colleagues and the speed-up for neural net training has been tremendous.
E.g. for some of my nets (NLP tasks) training time has gone from hours to 15-30 minutes. This is great when you are just experimenting with different approaches to a problem and want to iterate quickly.
It's also great that nVidia hasn't been sitting on their laurels and released a library (cuDNN) of NN primitives that nearly everyone has been rebasing their packages on.
It seems to me that there would be a massive demand to be able to work entirely in OS X by researchers for whom computing is a secondary task (I've seen this a lot in my neurobiology department). Most of these people are only knee-deep in the computing world, and learning Linux is not worth it to them. I know several researchers just getting into machine learning for their image processing/CNN projects, and they're just living with the speed of a 3+ year old Mac Pro, though they have the funding and desire to add plenty of GPU acceleration. Is there something else out there or in development that would help here?
[+] [-] sorenjan|10 years ago|reply
Why is Nvidia this dominant, and why are so many researchers using a proprietary API like Cuda when there's open alternatives?
Related to this, are anybody using Intel's Xeon Phi? Tianhe-2 [0], the worlds fastest supercomputer since June 2013 use them as co-processors, but I rarely if ever hear about them in other projects.
[0] https://en.wikipedia.org/wiki/Tianhe-2
[+] [-] lordCarbonFiber|10 years ago|reply
The difference in momentum stems from a difference in design paradigms. OpenCL is just a library that you can link to can compile using GCC. Because the code you're writing is C++ syntactically you're often jumping through a lot more hoops. Nvidia uses their own compiler, so when you're writing cuda programs you have something that looks a lot like C++, however they have the freedom to extend the syntax to ease development. The marketing arm of Nvidia, combined with the ease of use helped get Cuda the defacto research platform. Once that ball was rolling, it becomes easier to find cuda teachers and cuda examples so it's naturally become more prolific.
[+] [-] gcr|10 years ago|reply
I know Theano has had OpenCL support in the oven for quite a while now.
But it's hard to justify investing heavily into OpenCL when all the serious deep learning labs have already sunk $X0,000 into buying NVidia Titan cards by now. To these people (who are the stakeholders), OpenCL wouldn't add any benefit.
[+] [-] wyldfire|10 years ago|reply
CUDA came out first and was really good, IMO. OpenCL is also really good, but doesn't get as much love from NVIDIA. NVIDIA really hypes CUDA and when they add features, they add them to CUDA first (and OCL never in some cases).
See my other comment for more - https://news.ycombinator.com/item?id=10554601
[+] [-] bjwbell|10 years ago|reply
AMD's software quality is not so great compared to Nvidia. Rich Geldreich's post, http://richg42.blogspot.com/2014/05/the-truth-on-opengl-driv..., is revealing.
[+] [-] DHJSH|10 years ago|reply
We get better performance writing CUDA code directly than OpenCL, and it's easier for us to develop and debug.
Yes, it's "proprietary" but we need to get our job done.
[+] [-] ihsw|10 years ago|reply
I was under the impression that AMD's strongest R&D in the past few years has been directed at hybrid consumer desktops, surely this should translate directly into their server processing products.
[+] [-] wyldfire|10 years ago|reply
Meanwhile NVIDIA also seems to have really good marketing behind their GPU software stack. Lots of people who use CUDA have no idea that OpenCL exists and no idea that porting between CUDA and OCL is not super difficult. (It's been years since I've looked at CUDA it might have diverged more significantly since).
AMD's stability (on linux at least) is pretty poor. I don't know if it hurts adoption much, but it can't help.
Also, AMD absolutely dominates on integer workloads which is why it had been used so much for cryptocoin hashing machines prior to the widespread availability of ASICs for most algorithms. On floating point workloads, NVIDIA might do better than AMD, and those seem much more common in science than integer math.
[+] [-] bainsfather|10 years ago|reply
Things have not changed since then.
Here is the author of the book "OpenCL in Action" discussing his problems with AMD's drivers: http://www.openclblog.com/2014/10/amd-and-opencl-20.html
Best quote: "dealing with fglrx is murder. I have never installed an AMD graphics driver without repeated trial and error."
My own impression (using AMD consumer cards for OpenCL on Linux) is that AMD just do not give a damn - their documentation is way out of date, their support in the forums is poor/nonexistent, their installers are just broken most of the time, etc etc.
[+] [-] prodmerc|10 years ago|reply
You mean their APUs? They're taking the worst of both worlds (AMD CPUs and integrated graphics) and selling them as a package. They're pointless, imo...
[+] [-] marssaxman|10 years ago|reply
[+] [-] jjaredsimpson|10 years ago|reply
[+] [-] v4n4d1s|10 years ago|reply
Adoption rate isn't very high, since most applications still don't use that power.
[+] [-] fmeyer|10 years ago|reply
[+] [-] jawilson2|10 years ago|reply
[+] [-] microtonal|10 years ago|reply
E.g. for some of my nets (NLP tasks) training time has gone from hours to 15-30 minutes. This is great when you are just experimenting with different approaches to a problem and want to iterate quickly.
It's also great that nVidia hasn't been sitting on their laurels and released a library (cuDNN) of NN primitives that nearly everyone has been rebasing their packages on.
[+] [-] PascLeRasc|10 years ago|reply
[+] [-] mtw|10 years ago|reply