In one of their examples, they note “They saw ratings hover around 60% with their original, in-house tech — this improved by 7-8% with GPT-2 — and is now in the 80-90% range with the API.”
Bloomberg reports the API is based on GPT-3 and “other language models”.
If that’s true, this is a big deal, and it epitomizes OpenAI’s namesake. The largest NLP models require vast corporate resources to train, let alone put into production. Offering the largest model ever trained (with near-Turing results for some tasks) is a democratization of technology that would otherwise have been restricted to well-funded organizations.
Although the devil will be in the details of pricing and performance, this is a step worthy of respect. And it bodes well for the future.
How is this "democratization"? OpenAI trains a model, then they make it available through an API. You have no say in what that model is trained on or how (other than to say whether they can use your data- but not how), neither can you modify the model according to your needs. And of course, with no ability to modify the product you're buying you have no opportunity to innovate. You can wrap it up in a different kind of application, sure, but the nature and number of applications that it can be wrapped up in is restricted by the abilities of the model and therefore is entirely dependent on the choices made by OpenAI.
Imagine MS saying they "democratised" operating systems because, hey, you can buy their binaries, so everyone can use their operating system. Compare that kind of "democratisation" with open source oSs.
No, the truth is that as more and more resources are necessary to wring the last few drops of performance out of the current generation of deep neural net models it is only large, well-funded companies that have the resources to innovate - and everyone else is forced to follow in their wake. Any expectations that progress would lead to "democratisation" of deep neural networks research has gone out the window.
> They saw ratings hover around 60% with their original, in-house tech — this improved by 7-8% with GPT-2 — and is now in the 80-90% range with the API.
> The F1 score of its crisis classifier went up from .76 to .86, and the accuracy went up to 96%.
> With OpenAI, Algolia was able to answer complex natural language questions accurately 4x as often as it was using BERT.
I think the most informative are the first two, but the most _important_ is the final comparison with BERT (a Google model). I am, uh, a little worried about how fast things will progress if language models go from a fun lil research problem to a killer app for your cloud platform. $10m per training run isn't much in the face of a $100bn gigatech R&D budget.
$10m per training run gets me a lot of engineering time to build our own version of this system and lease it to other customers. Just skip one training run and I've got a pretty good team.
Taken together, this seems to imply that GPT-3 was more intended for a SaaS such as this, and it's less likely that it will be open-sourced like GPT-2 was.
But since the resources required for training such a model are only available to well-funded entities, it seems like offering the model as an API while releasing the original source-code is the best practical method of getting the model into the hands of people who would otherwise not have access?
Looks like OpenAI is going head to head with huggingface.
This makes a lot of sense and it seems they are telegraphing to monetize what they have been doing. It also seems like this is why they don't release their models in a timely manner.
The notable difference is that the base Huggingface library is open source, so you could in theory build something similar or more custom to the OpenAI API internally (which then falls into the typical cost/benefit analysis of doing so).
Whoa -- Speech to bash commands? That's a pretty novel idea to me with my limited awareness of NLP. I could see this same idea in a lot of technical applications -- Provisioning cloud infrastructure, creating a database query.. Very cool!
Cool indeed! While language-to-code (where code is a regular, general-purpose language) has only recently started to be workable, text-to-SQL has been a long running application/research area for semantic parsing.
It is not a novel idea and I don't think it is practical. If the natural language was practical for bash we would already have already "list directory" instead of "ls" and so on. "ls" is just 3 keystrokes while the natural language option is 15, 5 times more.
OpenAI started off wide-eyed and idealistic but it made the mistake of taking on investors for a non-profit mission. A non-profit requires sponsors, not investors. Investors have a fiduciary responsibility to maximize profits, not achieve social missions of open AI for all.
I guess Sama plans on manufacturing growth metrics by forcing YC companies to pretend that they're using this.
Generic machine learning APIs are a shitty business to get into unless you plan on hiring a huge sales team and selling to dinosaurs or doing a ton of custom consulting work, which doesn't scale the way VCs like it to. Anybody who will have enough know how to use their API properly can jus grab an open source model and tune it on their own data.
If they plan on commercializing things they should focus on building real products.
Not everyone wants to be an admin to their infrastructure. Real existing services like Heroku and Squarespace exist as useful services because even though you might know how to design and build a website from scratch, sometimes you just need something done quickly without too much worrying about details of the system that do not matter for your project at this point. I really don't see how this wouldn't apply to AI projects as well.
I could make a much better site coding my own website from scratch and setting up servers myself, but for some projects I wouldn't even think about it that way, because using Heroku or Squarespace I can save a LOT of time and get the results I need much quicker.
> I guess Sama plans on manufacturing growth metrics by forcing YC companies to pretend that they're using this.
That's wrong in almost too many ways to list. Sam left YC over a year ago, nor would he do such a thing. Nor does YC have that kind of power over companies, nor would it use it that way if it did. That would be wrong and also dumb.
I imagine they’re considering offering GPT-3, which would be cost prohibitive to fine-tune for most people. I also I heard inference was too slow to be practical. Perhaps they have some FPGA magic up their Microsoft sleeves.
OpenAI started as a non-profit, went for-profit. Still owned by the big players.... Something isn't right.
Is OpenAI just a submarine so the tech giants can do unethical research without taking blame??? Its textbook misdirection, nonprofit and "Open" in the name, hero-esque mission statement. How do you make the mental leap from "we're non-profit and we won't release things too dangerous" to "JK we're for-profit and now that GPT is good enough to use its for sale!!". You don't. This was the plan the whole time.
GPT and facial recognition used for shady shit? Blame OpenAI. Not the consortium of tech giants that directly own it. It may just be a conspiracy theory but something smells very rotten to me. Like OpenAI is a simple front so big names can dodge culpability for their research.
I know it's trendy (and partly justified) to look down on OpenAI, but can you actually give any basis for this claim?
What kind of research is OpenAI doing that all the other big AI players (Google/DeepMind, FB, Microsoft) aren't also invested in?
And even if others are doing the same, what part of OpenAI's research do you consider unethical?
wow you just made the connection for me. GPT2 was too dangerous to release, and now GPT3 is so much better - is there no point at which things become too dangerous anymore? what was the conclusion on that one?
More fake news and generated AI content there is more people would stop trusting social media. It will saturate to that tipping point that humanity will need to find more genuine ways to communicate. So I say bring it on.
Based on my experience with non-profits, they are just like regular corps except they don't pay taxes, and they're always attached to a for-profit interest. The real community organizations don't tend to incorporate, as then you have to hire people to manage the corp or do it yourself.
This OpenAI work is almost certainly a way for these bigger corps to collude. Proving that would be impossible, though.
GPT-2 is hard to do "shady" things right now
(speaking from experience)[1] but maybe GPT-3 might do better?
I could get poems to generate well. Tweets were a bit harder but I don't think we are at the point where you could just use a generative model to fool people that would be cheaper than actually hiring someone to write fake news. (Also shameless plug below)
I think it's simply because OpenAI is fundamentally created and controlled by venture capitalists, and the tech they created turned out to be just too juicy an opportunity to not monetize.
I can’t say I blame them, when they realize they are sitting on the technological equivalent of a mountain of gold. What would you do?
Natural language search is approximately $100B business. This might be first AI application that changes the search landscape from 1990s and finally puts an end to the question “where is money in AI?”.
In NLP there is a very clear and powerful new paradigm: train a HUGE language model using vast amounts of raw text. Then to solve the problem of interest, either fine-tune the model by training on your specific dataset (usually quite small), or 0/1-shot the learning somehow.
The crucial question is : is this paradigm viable for OTHER types of data?
My hypothesis is YES. If you train a HUGE image model using vast quantities of raw images, you will then be able to REUSE that model to work for specific computer vision problems, either by fine-tuning or 0/1-shotting.
I'm especially optimistic that this paradigm will work for image streams from autonomous vehicles. Classic supervised learning has proved to be difficult if not impossible to get to work for AV vision, so the new paradigm could be a game-changer.
> My hypothesis is YES. If you train a HUGE image model using vast quantities of raw images, you will then be able to REUSE that model to work for specific computer vision problems, either by fine-tuning or 0/1-shotting.
This has been demonstrated for many years, it's not news. Many of the SOTAs like BiT require pretraining on JFT-300M, or Instagram, or what have you.
An API that will try to answer any natural language question is a mind blowing idea. This is a universal thinking interface more than an application programming one.
I just sent in a request to join the waiting list, for the company I work at, Kognity. The potential for this in the EdTech field is mindblowingly amazing!
There are a few good examples of educational help on the list but it's really only scratching the surface.
I'm really excited and hope Kognity and EdTech in general can use this for even more value-full (both for students and teachers) tasks soon.
OpenAI seems like a completely disingenuous organization. They have some of the best talent in Machine Learning, but the leadership seems completely clueless.
1) (on cluelessness) If Sama/GDB were as smart as they claim to be, would they not have realized it is impossible to run a non profit research lab which is effectively trying "to compete" with DeepMind.
2) (on disingenuity) The original openAI charter made OpenAI an organization that was trying to save the world from nefarious actors and uses of AI. Who were such users? To me it seemed like, entities with vastly superior compute resources who were using the latest AI technologies for presumably profit oriented goals. There are few organizations in the world like that, namely FAANG, and their international counterparts. Originally OpenAI sounded incredibly appealing to me, and a lot of us here. But if their leadership had more forethought, they would perhaps not have made this promise. But given the press, and the money they accrued, it has now become impossible to go back on this charter. So the only way to get themselves out of the whole they dug into was by making it into a for profit research lab. And by commercializing perhaps a more superior version of the tools Microsoft, Google and the other large AI organizations are commercializing, is OpenAI any different from them?
How do we know OpenAI will not be the bad actor that is going to abuse AI given their self interest?
All we have is their charter to go by. But given how they are constantly "re-inventing" their organizational structure, what grounds do we have to trust them?
Do we perhaps need a new Open OpenAI? One that we can actually trust? One that is actually transparent with their research process? One that actually releases their code, and papers and has no interest in commercializing that? Oh, that's right, we already have that -- research labs at AI focused schools like MIT, Stanford, BAIR and CMU.
I am quite wary of this organization, and I would encourage other HN readers to think more careful about what they are doing here.
What happened to working on AI for the good of humanity, including AGI, and making sure it didn’t fall into the hands of bad actors? Wasn’t that the original aspiration? Now this reads like next generation Intercom/Olark tools.
there's zero understanding in any of this. This is still just superficial text parsing essentially. Show me progress on Winograd schema and I'd be impressed. It hasn't got anything to do with AGI, this is application of ML to very traditional NLP problems.
I'm skeptical. Amazing progress has been made in the last 5-10 years but it still feels like we need more paradigm shifting in the ML/AI field. It feels like we're approaching the upper limits of what stuffing mountains of data into model can do.
But with the speed of the field, maybe we can figure it out in three years. It just seems like we're still missing some key components. Primarily, reasoning and learning causality.
This depends how you pronounce 'h'. If you pronounce it "aitch", then "an HTTPS" is correct. If you pronounce it "haitch", then "a HTTPS" is correct. There's no universal pronunciation, and therefore no universally-right answer.
I dunno - looks fairly representative of the Silicon Valley population demographic. Maybe not so much an issue with this company?
I notice (as usual) under-representation of women. But that's endemic to the industry.
Remember how Microsoft trained their bot from reddit comments and it went anti human? Well I guess I have to start dropping hints for the skynet in all my repos.
wildermuthn|5 years ago
Bloomberg reports the API is based on GPT-3 and “other language models”.
If that’s true, this is a big deal, and it epitomizes OpenAI’s namesake. The largest NLP models require vast corporate resources to train, let alone put into production. Offering the largest model ever trained (with near-Turing results for some tasks) is a democratization of technology that would otherwise have been restricted to well-funded organizations.
Although the devil will be in the details of pricing and performance, this is a step worthy of respect. And it bodes well for the future.
denster|5 years ago
We saw this OpenAI demo: https://player.vimeo.com/video/427943452
and were just blown away. Very cool!!
I guess a spreadsheet is never too old [1] to learn new tricks :)
[1] Founder of https://mintdata.com here, so a bit biased (& opinionated about) spreadsheets, take the above with a pound or 10 of salt.
[2] I've sent them this example how we'd invoke their APIs, hopefully they'll let us into the beta, fingers crossed :) https://mintdata.com/docs/learn/core-mechanics/work-with-dat...
YeGoblynQueenne|5 years ago
Imagine MS saying they "democratised" operating systems because, hey, you can buy their binaries, so everyone can use their operating system. Compare that kind of "democratisation" with open source oSs.
No, the truth is that as more and more resources are necessary to wring the last few drops of performance out of the current generation of deep neural net models it is only large, well-funded companies that have the resources to innovate - and everyone else is forced to follow in their wake. Any expectations that progress would lead to "democratisation" of deep neural networks research has gone out the window.
azinman2|5 years ago
grizzlemeelmo|5 years ago
andyljones|5 years ago
> They saw ratings hover around 60% with their original, in-house tech — this improved by 7-8% with GPT-2 — and is now in the 80-90% range with the API.
> The F1 score of its crisis classifier went up from .76 to .86, and the accuracy went up to 96%.
> With OpenAI, Algolia was able to answer complex natural language questions accurately 4x as often as it was using BERT.
I think the most informative are the first two, but the most _important_ is the final comparison with BERT (a Google model). I am, uh, a little worried about how fast things will progress if language models go from a fun lil research problem to a killer app for your cloud platform. $10m per training run isn't much in the face of a $100bn gigatech R&D budget.
grogenaut|5 years ago
minimaxir|5 years ago
Recently, OpenAI set the GPT-3 GitHub repo to read-only: https://github.com/openai/gpt-3
Taken together, this seems to imply that GPT-3 was more intended for a SaaS such as this, and it's less likely that it will be open-sourced like GPT-2 was.
wildermuthn|5 years ago
gwern|5 years ago
zitterbewegung|5 years ago
This makes a lot of sense and it seems they are telegraphing to monetize what they have been doing. It also seems like this is why they don't release their models in a timely manner.
minimaxir|5 years ago
mcrider|5 years ago
nickswalker|5 years ago
Some interesting papers and datasets:
NL2Bash: https://arxiv.org/abs/1802.08979
Spider: https://yale-lily.github.io/spider
fudged71|5 years ago
jorgemf|5 years ago
typon|5 years ago
spookyuser|5 years ago
say_it_as_it_is|5 years ago
gdb|5 years ago
m_ke|5 years ago
Generic machine learning APIs are a shitty business to get into unless you plan on hiring a huge sales team and selling to dinosaurs or doing a ton of custom consulting work, which doesn't scale the way VCs like it to. Anybody who will have enough know how to use their API properly can jus grab an open source model and tune it on their own data.
If they plan on commercializing things they should focus on building real products.
antris|5 years ago
I could make a much better site coding my own website from scratch and setting up servers myself, but for some projects I wouldn't even think about it that way, because using Heroku or Squarespace I can save a LOT of time and get the results I need much quicker.
dang|5 years ago
That's wrong in almost too many ways to list. Sam left YC over a year ago, nor would he do such a thing. Nor does YC have that kind of power over companies, nor would it use it that way if it did. That would be wrong and also dumb.
unknown|5 years ago
[deleted]
wildermuthn|5 years ago
eggsnbacon1|5 years ago
Is OpenAI just a submarine so the tech giants can do unethical research without taking blame??? Its textbook misdirection, nonprofit and "Open" in the name, hero-esque mission statement. How do you make the mental leap from "we're non-profit and we won't release things too dangerous" to "JK we're for-profit and now that GPT is good enough to use its for sale!!". You don't. This was the plan the whole time.
GPT and facial recognition used for shady shit? Blame OpenAI. Not the consortium of tech giants that directly own it. It may just be a conspiracy theory but something smells very rotten to me. Like OpenAI is a simple front so big names can dodge culpability for their research.
sailingparrot|5 years ago
I know it's trendy (and partly justified) to look down on OpenAI, but can you actually give any basis for this claim?
What kind of research is OpenAI doing that all the other big AI players (Google/DeepMind, FB, Microsoft) aren't also invested in? And even if others are doing the same, what part of OpenAI's research do you consider unethical?
> It may just be a conspiracy theory
Yea, it very much looks like that to be honest.
gobengo|5 years ago
swyx|5 years ago
aivosha|5 years ago
solinent|5 years ago
This OpenAI work is almost certainly a way for these bigger corps to collude. Proving that would be impossible, though.
zitterbewegung|5 years ago
I could get poems to generate well. Tweets were a bit harder but I don't think we are at the point where you could just use a generative model to fool people that would be cheaper than actually hiring someone to write fake news. (Also shameless plug below)
[1] 1400 - TALK.8 - "A way to make fake tweets using GPT2" - Joshua Jay Herman https://thotcon.org/schedule.html
throwaway7281|5 years ago
And yes, there is often no need to call something open explicitly, if it really is. Is into OpenOS, or just Linux?
markshepard|5 years ago
ape4|5 years ago
etaioinshrdlu|5 years ago
I can’t say I blame them, when they realize they are sitting on the technological equivalent of a mountain of gold. What would you do?
gumby|5 years ago
What alternatives do people like?
nutanc|5 years ago
Is it a confidence problem? Are the OpenAI folks not confident on a single use case? Or did I miss the live demo somewhere?
gdb|5 years ago
sytelus|5 years ago
krallistic|5 years ago
d_burfoot|5 years ago
The crucial question is : is this paradigm viable for OTHER types of data?
My hypothesis is YES. If you train a HUGE image model using vast quantities of raw images, you will then be able to REUSE that model to work for specific computer vision problems, either by fine-tuning or 0/1-shotting.
I'm especially optimistic that this paradigm will work for image streams from autonomous vehicles. Classic supervised learning has proved to be difficult if not impossible to get to work for AV vision, so the new paradigm could be a game-changer.
gwern|5 years ago
This has been demonstrated for many years, it's not news. Many of the SOTAs like BiT require pretraining on JFT-300M, or Instagram, or what have you.
canjobear|5 years ago
sytse|5 years ago
dalys|5 years ago
There are a few good examples of educational help on the list but it's really only scratching the surface.
I'm really excited and hope Kognity and EdTech in general can use this for even more value-full (both for students and teachers) tasks soon.
owenshen24|5 years ago
kamikazehosaki|5 years ago
1) (on cluelessness) If Sama/GDB were as smart as they claim to be, would they not have realized it is impossible to run a non profit research lab which is effectively trying "to compete" with DeepMind.
2) (on disingenuity) The original openAI charter made OpenAI an organization that was trying to save the world from nefarious actors and uses of AI. Who were such users? To me it seemed like, entities with vastly superior compute resources who were using the latest AI technologies for presumably profit oriented goals. There are few organizations in the world like that, namely FAANG, and their international counterparts. Originally OpenAI sounded incredibly appealing to me, and a lot of us here. But if their leadership had more forethought, they would perhaps not have made this promise. But given the press, and the money they accrued, it has now become impossible to go back on this charter. So the only way to get themselves out of the whole they dug into was by making it into a for profit research lab. And by commercializing perhaps a more superior version of the tools Microsoft, Google and the other large AI organizations are commercializing, is OpenAI any different from them?
How do we know OpenAI will not be the bad actor that is going to abuse AI given their self interest?
All we have is their charter to go by. But given how they are constantly "re-inventing" their organizational structure, what grounds do we have to trust them?
Do we perhaps need a new Open OpenAI? One that we can actually trust? One that is actually transparent with their research process? One that actually releases their code, and papers and has no interest in commercializing that? Oh, that's right, we already have that -- research labs at AI focused schools like MIT, Stanford, BAIR and CMU.
I am quite wary of this organization, and I would encourage other HN readers to think more careful about what they are doing here.
chillee|5 years ago
LockAndLol|5 years ago
agakshat|5 years ago
nick_araph|5 years ago
lerax|5 years ago
alphagrep12345|5 years ago
mcemilg|5 years ago
zmitri|5 years ago
danielscrubs|5 years ago
dmvaldman|5 years ago
Barrin92|5 years ago
chundicus|5 years ago
But with the speed of the field, maybe we can figure it out in three years. It just seems like we're still missing some key components. Primarily, reasoning and learning causality.
azinman2|5 years ago
ericlewis|5 years ago
historyremade|5 years ago
jfoster|5 years ago
https://en.wikipedia.org/wiki/OpenAI
Grimm1|5 years ago
jaimex2|5 years ago
'an' is only mean to proceed a vowel. Should say
"OpenAI technology, just a HTTPS call away"
chad_oliver|5 years ago
wfme|5 years ago
organicfigs|5 years ago
I remember coming across it not too long ago and felt unwelcomed/disappointed.
JoeAltmaier|5 years ago
forgot_user1234|5 years ago
freediver|5 years ago
shitgoose|5 years ago
brainless|5 years ago
I want to create a software that can generate new code given business case hints, by studying existing open source code and their documentation.
I know this is vague, but sounds like what we eventually want for ourselves right?
anaganisk|5 years ago
mrmonkeyman|5 years ago
[deleted]