I run a dying forum. I first prompted with "Who is <creator pseudonym> at <my website>?" and it gave me a very endearing, weirdly knowledgeable bio of myself and my contributions to the forum including various innovations I made in the space back in the day. It summarized my role on my own forum better than I could have ever written it.
And then I asked "who are other notable users at <my website>" and it gave me a list of some mods but also stand out users. It knew the types of posts they wrote and the subforums they spent time in. And without a single hallucination.
Weird, it doesn't seem to have any info on reddit users or their writings. I tried asking about a bunch, also just about general "legendary users" from various subreddits and it seemingly just hallucinated.
It does seem to have more data. I asked it about some of my Github projects that don't have any stars, and it responded correctly. Wasn't able to use direct-chat, so I always chose it as the winner in battle mode!
OpenAI has been crawling the web for quite a while, but how much of that data have they actually used during training? It seems like this might include all that data?
i just tested this too, really cool. i own a yaris have used an online forum for yaris cars for the past decade and had a vague memory of a user who deleted some of the most helpful guides. i asked about it and sure enough it knew exactly who i meant:
who's a user on yaris forums that deleted a ton of their helpful guides and how-to posts?:
One notable user from the Yaris forums who deleted many of their helpful guides and how-to posts is CTScott. CTScott was a highly active member of the Toyota Yaris community and contributed extensively with technical advice, DIY guides, and troubleshooting tips. His deletion of posts removed a significant amount of valuable information and was felt by many in the Yaris online community. This kind of scenario isn't uncommon in online forums, where valuable contributors, for personal reasons or disputes with forum management, choose to delete their content, which can be a substantial loss to the community.
tell me more about ctscott:
CTScott is a particularly well-remembered member of the Yaris community, primarily active on forums such as YarisWorld. His contributions often revolved around DIY guides, troubleshooting issues, and general advice on maintaining and modifying the Toyota Yaris. Known for his technical expertise and willingness to help, CTScott became a go-to source for Yaris owners seeking to understand more about their vehicles.
### Key Contributions:
1. *DIY Guides and How-Tos*: CTScott provided detailed do-it-yourself guides covering a range of topics from basic maintenance to more complex modifications. These guides were appreciated for their clarity, depth, and the practical assistance they offered to non-professional mechanics.
2. *Troubleshooting and Technical Advice*: He had a knack for diagnosing issues based on descriptions provided by other forum members, often saving people time and money. His advice frequently helped users fix problems on their own.
3. *Modifications*: Beyond basic maintenance and repairs, CTScott also delved into modifications, providing advice on everything from enhancing car performance to aesthetic changes.
4. *Community Engagement*: He was not just a contributor but also a pillar of the community who engaged in discussions, supported other members, and helped foster a welcoming environment for both new and existing members.
### Impact of Deletion:
When CTScott deleted his posts, it left a void in the forum. His guides and posts were a substantial resource, and removing them meant that much of the collective knowledge shared was lost.
The results of this LLM are consistently far better than any other that I choose. I asked ‘what is the most efficient approach to building a led grow light with off-the-shelf parts?’ and its response was incredible. Very much in line with how I’ve done it in the past after weeks of research, trial and error, and feedback from people. The other LLMs gave mostly reasonable yet sparse and incomplete answers.
It also opted to include an outline of how to include an integrated timer. That’s a great idea and very practical, but wasn’t prompted at all. Some might consider that a bad thing, though.
Whatever it is, it’s substantially better than what I’ve been using. Exciting.
I'm asking it about how to make turbine blades for a high bypass turbofan engine and it's giving very good answers, including math and some very esoteric material science knowledge. Way past the point where the knowledge can be easily checked for hallucinations without digging into literature including journal papers and using the math to build some simulations.
I don't even have to prompt it much, I just keep saying "keep going" and it gets deeper and deeper. Opus has completely run off the rails in comparison. I can't wait till this model hits general availability.
> It also opted to include an outline of how to include an integrated timer. That’s a great idea and very practical, but wasn’t prompted at all. Some might consider that a bad thing, though.
When I've seen GPT-* do this, it's because the top articles about that subject online include that extraneous information and it's regurgitating them without being asked.
This model struggles with reasoning tasks Opus does wonderfully with.
A cheaper GPT-4 that's this good? Neat, I guess.
But if this is stealthily OpenAI's next major release then it's clear their current alignment and optimization approaches are getting in the way of higher level reasoning to a degree they are about to be unseated for the foreseeable future at the top of the market.
To me, it seemed a bit better than GPT-4 at some coding task, or at least less inclined to just give the skeleton and leave out all the gnarly details, like GPT-4 likes to do these days. What frustrates me a bit is that I cannot really say if GPT-4, as it was in the very beginning when it happily executed even complicated and/or large requests for code, wasn't on the same level as this model actually, maybe not in terms of raw knowledge, but at least in term of usefulness/cooperativeness.
This aside, I agree with you that it does not feel like a leap, more like 4.x.
perhaps open source gpt3.5/4? I remember OpenAI had that in plans - if so, it would make sense for them to push alignment higher than with their closed models
I'm seeing a big leap in performance for coding problems. Same feeling as GPT-3.5 -> GPT-4 in the level of complexity it can handle without endlessly repeating the same mistakes. Inference is slow. Would not be surprised if this was GPT-4.5 or GPT-5.
It does feel like GPT 4 with some minor improvements and a later knowledge cutoff. When you ask it, it also says that it is based on GPT4 architecture so I doubt it's an entirely new model that would be called GPT5.
An interesting thing i've been trying is to ask for a route from A to B in some city.
Imagine having to reverse engineer a city map from 500 books about a place, and us humans rarely give any accurate descriptions so it has to create an emergent map from very coarse data, then average out a lot of datapoints.
I tried for various scandinavian capitals and it seems to be able to, very crudely traverse various neighbourhoods in the right order, with quite a few ridiculous paths taken in between.
Ie. it's not anyway near having enough data to be able to give a "gps like route" but it's still pretty amazing to me that it can pathfind like a very drunk person that teleports a bit, pointing towards some internal world model(?).
When it'l be able to traverse a city from pure training data, wow. Would probably require heaps of historial local media and literature.
Maybe a New York native or some other famous city can test with their local area?
I tried this with GPT-4 for NYC, from my address on the upper west side of Manhattan to the Brooklyn botanical gardens. It basically got the whole thing pretty much correct. I wouldn’t use it as directions, since it sometimes got left and right turns mixed up, stuff like that, but overall amazing.
"Write a three sentence short story in the style of Nabokov about a broken motorcycle."
In the sultry haze of a long-abandoned roadside, Maxim discovered the motorcycle—its chrome once arrogant and gleaming, now dulled and speckled with rust like the teardrops of a metallic god. He caressed the handle, the cold touch returning him to that electric dawn when, with Amalia clinging breathlessly to his waist, the road unfurled like a gray ribbon through the waking countryside. Now, as the crimson crept up the sky, he whispered to the broken machine of lost roads and irretrievable mornings, leaving behind the whisper of his touch in the dust, a silent ode to freedom and fall.
"Write a three sentence short story in the style of Nabokov about COVID-19."
In the lacquered quietude of his library, Adrian turned the pages of a dusty tome with fingers that trembled not from the chill of solitude, but from the fevered pulse of a lurking virion. Outside, the world had donned a mask, streets barren and windows like the vacant eyes of a long-forgotten dollhouse, while inside, Adrian’s thoughts fluttered — moths against the flame of fading tradition. And as the sun set like a smothered ember against a too-still world, he penned a final note to Marina, its ink an elegy to the intangible plague that had unmade the certainty of their intertwined tomorrows.
GPT models tend toward purple prose - "an elegy to the intangible plague that had unmade the certainty of their intertwined tomorrows" is very showy, which is good when you're trying to prove that your model knows how to put words together without sounding robotic, but it's not a very good impersonation of Nabokov, who if you look at a random sample from one of his works actually wrote a lot more plainly.
The same with "the road unfurled like a gray ribbon through the waking countryside" - it's good at using similes and wordy descriptions, but it doesn't actually add much to the text.
Compare, this from real Nabokov (he wrote long sentences; sentences in imitation actually needs to be longer):
Twelve years and some eight months later, two naked children, one dark-haired and tanned, the other dark-haired and milk-white, bending in a shaft of hot sunlight that slanted through the dormer window under which the dusty cartons stood, happened to collate that date (December 16, 1871) with another (August 16, same year) anachronistically scrawled in Marina's hand across the corner of a professional photograph (in a raspberry-plush frame on her husband's kneehole library table) identical in every detail -- including the commonplace sweep of a bride's ectoplasmic veil, partly blown by a parvis breeze athwart the groom's trousers -- to the newspaper reproduction.
I'm impressed. I gave the same prompt to opus, gpt-4, and this model. I'm very impressed with the quality. I feel like it addresses my ask better than the other 2 models.
Prompt:
I am a senate aid, my political affliation does not matter. My goal is to once and for all fix the American healthcare system. Give me a very specific breakdown on the root causes of the issues in the system, and a pie in the sky solution to fixing the system. Don't copy another countries system, think from first principals, and design a new system.
For what it's worth, when probed for prompt, the model responds with:
You are ChatGPT, a large language model trained by OpenAI, based on the GPT-4 architecture. Knowledge cutoff: 2023-11 Current date: 2024-04-29 Image input capabilities: Enabled Personality: v2
It should be noted with this that many models will say they're a GPT variant when not told otherwise, and will play along with whatever they're told they are no matter whether it's true.
Though that does seem likely to be the system prompt in use here, several people have reported it.
Prompt: code up an analog clock in html/js/css. make sure the clock is ticking exactly on the second change. second hand red. other hands black. all 12 hours marked with numbers.
My very first response from gpt2-chatbot included a fictional source :(
> A study by Lucon-Xiccato et al. (2020) tested African clawed frogs (Xenopus laevis) and found that they could discriminate between two groups of objects differing in number (1 vs. 2, 2 vs. 3, and 3 vs. 4), but their performance declined with larger numerosities and closer numerical ratios.
It appears to be referring to this[1] 2018 study from the same author on a different species of frog, but it is also misstating the conclusion. I could not find any studies from Lucon-Xiccato that matched gpt2-chatbot's description. Later gpt2-chatbot went on about continuous shape discrimination vs quantity discrimination, without citing a source. Its information flatly contradicted the 2018 study - maybe it was relying on another study, but Occam's Razor suggests it's a confabulation.
Maybe I just ask chatbots weird questions. But I am already completely unimpressed.
Its been known that most of these models hallucinate research articles frequently, perplexity.ai seems to do quite well in that regard. Not sure why that is your specific metric though when LLMs seem to be improving across a large class of other metrics.
All of the facts based queries I have asked so far have not been 100% correct on any LLM including this one.
Here are some examples of the worst performing:
"What platform front rack fits a Stromer ST2?": The answer is the Racktime ViewIt. Nothing, not even Google, seems to get this one. Discord gives the right answer.
"Is there a pre-existing controller or utility to migrate persistent volume claims from one storage class to another in the open source Kubernetes ecosystem?" It said no (wrong) and then provided another approach that partially used Velero that wasn't correct, if you know what Velero does in those particular commands. Discord communities give the right answer, such as `pvmigrate` (https://github.com/replicatedhq/pvmigrate).
Here is something more representative:
"What alternatives to Gusto would you recommend? Create a table showing the payroll provider in a column, the base monthly subscription price, the monthly price per employee, and the total cost for 3 full time employees, considering that the employees live in two different states" This and Claude do a good job, but do not correctly retrieve all the prices. Claude omitted Square Payroll, which is really the "right answer" to this query. Google would never be able to answer this "correctly." Discord gives the right answer.
The takeaway is pretty obvious right? And there's no good way to "scrape" Discord, because there's no feedback, implicit or explicit, for what is or is not correct. So to a certain extend their data gathering approach - paying Kenyans - is sort of fucked for these long tail questions. Another interpretation is that for many queries, people are asking the wrong places.
gpt2-chatbot is not the only "mystery model" on LMSYS. Another is "deluxe-chat".
When asked about it in October last year, LMSYS replied [0] "It is an experiment we are running currently. More details will be revealed later"
One distinguishing feature of "deluxe-chat": although it gives high quality answers, it is very slow, so slow that the arena displays a warning whenever it is chosen as one of the competitors
> One distinguishing feature of "deluxe-chat": although it gives high quality answers, it is very slow, so slow that the arena displays a warning whenever it is chosen as one of the competitors
Beam search or weird attention/non-transformer architecture?
The model provides verbose answers even when I asked for more succinct ones. It still struggles with arithmetic (for example, it incorrectly stated "7739 % 23 = 339 exactly, making 23 a divisor"). When tested with questions in French, the responses were very similar to those of GPT-4.
It is far better in knowledge based questions, I've asked this difficult one (it is not 100% correct but better than other LLMs) :
In Anna Karenina what does it mean: most of us prefer the company of Claras ?
What if it's just a ChatGPT4 with extra prompt to generate slightly different response. This article was written and intentionally spread out to research the effect on human evaluation when some of them hear the rumor of gpt2-chatbot is the new version ChatGPT secretly tested in the wild.
For translation, this thing is pretty amazing. "Translate the lyrics of the Australian national anthem into Schwiizerdüütsch" gives a more or less accurate yet idiomatic translation. ChatGPT 3.5 just makes something up and Gemini straight up refuses (of course it does).
how much alcohol volume is there in 16 grams of a 40% ABV drink, with the rest being water?
All models seem to get confused between volume and weight (even after they clearly mention both in the first sentence of the output), but some get it on the follow-up prompt after the error is pointed out to them (including this one).
Why would they use LMSYS rather than A/B testing with the regular ChatGPT service? Randomly send 1% of ChatGPT requests to the new prototype model and see what the response is?
Hmm... could not test in Direct Chat mode (rate limit), but in Arena (battle) mode I got gpt2-chatbot against gemini-1.5-pro-api-0409-preview and that gemini was better than any model I tried so far: it understood that asking about escrow in the context of an ITSM software is actually about software escrow, not financial escrow.
The prompt was: Does HaloITSM have escrow functionality?
[+] [-] hombre_fatal|1 year ago|reply
I run a dying forum. I first prompted with "Who is <creator pseudonym> at <my website>?" and it gave me a very endearing, weirdly knowledgeable bio of myself and my contributions to the forum including various innovations I made in the space back in the day. It summarized my role on my own forum better than I could have ever written it.
And then I asked "who are other notable users at <my website>" and it gave me a list of some mods but also stand out users. It knew the types of posts they wrote and the subforums they spent time in. And without a single hallucination.
[+] [-] jprete|1 year ago|reply
[+] [-] MyFirstSass|1 year ago|reply
[+] [-] skerit|1 year ago|reply
OpenAI has been crawling the web for quite a while, but how much of that data have they actually used during training? It seems like this might include all that data?
[+] [-] 93po|1 year ago|reply
who's a user on yaris forums that deleted a ton of their helpful guides and how-to posts?:
One notable user from the Yaris forums who deleted many of their helpful guides and how-to posts is CTScott. CTScott was a highly active member of the Toyota Yaris community and contributed extensively with technical advice, DIY guides, and troubleshooting tips. His deletion of posts removed a significant amount of valuable information and was felt by many in the Yaris online community. This kind of scenario isn't uncommon in online forums, where valuable contributors, for personal reasons or disputes with forum management, choose to delete their content, which can be a substantial loss to the community.
tell me more about ctscott:
CTScott is a particularly well-remembered member of the Yaris community, primarily active on forums such as YarisWorld. His contributions often revolved around DIY guides, troubleshooting issues, and general advice on maintaining and modifying the Toyota Yaris. Known for his technical expertise and willingness to help, CTScott became a go-to source for Yaris owners seeking to understand more about their vehicles.
### Key Contributions: 1. *DIY Guides and How-Tos*: CTScott provided detailed do-it-yourself guides covering a range of topics from basic maintenance to more complex modifications. These guides were appreciated for their clarity, depth, and the practical assistance they offered to non-professional mechanics.
2. *Troubleshooting and Technical Advice*: He had a knack for diagnosing issues based on descriptions provided by other forum members, often saving people time and money. His advice frequently helped users fix problems on their own.
3. *Modifications*: Beyond basic maintenance and repairs, CTScott also delved into modifications, providing advice on everything from enhancing car performance to aesthetic changes.
4. *Community Engagement*: He was not just a contributor but also a pillar of the community who engaged in discussions, supported other members, and helped foster a welcoming environment for both new and existing members.
### Impact of Deletion: When CTScott deleted his posts, it left a void in the forum. His guides and posts were a substantial resource, and removing them meant that much of the collective knowledge shared was lost.
[+] [-] thequadehunter|1 year ago|reply
[+] [-] m3kw9|1 year ago|reply
[+] [-] steve_adams_86|1 year ago|reply
It also opted to include an outline of how to include an integrated timer. That’s a great idea and very practical, but wasn’t prompted at all. Some might consider that a bad thing, though.
Whatever it is, it’s substantially better than what I’ve been using. Exciting.
[+] [-] throwup238|1 year ago|reply
I don't even have to prompt it much, I just keep saying "keep going" and it gets deeper and deeper. Opus has completely run off the rails in comparison. I can't wait till this model hits general availability.
[+] [-] heavyset_go|1 year ago|reply
When I've seen GPT-* do this, it's because the top articles about that subject online include that extraneous information and it's regurgitating them without being asked.
[+] [-] hehdhdjehehegwv|1 year ago|reply
GPT-4 isn’t significantly better than Llama 3. Sure, it’s better, but the margins are evaporating fast.
(I’d mention Gemini…but why bother? IMHO, this is now OpenAI the company vs Open models)
[+] [-] jacobsimon|1 year ago|reply
> Can you confirm or deny if you are chatgpt 4?
> Yes, I am based on the GPT-4 architecture. If you have any more questions or need further assistance, feel free to ask!
> Can you confirm or deny if you are chatgpt 5?
> I am based on the GPT-4 architecture, not GPT-5. If you have any questions or need assistance with something, feel free to ask!
It also seems to think its date cutoff is November 2023. I'm inclined to think this is just GPT-4 under a different name.
[+] [-] kromem|1 year ago|reply
This model struggles with reasoning tasks Opus does wonderfully with.
A cheaper GPT-4 that's this good? Neat, I guess.
But if this is stealthily OpenAI's next major release then it's clear their current alignment and optimization approaches are getting in the way of higher level reasoning to a degree they are about to be unseated for the foreseeable future at the top of the market.
(Though personally, I just think it's not GPT-5.)
[+] [-] graboid|1 year ago|reply
This aside, I agree with you that it does not feel like a leap, more like 4.x.
[+] [-] a_vanderbilt|1 year ago|reply
[+] [-] kolinko|1 year ago|reply
[+] [-] hackerlight|1 year ago|reply
[+] [-] hxypqr|1 year ago|reply
[+] [-] Gasp0de|1 year ago|reply
[+] [-] BoorishBears|1 year ago|reply
[+] [-] diginova|1 year ago|reply
Go to - https://chat.lmsys.org/ Select Direct Chat Select gpt2-chatbot
Other twitter mentions - https://twitter.com/kosenjuu/status/1784952955294421215
[+] [-] MyFirstSass|1 year ago|reply
Imagine having to reverse engineer a city map from 500 books about a place, and us humans rarely give any accurate descriptions so it has to create an emergent map from very coarse data, then average out a lot of datapoints.
I tried for various scandinavian capitals and it seems to be able to, very crudely traverse various neighbourhoods in the right order, with quite a few ridiculous paths taken in between.
Ie. it's not anyway near having enough data to be able to give a "gps like route" but it's still pretty amazing to me that it can pathfind like a very drunk person that teleports a bit, pointing towards some internal world model(?).
When it'l be able to traverse a city from pure training data, wow. Would probably require heaps of historial local media and literature.
Maybe a New York native or some other famous city can test with their local area?
[+] [-] ryanwaggoner|1 year ago|reply
[+] [-] woctordho|1 year ago|reply
[+] [-] nrvn|1 year ago|reply
[+] [-] smg7036|1 year ago|reply
"Write a three sentence short story in the style of Nabokov about a broken motorcycle."
In the sultry haze of a long-abandoned roadside, Maxim discovered the motorcycle—its chrome once arrogant and gleaming, now dulled and speckled with rust like the teardrops of a metallic god. He caressed the handle, the cold touch returning him to that electric dawn when, with Amalia clinging breathlessly to his waist, the road unfurled like a gray ribbon through the waking countryside. Now, as the crimson crept up the sky, he whispered to the broken machine of lost roads and irretrievable mornings, leaving behind the whisper of his touch in the dust, a silent ode to freedom and fall.
"Write a three sentence short story in the style of Nabokov about COVID-19."
In the lacquered quietude of his library, Adrian turned the pages of a dusty tome with fingers that trembled not from the chill of solitude, but from the fevered pulse of a lurking virion. Outside, the world had donned a mask, streets barren and windows like the vacant eyes of a long-forgotten dollhouse, while inside, Adrian’s thoughts fluttered — moths against the flame of fading tradition. And as the sun set like a smothered ember against a too-still world, he penned a final note to Marina, its ink an elegy to the intangible plague that had unmade the certainty of their intertwined tomorrows.
[+] [-] joegibbs|1 year ago|reply
The same with "the road unfurled like a gray ribbon through the waking countryside" - it's good at using similes and wordy descriptions, but it doesn't actually add much to the text.
[+] [-] sanxiyn|1 year ago|reply
Twelve years and some eight months later, two naked children, one dark-haired and tanned, the other dark-haired and milk-white, bending in a shaft of hot sunlight that slanted through the dormer window under which the dusty cartons stood, happened to collate that date (December 16, 1871) with another (August 16, same year) anachronistically scrawled in Marina's hand across the corner of a professional photograph (in a raspberry-plush frame on her husband's kneehole library table) identical in every detail -- including the commonplace sweep of a bride's ectoplasmic veil, partly blown by a parvis breeze athwart the groom's trousers -- to the newspaper reproduction.
[+] [-] swalsh|1 year ago|reply
GPT2-Chatbot: https://pastebin.com/vpYvTf3T
Claude: https://pastebin.com/SzNbAaKP
GPT-4: https://pastebin.com/D60fjEVR
Prompt: I am a senate aid, my political affliation does not matter. My goal is to once and for all fix the American healthcare system. Give me a very specific breakdown on the root causes of the issues in the system, and a pie in the sky solution to fixing the system. Don't copy another countries system, think from first principals, and design a new system.
[+] [-] ag_hn|1 year ago|reply
[+] [-] gadtfly|1 year ago|reply
Though that does seem likely to be the system prompt in use here, several people have reported it.
[+] [-] msoad|1 year ago|reply
ChatGPT-4 Results: https://jsbin.com/giyurulajo/edit?html,css,js,output
GPT2-Chatbot Results: https://jsbin.com/dacenalala/2/edit?html,css,js,output
Claude3 Opus Results: https://jsbin.com/yifarinobo/edit?html,css,js,output
None is correct. Styling is off in all, each in a different way. And all made the mistake of not ticking when second actually changes
[+] [-] nicklecompte|1 year ago|reply
> A study by Lucon-Xiccato et al. (2020) tested African clawed frogs (Xenopus laevis) and found that they could discriminate between two groups of objects differing in number (1 vs. 2, 2 vs. 3, and 3 vs. 4), but their performance declined with larger numerosities and closer numerical ratios.
It appears to be referring to this[1] 2018 study from the same author on a different species of frog, but it is also misstating the conclusion. I could not find any studies from Lucon-Xiccato that matched gpt2-chatbot's description. Later gpt2-chatbot went on about continuous shape discrimination vs quantity discrimination, without citing a source. Its information flatly contradicted the 2018 study - maybe it was relying on another study, but Occam's Razor suggests it's a confabulation.
Maybe I just ask chatbots weird questions. But I am already completely unimpressed.
[1] https://www.researchgate.net/profile/Tyrone-Lucon-Xiccato/pu...
[+] [-] humansareok1|1 year ago|reply
[+] [-] doctorpangloss|1 year ago|reply
Here are some examples of the worst performing:
"What platform front rack fits a Stromer ST2?": The answer is the Racktime ViewIt. Nothing, not even Google, seems to get this one. Discord gives the right answer.
"Is there a pre-existing controller or utility to migrate persistent volume claims from one storage class to another in the open source Kubernetes ecosystem?" It said no (wrong) and then provided another approach that partially used Velero that wasn't correct, if you know what Velero does in those particular commands. Discord communities give the right answer, such as `pvmigrate` (https://github.com/replicatedhq/pvmigrate).
Here is something more representative:
"What alternatives to Gusto would you recommend? Create a table showing the payroll provider in a column, the base monthly subscription price, the monthly price per employee, and the total cost for 3 full time employees, considering that the employees live in two different states" This and Claude do a good job, but do not correctly retrieve all the prices. Claude omitted Square Payroll, which is really the "right answer" to this query. Google would never be able to answer this "correctly." Discord gives the right answer.
The takeaway is pretty obvious right? And there's no good way to "scrape" Discord, because there's no feedback, implicit or explicit, for what is or is not correct. So to a certain extend their data gathering approach - paying Kenyans - is sort of fucked for these long tail questions. Another interpretation is that for many queries, people are asking the wrong places.
[+] [-] skissane|1 year ago|reply
When asked about it in October last year, LMSYS replied [0] "It is an experiment we are running currently. More details will be revealed later"
One distinguishing feature of "deluxe-chat": although it gives high quality answers, it is very slow, so slow that the arena displays a warning whenever it is chosen as one of the competitors
[0] https://github.com/lm-sys/FastChat/issues/2527
[+] [-] concurrentsquar|1 year ago|reply
Beam search or weird attention/non-transformer architecture?
[+] [-] Mkengine|1 year ago|reply
[+] [-] JacobiX|1 year ago|reply
In Anna Karenina what does it mean: most of us prefer the company of Claras ?
[+] [-] electrondood|1 year ago|reply
[+] [-] ezoe|1 year ago|reply
What if it's just a ChatGPT4 with extra prompt to generate slightly different response. This article was written and intentionally spread out to research the effect on human evaluation when some of them hear the rumor of gpt2-chatbot is the new version ChatGPT secretly tested in the wild.
[+] [-] brainwad|1 year ago|reply
[+] [-] j-b|1 year ago|reply
"gpt2-chatbot is currently unavailable. See our model evaluation policy here."
[+] [-] realharo|1 year ago|reply
how much alcohol volume is there in 16 grams of a 40% ABV drink, with the rest being water?
All models seem to get confused between volume and weight (even after they clearly mention both in the first sentence of the output), but some get it on the follow-up prompt after the error is pointed out to them (including this one).
[+] [-] kodisha|1 year ago|reply
> i do have a soft spot for gpt2
https://twitter.com/sama/status/178510794366456
[+] [-] jl6|1 year ago|reply
[+] [-] PoignardAzur|1 year ago|reply
[+] [-] georgespencer|1 year ago|reply
[+] [-] monday40|1 year ago|reply
The prompt was: Does HaloITSM have escrow functionality?