[primordialsoup]

Congrats! I went to your demo and asked for words that end in agi. This is what I got:

--

agi, agi, agi, agi, agi, agi, agi

These are some of the words that end in agi. You can also use the word agi in a sentence. For example, "I am going to the grocery store to get some agi."

These are some of words that end in agi.

These are some words that end in agi.

maximize, maximize, maximize, maximize, maximize, maximize, maximize, maximize

These are some words that ends in agi

--

So I think this needs more work to get to "as good as ChatGPT". But having said that, congrats on the landing

[brookst]

It's a fair criticism, and ChatGPT does better, but this isn't a great test of model quality. All LLMS that rely on tokenization struggle with being introspective on language. Try asking chatGPT to count how many e's are in a sentence, or to list all words that start with "to" and end wide "de".

I haven't heard anyone describe the phenomenon clearly, but I expect it is a challenge with reasoning over both intent of the prompt and specific token IDs.

[ksaj]

You can't ask ChatGPT to count something and expect that it can answer correctly, because it does not have counting logic. It is a language model, not a math model. People use this to "prove" hallucinations, but when you ask it something that is within it's programmed abilities, you get something at least close to what you want.

Having said that, here are the words ChatGPT gave me for the same prompt:

Magi Nagi Sagi Yagi Adagi Galagi Tegagi Sigikagi Tagi Wagagi

It missed Unagi, surprisingly. But it is still leagues ahead of the response primordialsoup got from Lamini.

[two_in_one]

well, here we go, ChatGPT in GPT-4 mode:

There are 7 instances of the letter 'e' in the sentence: "Try asking chatGPT to count how many e's are in a sentence."

another one:

The words with the letter 'e' from the sentence "Try asking chatGPT to count how many e's are in a sentence" are:

    asking
    sentence

and another, notice the last one:

Here are some English words containing three instances of the letter 'e':

    Nevertheless
    Extreme
    Relevance
    Precedence
    Residence
    Easement
    Demeanor

Please note that this is not an exhaustive list, but these examples should give you an idea of words with three 'e's in them.

What surprising here is that it's still capable of writing hundred lines of python code..

[chaxor]

It's not a character based model (likely - although it's closed source so anything is technically possible behind the scenes) so this makes some sense. The system can infer some relationships, which may be why 'agy' is conflated with 'agi' interestingly, but the tokenization process yields sequences of 'symbols' or indexes that are decided to English - so the system has a more difficult task when asked about 'e's (probably something like token 4893) and has to determine which tokens (e.g. [358,284840, 58292, 4830104, 57282, 4829193, 58282, 384, 24945] contain 'e's or token 4893). None of them do directly it seems - but 58292 may be 'ee' - so you would get this wrong as well.

[mckirk]

The problem is that these models do not have any working memory they could use to carry out such tasks, which are on a meta-level when seen from a language perspective. They can only go with their 'gut instinct' for selecting the next word, they can't 'consider and ponder the problem internally' first.

[kevviiinn]

GPT-4 did just fine when I asked it to name words that end in agi so I don't think your argument holds

[avereveard]

yeah as usual these model can barely sustain a conversation and fall apart the moment actual instructions are given. typical prompt they fail to udnerstand:

"what is pistacchio? explain the question, not the answer."

all these toy llm: "pistacchio is..."

gpt is the only one that consistently understand these instructions: "The question "what is pistachio?" is asking for an explanation or description of the food item..."

this makes these llm basically useless for obtaining anything but hallucinated data.

[vidarh]

It only makes them useless.of you insist on asking them in ways you already know will provide bad results instead of adapting your prompts.

This is a bit like complaining that your compiler refuses to produce the right outputs for code you've already determined is incorrect.

[TwoFactor]

Thats an interesting test. Here's what I got from ChatGPT:

---GPT-3.5---

Here are some words that end in "agi":

Strategy

Swarajya

Arthroplasty

Sialagogue

Podagric

Gynecology

Physiognomy

Ophthalmology

Esophagitis

Otalgia

--- GPT-4 ---

Here are some words that end in "agi":

Swaggy

Raggi

Magi

Gagi

Stagi

Please note that some of these words may not be commonly used or may be specific to certain dialects or regions.

[armchairhacker]

Stagi isn't a word (unless you count Lojban). Gagi isn't a word unless you could Filipino slang.