WingNews logo WingNews
top | item 38629124

(no title)

fotcorn | 2 years ago

I can only access https://makersuite.google.com/ when using a VPN to the US. Also, it spams popups that get blocked by Firefox.

Some basic prompts, which are answered correctly most of the time by ChatGPT4:

There are 31 books in my house. I read 2 books over the weekend. How many books are still in my house?

> 29 books

Julia has three brothers, each of them has two sisters. How many sisters does Julia have?

> Three

If you place an orange below a plate in the living room, and then move the plate to the kitchen, where is the orange now?

> Under the plate in the kitchen.

So, not great.

discuss

order

tr3ntg|2 years ago

Since the Maker Suite doesn't have the same prompting as something like Bard or ChatGPT, I gave Gemini Pro a bit more context:

> Consider the following user chat. Reason out loud as needed to provide a reasonable response. User message: "Julia has three brothers, each of them has two sisters. How many sisters does Julia have?"

Gemini Pro:

> Julia's brothers have nothing to do with how many sisters she has. Julia's sisters are also her brothers' sisters, so the number of sisters Julia has is the same as the number of sisters her brothers have. Julia's brothers have two sisters each, so if we add up the number of sisters for each brother, we get: 2 + 2 + 2 = 6. Therefore, Julia has 6 sisters. The answer is 6.

staticman2|2 years ago

Mixtral 8x7b only gets the sister question correct when I ask it to show it's work, but also shows that it's aware of the brain teaser:

https://poe.com/s/5gtLvuWT52o9LEBAJouo

GPT 4 gets it right.

GPT 3.5 gets it hilariously wrong if I change the question slightly:

User: Julia has 8 brothers, each of them has 5 sisters. How many sisters does Julia have?

Assistant:

Julia has 8 brothers, and each of them has 5 sisters. Since Julia is one of those sisters, she has 8 brothers + 1 (herself) = 9 sisters.

https://chat.openai.com/share/3a49800f-7747-44d9-90e7-c1e037...

Also on 3.5 I tried this:

User: Julia has 8 brothers, each of them has 5 sisters. How many sisters does Julia have? Show your work step by step.

Assistant: Julia has 8 brothers, and each of them has 5 sisters, including Julia herself. So, Julia is one of those 5 sisters for each of her 8 brothers.

To calculate how many sisters Julia has, you can simply multiply the number of brothers by the number of sisters each brother has:

8 brothers * 5 sisters per brother = 40 sisters

Therefore, Julia has 40 sisters in total.

angm128|2 years ago

Mistral 7B (mistral-7b-instruct-v0.2.Q6_K.gguf): Let's break down this problem step by step. First, we know that Julia has three brothers. Then it is stated that each of her brothers has two sisters. This means there are two sisters for each brother. Since Julia is one of the sisters for her brothers, she also counts as one of their sisters. However, when counting the number of sisters for herself, she should not be included in the count. Therefore, Julia has three brothers and two sisters (excluding herself), so she has a total of five sisters.

hatsunearu|2 years ago

Isn't 6 still incorrect? Or am I dumb?

ajhai|2 years ago

Gemini Pro compares to GPT 3.5.. their biggest model that competes with GPT-4 is called Gemini Ultra and they say that's coming early next year.

isoprophlex|2 years ago

Asking these to GPT3.5 has been an utterly frustrating experience, lol. I guess gemini is at this level of intelligence right now, not GPT4... rigged demos notwithstanding;)

FergusArgyll|2 years ago

I asked this to mini-orca 3b and here was it's brilliant answer.

> If you read 2 books over the weekend, then there are 31 books in your house. However, if you only read one book, then there would be only 25 books left in your house.

BlindEyeHalo|2 years ago

No joke, even if trying to correct GTP3.5 it still gives a nonsense answer:

> I apologize for any confusion caused by my previous response. Reading a book doesn't physically remove it from your house. The assumption I made was a misunderstanding. If you read a book, it is still in your house unless you lend it, give it away, or otherwise remove it.

> So, if you started with 31 books and read 2 over the weekend, you would have 31 - 2 = 29 books still in your house.

la64710|2 years ago

To the second question chatGPT answered : The orange is in the kitchen, as that is where the plate, along with the orange, was moved.

Now a human can reason because a human can visualize in their mind that a plate is being moved without moving the orange. How is that kind of reasoning possible in LLMs which in my very basic understanding is predicting the next token.

tagraves|2 years ago

Here's the response chatGPT 4 gave for me:

> The orange remains in the living room. Moving the plate to the kitchen does not affect the location of the orange, since it was placed below the plate but not attached to it. Therefore, the orange stays where it was originally placed, which is in the living room.

You don't need to visualize it in your mind to understand the relationship between being _below_ and being _moved with_. Keep in mind that many people cannot visualize anything in their mind!

luma|2 years ago

Make sure you're including the GPT version, there's a wide gulf between GPT4 and prior models. GPT-4 answers:

Q. If you place an orange below a plate in the living room, and then move the plate to the kitchen, where is the orange now?

A. The orange remains in the living room. Moving the plate to the kitchen does not affect the position of the orange.

----

Q. There are 31 books in my house. I read 2 books over the weekend. How many books are still in my house?

A. There are still 31 books in your house. Reading books does not change their quantity

----

Q. Julia has three brothers, each of them has two sisters. How many sisters does Julia have?

A. Julia has one sister. Each of her three brothers has two sisters, and Julia is one of them. The other one is her sister, making it a total of one sister for Julia.

lern_too_spel|2 years ago

In order to predict next tokens in text with greater and greater accuracy, some parameters of the language model must represent a model of the world.

amf12|2 years ago

FWIW, Gemini Pro is equivalent to GPT 3.5, so expected

EZ-E|2 years ago

GPT4 gets the second question wrong for me, but in a weird way.

> Julia has three brothers, each of them has two sisters. How many sisters does Julia have?

Julia has two sisters. Each brother having two sisters refers to the same two sisters, Julia and another sister.

Then if I ask to verify its answer, it realizes the issue.

> Apologies for any confusion. If Julia has three brothers and each of them has two sisters, then Julia is one of the sisters. The other sister is common among all three brothers. Therefore, Julia has only one sister.

ryoshu|2 years ago

Mistral 7b gets the first two wrong, but gets the third one right: "The orange is still in the living room. When you moved the plate, you didn't move the orange. It remains in its original location until you pick it up and move it to a different place"

strobe|2 years ago

mistralai/Mixtral-8x7B-Instruct-v0.1 responses - which seems much better:

>>There are 31 books in my house. I read 2 books over the weekend. How many books are still in my house?

You still have 31 books in your house. The number of books doesn't change because you finished reading two of them; they're just no longer unread.

>>Julia has three brothers, each of them has two sisters. How many sisters does Julia have?

Julia has one sister.

Here's the logic behind this:

1. We know that Julia has three brothers. 2. Then we are told that each of her brothers has two sisters. 3. Since all four children (Julia and her brothers) share the same parents, they also share the same siblings. 4. Therefore, if each of Julia's brothers has two sisters, then there must be only one other sister besides Julia herself.

>> If you place an orange below a plate in the living room, and then move the plate to the kitchen, where is the orange now?

The orange is still in the living room, as moving the plate does not affect the location of the orange.

kuratkull|2 years ago

My attempt had GPT4 fail on Julia, it was adamant that Julia has two sisters. It took couple of back and forths for it to understand.

bool3max|2 years ago

Why would you expect an LLM to answer thoe questions correctly?