[hmage]

What you're seeing is Kagi making the model dumber to make it respond quicker (hence the _Fast_ in the name).

I don't know how exactly models work, but they managed to make Claude instant dumber than it is by default (see below)

Here's your question asked verbatim to all the models I have access to:

Claude instant:

  Step by step working is as below:
  * 20 + 20 = 40
  * 40 + 20 = 60
  * 60 + 21 = 81

  Hence, 20 + 20 + 20 + 21 = 81

Claude v1:

  20 + 20 + 20 + 21 = 81

Claude v2:

  Okay, let's solve this step-by-step: 20 + 20 + 20 + 21 = 20 + 20 + 20 + 21 = 60 + 21= 81. Therefore, 20 + 20 + 20 + 21 = 81.

mpt-30b-chat:

  The sum of 20 + 20 + 20 + 21 is 81.

gpt-3.5-turbo:

  20 plus 20 plus 20 plus 21 equals 81.

gpt-4:

  81

[hmage]

I think I've figured out how to make claude instant dumber. I've modified your prompt so that it answers only with a number (and avoids doing the summation on writing).

I've lowered the temperature to lowest to avoid randomization of answers.

Prompt:

  How much is 20 plus 20 plus 20 plus 21? Answer only with a number.

Claude instant:

  102

Claude v1:

  81

Claude v2:

  81

mpt-30b-chat:

  61

gpt-3.5-turbo:

  81

gpt-4:

  81

Here we see that smaller models like mpt (30b) and claude instant (52b) can't do math "inside their head" and need an aid by doing the calculations step by step.

I guess that's why all the models default to doing step-by-step when they see a math problem.

[im3w1l]

So crazy thought I had.. As far as I understand these models can only do a fixed amount of work per token of output. So asking for it to show its work has two benefits, it lets it reference previous results it worked out, but it also plain gives it more computational resources.

So I'm curious what would happen if you prompted it to stall for time a bit with an answer like "hmm.... err... let's see.. what about 81?"