[GraemeMeyer]

Interesting that you see a slower response time with a large input - I don't see any speed degradation at all. Is that maybe just on the free tier of ChatGPT?

[gtirloni]

I'm on paid (rich, I know) and the performance is all over the place. Sometimes it'll spit out a whole paragraph almost instantly and other times it's like I'm back to my 2400bps modem.

I haven't noticed prompt size having an impact jut I'll test that.

[ravenstine]

This reflects my experience. Sometimes I'll provide a single sentence (to GPT-4 with the largest context window) and it will slowly type out 3 or so words every 5 seconds, and in other cases I'll give it a massive prompts and it returns data extremely fast. This is also true of smaller context window models. There seems to be no way to predict the performance.

[darkerside]

I'm guessing there are so many other impacts of own on the model that size of print probably gets lost. I can see a future where people are forecasting updates to ChatGPT like we do with the weather.

[CuriouslyC]

I typically notice the character by character issue with complex prompts centered around programming or logic. It feels kind of like the model is thinking, but my guess is that the prompt is being dispatched to an expert model that is larger and slower.

[willy_k]

If you mean the “analyzing” behavior, the indicator can be clicked on to show what it’s doing. It’s still going character-by-character, but writing code that it executes (or attempts to) to get the contents of a file, the solution for an equation, etc. Possibly an expert model but it seems like it’s just using an “expert prompt” or whatever you want to call it.

[0xFEE1DEAD]

Interesting, no I'm on the pro tier aswell. So you're telling me you never get the character-by-character experience?

Edit: What prompt sizes are we talking about?

Even with small prompts I occasionally get rather slow responses but it becomes unbearable at 2000-3000 characters (the upper limit of custom instructions), at least for me it does.