(no title)
spauka | 3 years ago
In fact I think that's a great example of exactly what is actually discussed, namely that the context that ChatGPT is able to hold is limited as it's context is held completely in its input. There is never any modification to it's internal state, we're just passing a longer input vector in to the start of the GPT-3 black box. For long inputs the embedding vector becomes more and more sparse and it needs to make more assumptions to fill in it's output.
ionwake|3 years ago