[rnosov]

A question to the author. Can you perform an ablation study with respect to the chunks? In other words, if you put in the context irrelevant/random chunks from the document would the quality of answers decrease/stay similar?

Potential issue might be that chunks just serve to activate massive knowledge of GPT4 and not actually used as a basis for an answer. For example, GPT4 has surely seen Dune in its training corpus and could be answering from memory.

[naveedjanmo]

This is an interesting idea. I'll have a think about a way to start measuring it. In Unriddle, any responses given that aren't drawn from the document are prefaced with a message to that effect. The bot usually says something like "I appreciate your curiosity about [query], but as an AI assistant, my primary focus is to provide advice on [document description]."