[xiomrze]

Honest question, how do you know if it's pulling from context vs from memory?

If I use Opus 4.6 with Extended Thinking (Web Search disabled, no books attached), it answers with 130 spells.

[ozim]

Exactly there was this study where they were trying to make LLM reproduce HP book word for word like giving first sentences and letting it cook.

Basically they managed with some tricks make 99% word for word - tricks were needed to bypass security measures that are there in place for exactly reason to stop people to retrieve training material.

[pron]

This reminds me of https://en.wikipedia.org/wiki/Pierre_Menard,_Author_of_the_Q... :

> Borges's "review" describes Menard's efforts to go beyond a mere "translation" of Don Quixote by immersing himself so thoroughly in the work as to be able to actually "re-create" it, line for line, in the original 17th-century Spanish. Thus, Pierre Menard is often used to raise questions and discussion about the nature of authorship, appropriation, and interpretation.

[ck_one]

Do you remember how to get around those tricks?

[petercooper]

One possible trick could be to search and replace them all with nonsense alternatives then see if it extracts those.

[andai]

That might actually boost performance since attention pays attention to stuff that stands out. If I make a typo, the models often hyperfixate on it.

[jazzyjackson]

A fine instruction following task but if harry potter is in the weights of the neural net, it's going to mix some of the real ones with the alternates.

[ck_one]

When I tried it without web search so only internal knowledge it missed ~15 spells.

[clanker_fluffer]

What was your prompt?