[mamp]

Unfortunately, I think the context rot paper [1] found that the performance degradation when context increased still occurred in models using attention sinks.

1. https://research.trychroma.com/context-rot

[giancarlostoro]

Saw that paper have not had a chance to read it yet, are there other techniques that help then? I assume theres a few different ones used.