[leogao]

There's been some work (e.g RASP - https://arxiv.org/abs/2106.06981) on taking logical computations and compiling them into transformer weights.

[astrange]

Sakana AI is also working on merging different transformer models together to combine skills.

https://sakana.ai/evolutionary-model-merge/