[chumpro]

I am also working on something similar. I have written a grammar system for controlling the llm that is not context free.

One of the main challenges is to have a database of what is reality, id -> object, and then have the llm return ids for objects referenced by the user. I am doing a system where something that has no id has not been created yet.

I use story templates with labeled segments to create characters, items, location as well as events. Example: "Once upon a time there was a <role>cowardly knight</role> named <name>Billy Bonkers</name>. The named qualities become the data structure associated with the id of the new character. It can be hilarious to prompt the llm to do nerdy space humor!

I seems to be very possible to create a "narrator driven" game happening in a "real" world.

I plan on using a multimodal llm and all the stuff that is created will have an image. That combined with id-less objects existing but not having a detailed description will make qualities and objects visible in the created images also part of the world.