[Arimbr]

If all the pipeline and the vector index is keep in memory... does Pathway still persist state somewhere?

[dxtrous]

(Adrian from the Pathway team here.) Indeed, everything is RAM-based, and persistence/cache relies on file backends. The precise backend to use is a code configuration parameter. S3 or local filesystem are the currently supported options. For documentation, see the user guide under Deployment -> Persistence.

[Arimbr]

Nice, thanks! I was reading https://pathway.com/developers/user-guide/deployment/persist.... If I understand correctly you persist both source data and internal state, including the intermediary state of the computational graph. And you only rely on the backend to recover from failures and upgrades. So if I want to clone a Pathway instance, I don't need to reprocess all source data, I can recover the intermediary state from the snapshot.

Is it the same logic for the VectorStoreServer? https://pathway.com/developers/user-guide/llm-xpack/vectorst...