[ogrisel]

You cannot share arbitrarily structured objects in the `ShareableList`, only atomic scalars and bytes / strings.

If you want to share structured Python objects between instances, you have to pay the cost of `pickle.dump/pickle.dump` (CPU overhead for interprocess communication) + the memory cost of replicated objects in the processes.

[notpushkin]

We need a dataclass-like interface on top of a ShareableList.

[notpushkin]

Actually, ShareableList feels like a tuple really (as it’s impossible to change its length). If we could mix ShareableList and collections.namedtuple together, it would get us 90% there (99.9% if we use typing.NamedTuple). Unfortunately, I can’t decipher either one [1, 2] from the first glance – maybe if I get some more sleep?

[1]: https://github.com/python/cpython/blob/3.13/Lib/collections/...

[2]: https://github.com/python/cpython/blob/3.13/Lib/typing.py#L2...

[tomrod]

I can fit a lot of json into bytes/strings though?

[frollogaston]

If all your state is already json-serializable, yeah. But that's just as expensive as copying if not more, hence what cjbgkagh said about flatbuffers.

[cjbgkagh]

Perhaps flatbuffers would be better?

[reliabilityguy]

What’s the point? The whole idea is to share an object, and not to serialize them whether it’s json, pickle, or whatever.

[vlovich123]

That’s even worse than pickle.

[sgarland]

So don’t do that? Send data to workers as primitives, and have a separate process that reads the results and serializes it into whatever form you want.