WingNews logo WingNews
top | item 45702373

(no title)

Locutus_ | 4 months ago

I've had exactly the same situation, ~2M MAU service with REDIS as the only persistence system, all data being JSON serialized Pydantic models. The storage overhead was just terrible and cost real money.

This would have been a super nice to have back then.

I wonder though how much sense it would make to get something like this mainlined into upstream Pydantic? as having this downstream would give many continuity and dependency lock concerns. And having it as part of the main library would significantly drive adoption rate.

discuss

order

sijokun|4 months ago

Thanks, your Redis story sounds exactly like my motivation. When the cache layer becomes the primary persistence and you’re pushing around millions of JSON-encoded Pydantic objects, the overhead hits both RAM and the bill very quickly. That pain is real.

Regarding getting this into Pydantic core: I doubt the maintainers would want to ship and maintain a binary serialization layer inside the main project. That said, it might still be worth opening an issue to hear their perspective - worst case, it becomes a documented "won’t include".

As for continuity and lock-in concerns - my primary production workload depends on this library, so I fully intend to keep it updated alongside new Pydantic and Python releases. I also intentionally designed it with minimal dependencies: core functionality uses only pydantic and typing-extensions. The only extra is leb128, and that’s just for UInt128, which almost nobody truly needs - I’m considering moving that to an optional extra to keep the base install even leaner.