top | item 29720197

(no title)

glyph | 4 years ago

To the runtime-validation point; our team used attrs with runtime validation enforced everywhere (we even wrote our own wrapper to make it always use validation, with no boilerplate) and this ended up being a massive performance hit, to the point where it was showing up close to the top of most profile stats from our application. Ripping all that out made a significant improvement to interactive performance, with zero algorithmic improvements anywhere else. It really is very expensive to do this type of validation, and we weren't even doing "deep" validation (i.e. validating that `list[int]` really did include only `int` objects) which would have been even more expensive.

Python can be used quite successfully in high-performance environments if you are judicious about how you use it; set performance budgets, measure continuously, make sure to have vectorized interfaces, and have a tool on hand, like PyO3, Cython, or mypyc (you should probably NOT be using C these days, even if "rewrite in C" is the way this advice was phrased historically) ready to push very hot loops into something with higher performance when necessary. But if you redundantly validate everything's type on every invocation at runtime, it does eventually become untenable for anything but slow batch jobs if you have any significant volume of data.

discuss

order

No comments yet.