top | item 39673570

(no title)

colatkinson | 2 years ago

Adding on to the other comment, multiprocessing is also kinda broken on Linux/Mac.

1. Because global objects are refcounted, CoW effectively isn't a thing on Linux. They did add a way to avoid this [0], but you have to manually call it once your main imports are done.

2. On Mac, turns out a lot of the system libs aren't actually fork-safe [1]. Since these get imported inadvertently all the time, Python on Mac actually uses `spawn` [2] -- so it's roughly as slow as on Windows.

I haven't worked in Python in a couple years, but handling concurrency while supporting the major OSes was a goddamn mess and a half.

[0]: https://docs.python.org/3.12/library/gc.html#gc.freeze

[1]: https://bugs.python.org/issue33725

[2]: https://docs.python.org/3.12/library/multiprocessing.html#co...

discuss

order

fulafel|2 years ago

Re (1), are there publicly documented cases with numbers on observed slowdowns with it?

I see this mentioned from time to time, but intuitively you'd think this wouldn't pose a big slowdown since the system builtin objects would have been allocated at the same time (startup) and densely located on smaller nr of pages. I guess if you have a lot of global state in your app it could be more significant.

Would also be interesting to see a benchmark using hugepages, you'd think this could solve remaining perf problems if they were due to large number of independent CoW page faults.