Unladen Swallow 2009Q3 Released with LLVM optimized Python

[cool-RR]

Firstly let me say I think this is a wonderful project and one of the most important Python projects I know.

Something I noticed though, in the Project Plan page:

http://code.google.com/p/unladen-swallow/wiki/ProjectPlan#20...

They state that one of the goals for "2009Q3 and beyond" is "to remove the GIL and fix the state of multithreading in Python." Which I think is an awesome and ambitious goal that I would find very useful as will many others.

So I hoped the GIL thing would fall into the "2009Q3", but apparently it's in the "beyond" category. Anyway, they need to update that project plan.

[tlb]

My big python app (the robot end of a telepresence connection) suffers from GIL problems. Mostly, threads are used for C-only stuff like video compression so they can easily release the GIL while running. But I gotta say, the complexities of GIL correctness, thread performance, gtk/gobject, callbacks, boost.python, GC-correctness of both Python and C++ objects, and polling make my head hurt. Simplicity would be a find goal for the new system.

[kingkilr]

If you head over to python-ideas there's actually some really good discussion about removing the GIL right now. Someone has proposed using CAS operations (which I know nothing about) to remove the fine grained locking that is needed under the refcounting GC without the GIL. In addition Antoine Pitrou has been working on a patch to make the GIL more performant on POSIX systems.

[bdr]

They mentioned this at the recent SF Python Meetup. They had found an academic paper that they thought they could apply to Python to get rid of the GIL, but it turned out someone had already tried it unsuccessfully. I think that goal of unladen-swallow is definitely "beyond" for now.

[mbrubeck]

"2009Q3 uses up to 930% less memory than the 2009Q2 release."

Does that math make sense in some way I'm not seeing?

[wisesage5001]

From their benchmarks rietveld: $ ./perf.py -r -b rietveld --args "-j always," --track_memory ../q2/python ../q3/python Mem max: 575116.000 -> 55952.000: 927.87% smaller

This 930% basically means that 2009Q3 can use as little as ~1/10 of the memory of 2009Q2.

[sparky]

It's unconventional, but 9.3x less memory can be seen in some sense as 1/9.3=10.7 percent as much memory. Not defending it, just trying to interpret :P

[ErrantX]

I never checked but I assumed it meant that the memory savings of Q3 compared to the savings of Q2 increased 930%

[simonw]

Is it noticably faster than regular Python yet?

[sparky]

Their first release (2009Q1) consisted mostly of patches to mainlines CPython, which improved performance for everybody by roughly 10%. They've added about 10-15% performance on each of the last 2 releases in Unladen Swallow-specific code, but it's unclear (to me) how CPython performance has evolved over the last 6 months. They haven't compared explicitly since 2009Q1, which may or may not be revealing.

From the release notes:

Lowlights:

    * LLVM's JIT and other infrastructure needed more work than was expected. As a result, we did not have time to improve performance as much as we would have liked.
    * Memory usage is still 2-3x that of Python 2.6.1. However, there is more overhead that can be eliminated for the 2009Q4 release. 

The last 2 releases have been mostly building up infrastructure; hopefully most of that is behind them and they can work on getting the 5x performance they're after.

[clopey]

Has anyone got this working on Windows?

[rbanffy]

I am considering giving it a try under Cygwin. If you get there first, please, tell us.

[maksimka]

What's the plan for Unladen Swallow to support py3k?