It's an interesting article, as a tour of some Python internals.
But it comes off as a rant without a real suggestion.
Python 3 was an internal interpreter cleanup. That's actually part of its lack of widespread popularity. The core developers didn't add a ton of unique functionality (and most of the new stuff was backported to 2.7 anyway), but they fixed some annoying problems in the CPython codebase and broke some things in the name of cleanup/unification of concepts. They made Python easier to maintain and easier to build new features atop. They sharpened the axe.
Yes, they didn't clean up Armin's pet thing -- slots. And they created new problems for library developers in their bytes vs unicode changes. But they cleaned up a whole lot of other stuff.
I personally think with the maturity of pypy and the stability of both the 2.7 and 3.4 lines of development, the Python ecosystem has never been more exciting. The advances in pypy make Python attractive for CPU-bound work and the inclusion of asyncio in the stdlib will make it more and more attractive for I/O-bound work over time. Python has long been a winner for mixed workloads, and the ecosystem around Python, especially pydata utilities like numpy and pandas, keep getting better. Stop complaining -- let's just build an awesome future atop this marvelous language.
>But it comes off as a rant without a real suggestion.
Did you actually read TFA? It makes some very real suggestions towards the end, and pinpoints clear issues and how they could be changed all the way through.
>But they cleaned up a whole lot of other stuff.
Which is irrelevant to the current discussion. How is "they cleaned X" a response for "they should clean Y"?
> But it comes off as a rant without a real suggestion.
How exactly is that a rant? It's an exploration about a design decision / mistake that was carried through Python for 25 years and has left an impact. There is no complaint and there is no suggestion for Python.
As I said in the early paragraph it's something that's interesting for people that are interested in language design.
> Instead of having slots and dictionaries as a vtable thing, let's experiment with just dictionaries. Objective-C as a language is entirely based on messages and it has made huge advances in making their calls fast. Their calls are from what I can see much faster than Python's calls in the best case. Strings are interned anyways in Python, making comparisons very fast. I bet you it's not slower and even if it was a tiny bit slower, it's a much simpler system that would be easier to optimize.
Breaking the print statement is probably a large reason for lack of adoption. It used to be much more convenient. They've lost sight of 'practicality beats purity'. What used to be print a,b,c is now print(a + " " + b + " " + c), or using the string formatting method. What used to be
print linewithnonewline,
with the comma indicating the lack of newline, now has to be done through print(line, end=""). Which is more practical and pythonic?
I follow Ronacher's work, open source and posts and I agree with his arguments.
This post remember me of Spolsky's one (http://www.joelonsoftware.com/articles/LeakyAbstractions.htm...) though. When you get to a point where you get to know what's under the hood and why it's not working the way you are trying to use, however in a majority of cases you are just fine and don't need to know what's happening under the hood and maybe if this majority is big enough that's just fine.
In modern usage, this question also serves as a
metaphor for wasting time debating topics of no
practical value, or questions whose answers hold
no intellectual consequence.
Python will (or won't) be used for a particular application regardless of whether some contrived test takes 0.158 usec or takes 0.256 usec per iteration.
This sort of misunderstanding comes up so often that there are a plethora of cliches for it. Here's another one: "Missing the forest for the trees.".
For anyone else who found this as confusing as I did ("wtf, how can proxy actually be 42?"), what's going on here is that it's calling `proxy.__repr__()` when it attempts to display `proxy`, which in turn calls `42.__repr__()`. (Similarly, `proxy + 1` calls `42.__add__(1)`.)
I'm learning Python and I must say it is a wonderful language. Being able to concatenate strings by simply saying "a + b" is a great productivity boost (coming from C++). Python libraries are powerful. I can read an Excel spreadsheet with one line of code. I can create plots in PDF format with a half-dozen lines. Amazing!
However, I am disappointed with the difficulty of turning a program into a Windows EXE. I wrote a small program (couple of thousand lines), tried Py2exe which failed to handle the Numpy (or Matplotlib, I forget which) imbroglio.
PyInstaller works, except that the EXE is 85MB, and takes one minute to start up. Not practical for customer distribution. I can't expect my customers to install the Python runtime. In contrast, my 500KLOC C++ program, with all its third-party libraries, is 19MB. Yes, I know, Python needs everything including the kitchen sink. Still, 85MB is not practical.
> Being able to concatenate strings by simply saying "a + b" is a great productivity boost (coming from C++)
C++ has had this operator overloaded for strings for decades.
With regard to the size of compiled executables, I can't really say much except "that's not what it's made for". If you need to ship compiled executables to people, Python is an extraordinarily inappropriate choice; stick with C++.
you might like D -it's worth a look at the least. likewise go and nimrod, which also aim for better programmer productivity combined with compilation to efficient native code
I agree that OldStyleClasses might be simpler (while less featureful), but I think I'd care more for the footprint of instances, rather than class objects themselves
I switched to Python 3 this year and I haven't looked back -there are some niggles, but overall its a great improvement.
A great feature that's not really talked about is the __prepare__ function in metaclasses: you can supply a custom type that stores all class members. You could whip up multiple-dispatch using this (it lets you handle classes with duplicate property names) in conjunction with signature annotations, which I think is pretty neat.
This has been one of the reasons why I'm not currently doing anything for Google App Engine, they still lack Python 3 support. It feels awkward to code in Python 2 after Python 3.
This post is surprisingly confused, it is phrased as a complaint about the language, then immediately degrades into CPython implementation specifics that have little bearing on the usability of the language itself.
Ronacher should also know better than to post microbenchmarks like the one provided here, especially without corresponding (C) profiler output. At the C level, slots allow the implementation constant-time access to the most common code paths for an object, and especially when you have C code calling other C code via the type system (IMHO the primary use for Python, and still its strongest use case), "interpreter overhead" is reduced to a few extra memory indirection operations.
In the alternative world, sure, perhaps some microbenchmark may behave faster, but now systemically, and for e.g. "reduce(operator.add, range(1000))" requires more hash table lookups than I can count.
Python is all about providing a lightweight way to compose bits of fast code (the kernel, network stack, NumPy, MySQL, whatever). Unfortunately somewhere along the way solutions like Django got popular, which are almost the antithesis to this old viewpoint. Ronacher seems to be advocating that we should punish the CPython implementation's traditional strong areas in favour of.. actually, he didn't even describe any vision that was impacted by his complaints. He just seems to want the CPython implementation to suffer.
Perhaps his complaint about new-style __getattribute__ would be better suited as a bug report, it seems the only substantial observation made about the language itself in this post.
> This post is surprisingly confused, it is phrased as a complaint about the language, then immediately degrades into CPython implementation-specifics that have little bearing on the actual usability of the language itself.
You might think that, but you are very wrong and I should probably make that point in another blog post. These CPython specifics are enshrined in the language. While PyPy does not have the actual structs, it needs to implement the same user exposed API as it slots were used.
PyPy cannot just say "a + b" means "a.__add__(b)", it needs to implement the exact same dispatch logic that CPython has because people's code depends on it.
//EDIT:
> Ronacher seems to be advocating that we should punish the CPython implementation's traditional strong areas in favour of.. actually, he didn't even describe any vision that was impacted by his complaints.
Maybe I did not make my point very clear but the whole last paragraph advocates about trying a version of Python that abolishes the internal slot system.
>This post is surprisingly confused, it is phrased as a complaint about the language, then immediately degrades into CPython implementation-specifics that have little bearing on the actual usability of the language itself.
Ronacher says:
"Python is definitely a language that is not perfect. However I think what frustrates me about the language are largely problems that have to do with tiny details in the interpreter and less the language itself. These interpreter details however are becoming part of the language and this is why they are important."
He is right, Python has not real standard with clearly defined semantics. Actually, the standard is CPython, but CPython is full of crap that other implementations are then forced to implement thus making that crap part of the language. You should avoid Python at any cost, into the trash it goes.
His complaint is that implementation details of the interpreter have leaked into the specification of the language, such as it exists. The result is that alternate implementations have to mimic those quirks of the interpreter in order to achieve compatibility.
Having an abstract specification of the language, rather than "whatever CPython does", would give implementors more freedom, and thus allow the use of Python in more places and circumstances.
I think most of the things I'd want to see first would come in the standard library and in unicode handling.
I'd probably argue that most uses of metaclasses are a reason to step back and make the code simpler. There are a few cases where they come in very usefully, but at least in open source code, they create a barrier to understanding and increase complexity.
ython starts to get ugly when you start writing code that does "automagical things", and that's somewhat tolerated because it's not a language for building automagical things.
Or, to say it another way, if you have to think about how the interpreter works, your python has jumped off the idiomatic wagon a while ago.
That being said __slots__ as a performance enhancement is crazy useful - and I'd like to see more things in that area.
Though for most people, the thing that would lead to cleaner code would be a more powerful and elegant standard library.
> That being said __slots__ as a performance enhancement is crazy useful - and I'd like to see more things in that area. Though for most people, the thing that would lead to cleaner code would be a more powerful and elegant standard library.
__slots__ as PyPy shows is completely unnecessary.
>>"In recent years there is a clear trend of making Python more complex as a language. I would like to see the inverse of that trend. I would like to see an internal interpreter design could be based on interpreters that work independent of each other, with local base types and more, similar to how JavaScript works."
Sounds like OP should investigate Lua (and LuaJIT in particular).
He might like Lua more than Python.
(Given that OP created Flask, I'd love to see a Flask equivalent developed in Lua)
Good job not reading the article and skipping to the end, I guess?
>> This is in fact how many other dynamic languages work. For instance this is how lua implementations operate, how javascript engines work etc. The clear advantage is that you can have two interpreters. What a novel concept.
(Just a joke, people. Just a joke.) Anyway, I wonder why Armin's not more involved with core Python things, his perspective should be valuable. As someone doing some Python stuff on the side his articles are always worth reading carefully.
The Python community has always been the best part of the language, but the important subcommunities like NumPy, Twisted and PyPy has always seemed like they are a bit outside looking in. I don't know why. Perhaps the language would have evolved differently if these projects were a bit more involved in the actual language development.
Go and Javascript are "winning"? That's news to me. They might be buzzier right now, but their compiler/interpreters only appeared in 2009 and Python has been around since 1991. Is it really such a surprise that the hype period is over for Python as a language spec?
Now people are just getting a lot of stuff done with this productive language and nobody needs to be convinced any longer how good a language it is. Most people already know.
Now, we're onto the good work of building a huge ecosystem of open source modules atop it. Call me when Go and JavaScript have the equivalent of the PyData stack and rock-solid drivers for every database technology on the planet. Then maybe I'll look at them for anything other than niche use cases. (Obviously JavaScript is still the only game in town inside the browser and Go seems like a pretty interesting alternative to C for UNIX system utilities.)
> We have PyPy now; why not use the tremendous advances there in the next Python?
Because PyPy as cool as the project is, is terrible to develop on unless you are a PyPy person. I would consider myself a reasonable programmer but PyPy makes me go crazy. Slow iteration times, super complicated code.
I really would like to see, when the suggested, simpler system would be tried. It is sometimes amazing, how by simplification gains could be reached against common wisdom. And even, when no gains or little losses would be there with the new, simpler system, it could be worthwhile to change, since the current system has his huddles, that I also already stumbled about, and which are not clearly communicated, since it is such a deep implementation detail of CPython. Also it was mentioned, that a simpler system could be even more powerful.
Thanks for the very informative post. I've been a full-time Python developer for only the past 7 or 8 months now. I had not in that time learned about the slot system or the CPython implementation.
I moved from Javascript to Python 2.7 and haven't looked back since. I now strictly use Python in all of my projects, Javascript only because browser doesn't support it.
I wish that Python would introduce a few things from the Javascript world. Easy async, and a real-time Meteor like framework.
[+] [-] pixelmonkey|11 years ago|reply
But it comes off as a rant without a real suggestion.
Python 3 was an internal interpreter cleanup. That's actually part of its lack of widespread popularity. The core developers didn't add a ton of unique functionality (and most of the new stuff was backported to 2.7 anyway), but they fixed some annoying problems in the CPython codebase and broke some things in the name of cleanup/unification of concepts. They made Python easier to maintain and easier to build new features atop. They sharpened the axe.
Yes, they didn't clean up Armin's pet thing -- slots. And they created new problems for library developers in their bytes vs unicode changes. But they cleaned up a whole lot of other stuff.
I personally think with the maturity of pypy and the stability of both the 2.7 and 3.4 lines of development, the Python ecosystem has never been more exciting. The advances in pypy make Python attractive for CPU-bound work and the inclusion of asyncio in the stdlib will make it more and more attractive for I/O-bound work over time. Python has long been a winner for mixed workloads, and the ecosystem around Python, especially pydata utilities like numpy and pandas, keep getting better. Stop complaining -- let's just build an awesome future atop this marvelous language.
[+] [-] coldtea|11 years ago|reply
Did you actually read TFA? It makes some very real suggestions towards the end, and pinpoints clear issues and how they could be changed all the way through.
>But they cleaned up a whole lot of other stuff.
Which is irrelevant to the current discussion. How is "they cleaned X" a response for "they should clean Y"?
[+] [-] the_mitsuhiko|11 years ago|reply
How exactly is that a rant? It's an exploration about a design decision / mistake that was carried through Python for 25 years and has left an impact. There is no complaint and there is no suggestion for Python.
As I said in the early paragraph it's something that's interesting for people that are interested in language design.
[+] [-] thomasahle|11 years ago|reply
[+] [-] jimmaswell|11 years ago|reply
[+] [-] nakovet|11 years ago|reply
This post remember me of Spolsky's one (http://www.joelonsoftware.com/articles/LeakyAbstractions.htm...) though. When you get to a point where you get to know what's under the hood and why it's not working the way you are trying to use, however in a majority of cases you are just fine and don't need to know what's happening under the hood and maybe if this majority is big enough that's just fine.
[+] [-] PhantomGremlin|11 years ago|reply
In particular:
Python will (or won't) be used for a particular application regardless of whether some contrived test takes 0.158 usec or takes 0.256 usec per iteration.This sort of misunderstanding comes up so often that there are a plethora of cliches for it. Here's another one: "Missing the forest for the trees.".
[+] [-] philh|11 years ago|reply
[+] [-] GnarfGnarf|11 years ago|reply
However, I am disappointed with the difficulty of turning a program into a Windows EXE. I wrote a small program (couple of thousand lines), tried Py2exe which failed to handle the Numpy (or Matplotlib, I forget which) imbroglio.
PyInstaller works, except that the EXE is 85MB, and takes one minute to start up. Not practical for customer distribution. I can't expect my customers to install the Python runtime. In contrast, my 500KLOC C++ program, with all its third-party libraries, is 19MB. Yes, I know, Python needs everything including the kitchen sink. Still, 85MB is not practical.
Too bad. Not all programs run on a server.
[+] [-] theseoafs|11 years ago|reply
C++ has had this operator overloaded for strings for decades.
With regard to the size of compiled executables, I can't really say much except "that's not what it's made for". If you need to ship compiled executables to people, Python is an extraordinarily inappropriate choice; stick with C++.
[+] [-] alexchamberlain|11 years ago|reply
[+] [-] Shorel|11 years ago|reply
The template syntax is a great productivity boost compared to C++.
[+] [-] zem|11 years ago|reply
[+] [-] berdario|11 years ago|reply
[+] [-] orf|11 years ago|reply
A great feature that's not really talked about is the __prepare__ function in metaclasses: you can supply a custom type that stores all class members. You could whip up multiple-dispatch using this (it lets you handle classes with duplicate property names) in conjunction with signature annotations, which I think is pretty neat.
[+] [-] Sami_Lehtinen|11 years ago|reply
[+] [-] hosay123|11 years ago|reply
Ronacher should also know better than to post microbenchmarks like the one provided here, especially without corresponding (C) profiler output. At the C level, slots allow the implementation constant-time access to the most common code paths for an object, and especially when you have C code calling other C code via the type system (IMHO the primary use for Python, and still its strongest use case), "interpreter overhead" is reduced to a few extra memory indirection operations.
In the alternative world, sure, perhaps some microbenchmark may behave faster, but now systemically, and for e.g. "reduce(operator.add, range(1000))" requires more hash table lookups than I can count.
Python is all about providing a lightweight way to compose bits of fast code (the kernel, network stack, NumPy, MySQL, whatever). Unfortunately somewhere along the way solutions like Django got popular, which are almost the antithesis to this old viewpoint. Ronacher seems to be advocating that we should punish the CPython implementation's traditional strong areas in favour of.. actually, he didn't even describe any vision that was impacted by his complaints. He just seems to want the CPython implementation to suffer.
Perhaps his complaint about new-style __getattribute__ would be better suited as a bug report, it seems the only substantial observation made about the language itself in this post.
[+] [-] the_mitsuhiko|11 years ago|reply
You might think that, but you are very wrong and I should probably make that point in another blog post. These CPython specifics are enshrined in the language. While PyPy does not have the actual structs, it needs to implement the same user exposed API as it slots were used.
PyPy cannot just say "a + b" means "a.__add__(b)", it needs to implement the exact same dispatch logic that CPython has because people's code depends on it.
//EDIT:
> Ronacher seems to be advocating that we should punish the CPython implementation's traditional strong areas in favour of.. actually, he didn't even describe any vision that was impacted by his complaints.
Maybe I did not make my point very clear but the whole last paragraph advocates about trying a version of Python that abolishes the internal slot system.
[+] [-] topkekz|11 years ago|reply
Ronacher says: "Python is definitely a language that is not perfect. However I think what frustrates me about the language are largely problems that have to do with tiny details in the interpreter and less the language itself. These interpreter details however are becoming part of the language and this is why they are important."
He is right, Python has not real standard with clearly defined semantics. Actually, the standard is CPython, but CPython is full of crap that other implementations are then forced to implement thus making that crap part of the language. You should avoid Python at any cost, into the trash it goes.
[+] [-] cwp|11 years ago|reply
Having an abstract specification of the language, rather than "whatever CPython does", would give implementors more freedom, and thus allow the use of Python in more places and circumstances.
[+] [-] mpdehaan2|11 years ago|reply
I'd probably argue that most uses of metaclasses are a reason to step back and make the code simpler. There are a few cases where they come in very usefully, but at least in open source code, they create a barrier to understanding and increase complexity.
ython starts to get ugly when you start writing code that does "automagical things", and that's somewhat tolerated because it's not a language for building automagical things.
Or, to say it another way, if you have to think about how the interpreter works, your python has jumped off the idiomatic wagon a while ago.
That being said __slots__ as a performance enhancement is crazy useful - and I'd like to see more things in that area. Though for most people, the thing that would lead to cleaner code would be a more powerful and elegant standard library.
[+] [-] masklinn|11 years ago|reply
2. __slots__ are not actually useful, Pypy does what they do by default and deoptimizes as needed, CPython probably could as well
3. the only thing __slots__ do is lower memory usage (by not allocating a dict per instance) and PEP 412 has already improved that
__slots__ is not "crazy useful" by any definition of the expression, it is somewhat useful when you have a huge living number of simple objects.
[+] [-] DasIch|11 years ago|reply
__slots__ as PyPy shows is completely unnecessary.
[+] [-] Dylan16807|11 years ago|reply
Can you be more specific? The article claims that it's not a performance boost at all. Where is it useful?
[+] [-] alberth|11 years ago|reply
>>"In recent years there is a clear trend of making Python more complex as a language. I would like to see the inverse of that trend. I would like to see an internal interpreter design could be based on interpreters that work independent of each other, with local base types and more, similar to how JavaScript works."
Sounds like OP should investigate Lua (and LuaJIT in particular).
He might like Lua more than Python.
(Given that OP created Flask, I'd love to see a Flask equivalent developed in Lua)
Edit: typo
[+] [-] scrollaway|11 years ago|reply
>> This is in fact how many other dynamic languages work. For instance this is how lua implementations operate, how javascript engines work etc. The clear advantage is that you can have two interpreters. What a novel concept.
[+] [-] futuredale|11 years ago|reply
[+] [-] xorcist|11 years ago|reply
(Just a joke, people. Just a joke.) Anyway, I wonder why Armin's not more involved with core Python things, his perspective should be valuable. As someone doing some Python stuff on the side his articles are always worth reading carefully.
The Python community has always been the best part of the language, but the important subcommunities like NumPy, Twisted and PyPy has always seemed like they are a bit outside looking in. I don't know why. Perhaps the language would have evolved differently if these projects were a bit more involved in the actual language development.
[+] [-] rattray|11 years ago|reply
So I think it's about time that leaders of the community like Ronacher, Rietz, Gaynor, etc come together to talk about the next generator of Python.
We have PyPy now; why not use the tremendous advances there in the next Python?
We have Go and Javascript to compete with and we understand a lot of why they're winning; it's time Pythonistas fought back instead of just leaving.
Python 3 was supposed to be "the next thing" five years ago; it fizzled. Let's make the real "next thing" for 2016. Python 4.
[+] [-] pixelmonkey|11 years ago|reply
Now people are just getting a lot of stuff done with this productive language and nobody needs to be convinced any longer how good a language it is. Most people already know.
Now, we're onto the good work of building a huge ecosystem of open source modules atop it. Call me when Go and JavaScript have the equivalent of the PyData stack and rock-solid drivers for every database technology on the planet. Then maybe I'll look at them for anything other than niche use cases. (Obviously JavaScript is still the only game in town inside the browser and Go seems like a pretty interesting alternative to C for UNIX system utilities.)
[+] [-] the_mitsuhiko|11 years ago|reply
Because PyPy as cool as the project is, is terrible to develop on unless you are a PyPy person. I would consider myself a reasonable programmer but PyPy makes me go crazy. Slow iteration times, super complicated code.
[+] [-] maxerickson|11 years ago|reply
There's lots of British comedy troupes that came after Monty Python, and lots of snakes if you want to go in that direction.
[+] [-] PythonicAlpha|11 years ago|reply
[+] [-] fndrplayer13|11 years ago|reply
Also, <3 Flask. So good.
[+] [-] scardine|11 years ago|reply
[+] [-] notastartup|11 years ago|reply
I wish that Python would introduce a few things from the Javascript world. Easy async, and a real-time Meteor like framework.
[+] [-] lmm|11 years ago|reply