This is a fairly weak article, full of deflections around the real weaknesses of python. I'm surprised it is on the Paypal Engineering blog.
Like all languages, Python has strengths and weaknesses, and there is no shame in that. An honest article would address the negatives head on, acknowledge them as potential negatives,(not skip around them) and provide alternatives, .
The strawman "Python has a weak type system" is a good example of such deflection.
No one (who understands type systems) would complain that "Python has a weak type system"
A more common "criticism" would be "Python does not have a static type system, which is handy for large codebases worked on by large teams".
Address that upfront, and you have a decent article. This is just a fanboy (who just happens to be at Paypal) praising his favorite language and ignoring its weaknesses.
The keyword here is "Enterprise" Python. The Enterprise is a place where you have "decision makers" which are often people who don't understand type systems, and have absolutely no incentive to do so.
Paypal is a large organization which needs to hire Python programmers. Everyone reading this now has a reminder that Python is popular at Paypal. We also know that endless PLT arguments are probably not common in the workplace.
Some people - who do understand type systems - only think of static type systems as type systems. And in that regard they might look at "dynamically typed langauges" as having "weak type systems", since they are often just unityped (only have one type).
If you want me to trust what you say about a language — or any technology, actually — be forthright about its deficiencies.
Because I don't believe you really, truly understand a language until you can tell me what sucks about it. It takes significant time (in a reasonably decent language) to discover the corner cases, performance bottlenecks, quirks, big-deficiencies-hidden-in-plain-sight and outright bugs in a language like, say, python.
Something as "rah-rah" this, which goes so far as to basically call the GIL a source of unicorns and rainbows, is convincing almost in inverse proportion to its stridency. Suddenly I'm wondering if all these "myths" about python might be smoke pointing toward a fire. That's probably a bit unfair, but it's hard to know what to take seriously when you're listening to a voice that's less than credible.
I've had a chance to use Python several times in various projects.
The language really excelled at small projects, where the expressiveness of the language and the excellent libraries available really let us get a lot done with little code and time. Avoiding the recompile/redeploy steps also helped.
But using the language for larger projects was quite a different thing. Suddenly you could find yourself in code several layers down, being passed an object of God knows what type from unfamiliar code, and having to find out what you were receiving essentially by trial and error. And once the execution times started to climb, it became increasingly frustrating to see trivial problems that would have been caught during compilation in a statically typed language appear in Python only after 30-60 minutes of execution.
There are surely ways to alleviate these specific problems -- I can think of some things to try myself -- but my experience suggests that the sweet spot for dynamic languages like Python is in projects that fit between one pair of ears, and stricter statically typed languages become increasingly useful as things scale up.
>Something as "rah-rah" this, which goes so far as to basically call the GIL a source of unicorns and rainbows
The GIL is definitely no source of unicorns and rainbows, but I think the case against it is usually overstated. There are numerous ways of sidestepping it (multiprocessing, pypy, C extensions, etc.), and it does serve a useful purpose.
This is one of my favorite questions when interviewing software developers: "What is your favorite language and why?" Followed by: "Tell me what you would change about it if you could."
Meh.... I think he's right that there's no good reason _not_ to use Python.
But I think he overstates the case.
Calling Python "compiled" doesn't seem right -- he's more right that this doesn't matter, there's no reason not to use something because it's "not compiled".
Calling Python's typing "strong typing", eh, really? He provide a link to a wikipedia article on typing in general, how about a link to anyone else making the case that Python should be considered 'strongly typed', for a particular definition of 'strongly typed'? I'm dubious that it would be any definition of 'strongly typed' that those who think they want it... want. Again, I think a better argument might be: So what, you don't need strong typing for success, because of a b and c.
And the concurrency discussions is just silly. Sure, you can do _all sorts_ of things without multi-core concurrency, you may not need it or even benefit from it, and scaling to multi-process can sometimes work just fine too. But there are also _all sorts_ of cases where you do really want multi-core parallelism, and Python really doesn't have it -- and trying to justify this as "makes it much easier to use OS threads" is just silly. I guess it makes it 'easier' in that it protects you from some but not all race conditions in fairly unpredictable ways -- but only by eliminating half the conceptual use cases for threads in the first place (still good for IO-bound work, no longer good for CPU-bound work).
I think there's a reasonable argument to be made that Python will work just fine for 'enterprise' work. There are also, surely, like for all languages/platforms, cases where it would be unsuitable. By overstating the case with some half-truths and confusions, he just sounds like a fanboy and does not help increase understanding or accurate decision-making -- or confidence in the Python community's understanding!
I can think of three ways of thinking about the term "strongly typed" once you realize it doesn't mean "statically typed".
1) It's meaningless--it's almost impossible to produce a real ranking of languages on the strength of their types.
2) These dynamic languages are all unityped because they're not statically typed. The fact that runtime tags won't allow certain operations to succeed has nothing to do with types.
3) Strongly typed is a rough and ready way of saying you don't do many implicit conversions.
I sympathize with (1). (2) is accurate, but really a terminological point. There's clearly a phenomenon to talk about, even if "typed" is the wrong word. I also sympathize with (3).
Python does far fewer implicit conversions than PHP, JavaScript or Perl. It even doesn't do at least one that Java does. So people often say Python is strongly, but dynamically typed. It's a real aspect of the language that you can view as a plus or minus.
There are plenty of options for concurrent and parallel operations in Python[0]. It is heavily used in data analytics and the scientific programming communities. Though it does often need a little help from C to take advantage of SMP hardware.
Python is compiled in the sense that it produces bytecode, but in being so it lacks the benefits of other compiled languages such as thorough compile time error checking. I've found that this is sometimes why people don't even notice the compiler.
As a pythonista, the only issue I had is with types. A lot of the time, I don't know what to expect from a function, it could be None or a int (legacy code). In a typed language like go, you would not have the ability to do this. So, although powerful, it opens the door for potentially bad decisions.
Python's proposal for type hinting will dampen the effect of type inconsistency I feel.
Go is not a good example of a type system - its types are limited and restrictive and really will stop you writing some good code that you wanted to write in Python. If you want a Pythonic language with static types, maybe take a look at OCaml (or my personal favourite, Scala - though that language suffers from a bunch of warts for the sake of JVM compatibility). http://roscidus.com/blog/blog/2014/06/06/python-to-ocaml-ret... is one example of someone making the move.
Agreed on the "lack of declared types" thing. While I'm exceptionally fast writing new Python code, reading other people's Python code is a different story - particularly, if they are new to Python.
While there are a lot of concurrency options to not get you in trouble with the GIL, it is annoying that there is no canonical way or pythonic way of doing it. I saw people using things like monkey patching, which made my jaw drop. I had to implement Websockets recently and was totally confused as to the right way of doing it. I ended up evaluating NodeJS and Go for this function, and went with Go.
Overall ... I love Python. It is perfect for small projects. For larger codebases (3KLOC+), I am slightly skeptical. It can be done but developers need to have quality code and strong test coverage.
Indeed, this is one of the biggest problems of Python for large-scale projects (saying this as an avid Pythonista myself). This is due to dynamic typing, though, not weak typing (Python's typing is actually strong).
Weak typing has its merits but sometimes static typing is very useful for reading and understanding code.
I know we can get picky about the exact characteristics of a languages type-system, but for me they fall into two camps: "Typeful" languages and "(effectively) Typeless" languages.
The former have a "good" typesystem, and expose it as a tool the programmer can use to help themselves solve problems. In the latter case, there may or may not be a type system lurking in the language somewhere, but the language either doesn't expose that system as a tool for the programmer to use, or it does so inconsistently.
Personally I think python is in the later camp, it has types but it doesn't really give the programmer much help when solving problems.
Clojure is another example. The Java type-system is down there somewhere in the murk, but idiomatic Clojure code doesn't use it, and it's not really a tool for the programmer to use, more an accident of the platform.
I can relate, on some level. Sometimes I feel like writing test cases that increase coverage is not enough. There are days where I think I need a sat solver in order to get confidence that I'll never see incorrect behavior from my code.
> type hinting will dampen the effect of type inconsistency I feel
I'm not sure that I see type hinting being that advantageous. I kinda hope that it could allow PyPy and other VMs to make assumptions that would otherwise require heuristics.
Perhaps type hinting could make static checkers/linters for Python more effective, though.
Yes, in theory this is a problem in lots of languages which are weakly typed. There are lots of ways of handling this, some generic and some language specific.
In python I would suggest wrapping your call in a try / expect block or using "isinstance()".
If you are using a publicly available library or popular piece of code that can return None or an Integer, I would argue that piece of code was written incorrectly. Newbie python users might do this, but I think experienced python devs would see the problem.
Finally, some documentation or justification for the reasoning behind this decision at the top of the function or on a web page somewhere would help as well.
As with most powerful , full featured languages it is pretty easy to shoot yourself in the foot if you are not careful, I don't think this is a python specific problem...
Python is great for relatively small projects written by a single person or for glue code, but large Python projects are scary. Diving in and reading someone else's code can be quite difficult (what are the types of these parameters? they're being used like dicts, but what if they're really a completely different class with different behavior? how can I trace this down when it goes through multiple IPC indirections since they chose to use multiple processes to get around the GIL? oh i'm missing packages and they don't seem to be in pip...). The bigger the project, the more places there are for runtime errors that a statically compiled language would have caught or at least has better tooling to find (syntax errors in an error path block that rarely gets hit, type errors, uncaught exceptions, missing methods, accidentally fat-fingering a variable name and instantiating a new variable). You end up leaning really heavily on tests that wouldn't have to be written in other languages and documentation that would otherwise be statically enforced in many other languages. Even when using Python for glue code, be wary that it will need to be in-charge (i.e. embedding a Python interpreter in your code versus embedding your code in Python especially with CPython).
Not saying it's not possible and doesn't work well for a lot of companies, but I do think you take on technical debt to get up and running fast when you choose Python. It's a choice I would personally think twice about.
I've found over the years that there's a fairly widespread prejudice among many enterprise developers in general and .NET developers in particular towards Python. Mention to a few random .NET developers that you're using it on a side project and perhaps about half of them will give you a look of puzzlement at best.
The main problem isn't anything to do with technical failings of the language, but a misconception that it's an obscure niche language that nobody uses outside of academia, with an ecosystem bordering on the nonexistent. People seem to think that if you start using Python you're going to end up with an incomprehensible codebase that's impossible to maintain because you can't hire developers who know it. They're generally quite surprised when I tell them how widely used it is and what all it gets used for.
A .NET developer recently was telling me the .NET community tends to be very insular, a lot of interesting ideas are ignored unless Microsoft makes them first-class citizens of the ecosystem. That's unfortunate IMO.
Relative to the .NET stack, Python's toolchain has always been no-cost for anyone to use, and the language is so fun that there are many people who enjoy just coding in it for free.
I can almost see these guys' angle, of why they might see this as a possible threat that could undermine their career. .NET coding pays well, and is usually semi-exclusive to those who previously had an employer pay for them to use the tools, even though that's not theoretically necessary.
Considering it's replacing C++ as the starter language at many universities this may not be true for too much longer as younger developers get more exposure to Python.
I have a fairly large code base (80K LOC). When the size grows, lack of typing can become a problem. 95% percent of the time, the code is explicit enough. However, when you end up with meta code, then it can become really difficult to track the types down in the n-th level of recursion...
Python2 to 3 migration is not easy. There are tools but the problem is that they don't see everything and therefore, you end up with 95% of your code converted. Then it's up to you to figure out the last 5%, which is an order of magnitude harder that the first 95%... So I ended up having a fairly long transition of migrating Python2 code to Python2+3 code.
For bothe these issues, the common discourse is : have proper test coverage. But well, we live in a real world, and maintaining a code coverage strong enough to allevaite the problems (around 90%) is just very hard. If you're coding alone, that may be just too much (my case). In a team setting, with money to spend, that may be possible, but you'd need a very disciplined team.
But anyway, AFAIC, working with Python is just super productive (I compare to Java). It also feels much more battle tested than, say, Ruby.
For me python is not a scripting language but a "glue" language set in the middle of a huge libraries ecosystem.
Now, I didn't do XA transaction stuff, high performance stuff, etc. For me it's more alike a well done Visual Basic : you can achieve a lot very quickly. Contrary to VB, the language and its ecosystem are really clean.
I work on a team of three that has a project slightly larger than you 100k-110K LOC (and growing by about 5k LOC a week), we've managed to keep test coverage at about 95%, and have found it's worth the investment upfront, as it makes refactoring so much easier.
Looking back, I don't think even for a personal project, I would ever do something that wasn't a one-off without good test coverage. It's essentially taking on technical debt, as it makes you much more afraid of fixing anything.
I developed this library ( https://github.com/hhuuggoo/thedoctor ) to help deal with this problem. My belief is that type validation is only part of the problem - being guaranteed about properties of your inputs and outputs is also necessary (for example, make sure the input dataframes have datetime indices, and include the following columns, or ensure that this matrix is nonsingular). I have worked on some of the largest python projects at nyc investment banks, I think I've seen what I can safely call the scariest python project in the world
There is also another library out there which approaches the problem a bit differently
>I have a fairly large code base (80K LOC). When the size grows, lack of typing can become a problem.
I do as well, and I find that while we occasionally run into typing bugs, the tests nearly always catch it and they catch it quickly. Moreover, these are tests that we'd write in any language, and static typing would, in most other languages, mean more verbose code. Overall we still come out ahead.
>However, when you end up with meta code, then it can become really difficult to track the types down in the n-th level of recursion...
Yea, I try to avoid that. If there's a library that does meta-code that's unit tested to hell and back, maybe. If I have the time to write one and unit test it, maybe. But, I still try and keep regular projects clear of it.
Statements like "Python is more strongly-typed than Java" can mean too many different things without a precise definition of what "strongly-typed" means. The Wikipedia page linked from the article even supports the position that there isn't an accepted definition for the strong vs. weak! (https://en.wikipedia.org/wiki/Type_system#.22Strong.22_and_....)
These terms are not very illuminating, and I don't understand the post's argument about types as a result, especially w.r.t None vs. null.
One argument that the post might be making, and I'd like to see fleshed out, is "Python's expressions and built-in operators check their inputs at runtime in a way that gives useful and effective error messages in practice." That seems like a lesson from experience that could be backed up with anecdotes and provide some useful feedback on the language. That avoids the terminology debate about types, which is more about picking definitions than about the quality of the language for certain purposes.
It does require defining "useful and effective" for error messages, but I'm more interested in that debate :-)
Python is very much an established languages and myths like these need to go away in order for people to feel comfortable using python for serious, well designed projects that scale and do every and any thing that a person may want or need. Great post.
I love python, and this is a terrible article, as many others have said here.
You have to understand that these articles are written for a number of reasons:
* The company (here, paypal/ebay) needs to recruit people.
* Often, the language or technology is under siege internally, and these external posts strengthen its position.
* The company wants to get some message out there, but it doesn't actually have any interesting technology of its own, so it goes on and on about some known tech.
I was using the Requests library last night to work with some APIs, it's an absolute gem, it's so easy to use. The Python library manager (pip) just works and the idea to isolate your dev environment with virtualenv is fantastic. PEP8 for a universal style guide is underestimated in large projects.
>> Python has great concurrency primitives, including generators, greenlets, Deferreds, and futures.
It's controversial statement.
Generators, greenlets, deferreds, and futures, it's all not great concurrency primitives definitely. It's okay for typical "python" tasks, but many applications have needs in more powerful solutions like golang channels for example.
Mahmoud makes a spirited defense of Python with ten general themes, but the most important thing he had to tell us about PayPal's experience with Python was this:
"Our most common success story starts with a Java or C++ project slated to take a team of 3-5 developers somewhere between 2-6 months, and ends with a single motivated developer completing the project in 2-6 weeks (or hours, for that matter)."
This article overshoots. Yes, Python is slow, and yes, all four runtimes cited are slow. As I've said before, I no longer believe the idea that languages don't have performance characteristics, only runtimes do, and as it happens, the decades-long efforts to speed up Python and their general failure to get much past 10-20x slower than C are a big part of why I believe that. NumPy being fast doesn't make Python fast; it's essentially a binding. A great binding that, if it meets your use case, means you can do great work in Python, but does nothing to help you if you don't need that stuff.
As for PyPy's "faster than C" performance, people really really really need to stop believing anything about a JIT running a single tight loop that exercises one f'ing line of code! Follow that link and tell me if your code even remotely resembles that code. In practice, PyPy is, I believe, "faster than CPython" with a lot of caveats still, but "faster than CPython" isn't a very high bar.
(Similarly, though another topic, Javascript is not a "fast language". People seem to believe this partially because of JIT demonstrations in which integers being summed in a tight loop runs at C speed. But this is easy mode for a JIT, the base level of functionality you expect from one, not proof that the whole language can be run at C speeds.)
There is no version of Python that will run you at anything like C or C++ or Java or Go or LuaJIT speeds on general code. It can't; for one thing you have no choice but to write cache-incoherent code in Python, to say nothing of the numerous other problems preventing Python from going fast. (Copious hash lookups despite the various optimizations, excessive dynamicness requiring many things to be continuously looked up by the interpreter or verified by the JIT, etc.)
I've drilled down on this one, but there several other "debunkings" here that are equally questionable. Python does not have a "great" concurrency story... it has a collection of hacks of varying quality (some really quite good, though; gevent is awesome) that get your around various issues, at the cost of some other tradeoff. The definition of "strongly typed" that Python conforms to is almost useless, because everything is a "strongly typed" language by that definition. In practice, it's a dynamically typed language, and yes, that can cause problems. Another collection of hacks are available to get around that, but they're add-ons, which means the standard library and other libraries won't use or support them. Yes, Python is a scripting language, it's just that it turns out "scripting languages" are a great deal more powerful that was initially conceived.
Wow, I must hate Python, huh? Nope. It's a fantastic language, certainly in my top 3, and still probably underutilized and underrespected despite its general acceptance. When I hear I get to work with it, I generally breath a sigh of relief! It is suitable for a wide variety of tasks and should certainly be in consideration for a wide variety of tasks you may have to solve.
But, it is always bad advocacy to gloss over the problems a language has, and all languages have problems since no one language can solve all problems perfectly. If you are doing something really performance sensitive, stay away from Python. If you've got something highly concurrent, think twice... your problem needs to comfortably fit on one CPU using one of the existing solutions or there's hardly any reason to prefer Python. (Yes, Python can sort of use more than one CPU but if you're going to do that you'll probably be happier elsewhere. "Just throw lots of processors at the problem" isn't generally a good solution when you're starting from a language that can easily be ~50-100x slower than the competition... that's still an expensive answer, even today.) Yes, it is dynamically typed and there are situations where that is OK and situations where it is counterindicated. In the long term, you don't help a language by trying to minimize the problems... you help by making it clear exactly what it is good for, when you should use it, and when you shouldn't. Otherwise, you convince a hapless programmer to pick up your solution, pour a year or two into discovering it doesn't actually do what you said it did, and now you've made an enemy for your language. Better for them to never pick it up because you truthfully told them it wasn't suitable.
That said, though, be sure your task is performance sensitive before just writing Python off... modern machines are really hard to understand the performance of and most people's intuitions are pretty bad nowadays. Dynamic typing has its problems, but so does static. Etc. etc. No easy answers, alas.
It's not. some people are solving this problem with https://github.com/conda/conda. There are many ways to use conda for deployment. My preferred approach is the following:
For builds:
- build a conda environment for your project
- export your conda environment using conda list -e, and then take all the conda packages for those and put them into a single tarball
For deployment
- bootstrap a base python environment using miniconda http://conda.pydata.org/miniconda.html
- install that tarball of packages with conda install <mytarball.tar>
It's not as simple as a jar, but it's reasonably close
I'm the project lead of Crowdcrafting (powered by PyBossa) with 21.7k lines of code written in Python.
Our stack in Crowdcrafting is fairly simple in terms of hardware (we've two servers with 2 GB of RAM each and only 2 cores, while the DBs have 4 GB of RAM with 4 cores). You can read more about it here: http://daniellombrana.es/blog/2015/02/10/infrastructure.html
In my experience the problems that we've had are always related about how we develop, not the technologies. One of our ways of solving problems is avoiding increasing the resources of the hardware as it will "hide" the real issues of your software. Thanks to this approach we were able to store more than 1.5 records per second over a period of 24 hours in 2014 with a DB running on less than 1GB of RAM and our servers with 2GB of them. It worked really well! (Actually after that our provider called us to expend more on our hardware).
Our platform is designed to scale horizontally without problems. We use a very simple set of technologies that are widely used, and we try to use very simple libraries all the time. This has proven to us to be very efficient and we've managed to scale without problems.
We heavily test and analyze our software. This is mandatory. We've almost 1000 tests written covering 97% of our code base. We also analyze the quality of our code using a third party service and we've an score of 93%, and this helps people to help sending patches and new developers joining the platform.
Right now our servers are responding in less than 50ms in average, and we are not even using Varnish (check out this blog post: http://daniellombrana.es/blog/2015/03/05/uwsgi.html). Thus, yes Python is a good language, it can scale, and if you usually have a problem it will be basically in your own code. Hence debug it!
As it pertains to "Enterprise" and Python, I think the most interesting project is OpenStack. I don't use OpenStack but it has been interesting that in conversations with Enterprise devs/managers OpenStack has come up several times. It turns out Enterprise is still really skeptical of the public cloud and are going through the efforts of running their own IaaS/PaaS internally. In my experience OpenStack seems to be coming up much more frequently than CloudFoundry.
And for the record, by Enterprise I'm not talking Silicon Valley Enterprises. These are strictly non-tech Enterprises that traditionally view anything software related as a cost center.
The section on concurrency is a hilarious joke that deftly dodges the obvious drawback of Python's concurrency model while peppering the reader with distracting solutions built on top of multi-processes or single threads. What is missing here? True, multi-threaded (at the OS level) concurrency.
We're engineers here so let's be real and recognize that different tools are suited to different tasks. Limiting oneself to concurrency without OS threads is, to be polite, not necessary. Obviously you CAN build pretty much anything you want with pretty much any tool, as the examples in this article show, but that doesn't mean you should.
I'd also add to this that the government is pretty heavily invested into python (at least NASA, DoD, and DoE based on direct experience). It's kind of becoming the scientist's and engineer's new Fortran.
[+] [-] plinkplonk|11 years ago|reply
Like all languages, Python has strengths and weaknesses, and there is no shame in that. An honest article would address the negatives head on, acknowledge them as potential negatives,(not skip around them) and provide alternatives, .
The strawman "Python has a weak type system" is a good example of such deflection.
No one (who understands type systems) would complain that "Python has a weak type system"
A more common "criticism" would be "Python does not have a static type system, which is handy for large codebases worked on by large teams".
Address that upfront, and you have a decent article. This is just a fanboy (who just happens to be at Paypal) praising his favorite language and ignoring its weaknesses.
[+] [-] dvanduzer|11 years ago|reply
Paypal is a large organization which needs to hire Python programmers. Everyone reading this now has a reminder that Python is popular at Paypal. We also know that endless PLT arguments are probably not common in the workplace.
[+] [-] TorKlingberg|11 years ago|reply
I have not seen anyone complain that Python is not compiled for at least a decade, but maybe the author has.
[+] [-] Dewie|11 years ago|reply
[+] [-] mapgrep|11 years ago|reply
Because I don't believe you really, truly understand a language until you can tell me what sucks about it. It takes significant time (in a reasonably decent language) to discover the corner cases, performance bottlenecks, quirks, big-deficiencies-hidden-in-plain-sight and outright bugs in a language like, say, python.
Something as "rah-rah" this, which goes so far as to basically call the GIL a source of unicorns and rainbows, is convincing almost in inverse proportion to its stridency. Suddenly I'm wondering if all these "myths" about python might be smoke pointing toward a fire. That's probably a bit unfair, but it's hard to know what to take seriously when you're listening to a voice that's less than credible.
[+] [-] johan_larson|11 years ago|reply
The language really excelled at small projects, where the expressiveness of the language and the excellent libraries available really let us get a lot done with little code and time. Avoiding the recompile/redeploy steps also helped.
But using the language for larger projects was quite a different thing. Suddenly you could find yourself in code several layers down, being passed an object of God knows what type from unfamiliar code, and having to find out what you were receiving essentially by trial and error. And once the execution times started to climb, it became increasingly frustrating to see trivial problems that would have been caught during compilation in a statically typed language appear in Python only after 30-60 minutes of execution.
There are surely ways to alleviate these specific problems -- I can think of some things to try myself -- but my experience suggests that the sweet spot for dynamic languages like Python is in projects that fit between one pair of ears, and stricter statically typed languages become increasingly useful as things scale up.
(Hardly a radical position, I know.)
[+] [-] crdoconnor|11 years ago|reply
The GIL is definitely no source of unicorns and rainbows, but I think the case against it is usually overstated. There are numerous ways of sidestepping it (multiprocessing, pypy, C extensions, etc.), and it does serve a useful purpose.
[+] [-] patrickmay|11 years ago|reply
It's a good fanboy filter.
[+] [-] jrochkind1|11 years ago|reply
But I think he overstates the case.
Calling Python "compiled" doesn't seem right -- he's more right that this doesn't matter, there's no reason not to use something because it's "not compiled".
Calling Python's typing "strong typing", eh, really? He provide a link to a wikipedia article on typing in general, how about a link to anyone else making the case that Python should be considered 'strongly typed', for a particular definition of 'strongly typed'? I'm dubious that it would be any definition of 'strongly typed' that those who think they want it... want. Again, I think a better argument might be: So what, you don't need strong typing for success, because of a b and c.
And the concurrency discussions is just silly. Sure, you can do _all sorts_ of things without multi-core concurrency, you may not need it or even benefit from it, and scaling to multi-process can sometimes work just fine too. But there are also _all sorts_ of cases where you do really want multi-core parallelism, and Python really doesn't have it -- and trying to justify this as "makes it much easier to use OS threads" is just silly. I guess it makes it 'easier' in that it protects you from some but not all race conditions in fairly unpredictable ways -- but only by eliminating half the conceptual use cases for threads in the first place (still good for IO-bound work, no longer good for CPU-bound work).
I think there's a reasonable argument to be made that Python will work just fine for 'enterprise' work. There are also, surely, like for all languages/platforms, cases where it would be unsuitable. By overstating the case with some half-truths and confusions, he just sounds like a fanboy and does not help increase understanding or accurate decision-making -- or confidence in the Python community's understanding!
[+] [-] hyperpape|11 years ago|reply
1) It's meaningless--it's almost impossible to produce a real ranking of languages on the strength of their types.
2) These dynamic languages are all unityped because they're not statically typed. The fact that runtime tags won't allow certain operations to succeed has nothing to do with types.
3) Strongly typed is a rough and ready way of saying you don't do many implicit conversions.
I sympathize with (1). (2) is accurate, but really a terminological point. There's clearly a phenomenon to talk about, even if "typed" is the wrong word. I also sympathize with (3).
Python does far fewer implicit conversions than PHP, JavaScript or Perl. It even doesn't do at least one that Java does. So people often say Python is strongly, but dynamically typed. It's a real aspect of the language that you can view as a plus or minus.
[+] [-] dgemm|11 years ago|reply
[+] [-] wbkang|11 years ago|reply
[+] [-] eliben|11 years ago|reply
[+] [-] agentultra|11 years ago|reply
[0] https://wiki.python.org/moin/ParallelProcessing
[+] [-] slantedview|11 years ago|reply
[+] [-] slantedview|11 years ago|reply
These two statements conflict.
[+] [-] gamesbrainiac|11 years ago|reply
Python's proposal for type hinting will dampen the effect of type inconsistency I feel.
[+] [-] lmm|11 years ago|reply
[+] [-] go1979|11 years ago|reply
While there are a lot of concurrency options to not get you in trouble with the GIL, it is annoying that there is no canonical way or pythonic way of doing it. I saw people using things like monkey patching, which made my jaw drop. I had to implement Websockets recently and was totally confused as to the right way of doing it. I ended up evaluating NodeJS and Go for this function, and went with Go.
Overall ... I love Python. It is perfect for small projects. For larger codebases (3KLOC+), I am slightly skeptical. It can be done but developers need to have quality code and strong test coverage.
[+] [-] eliben|11 years ago|reply
Weak typing has its merits but sometimes static typing is very useful for reading and understanding code.
[+] [-] s_kilk|11 years ago|reply
I know we can get picky about the exact characteristics of a languages type-system, but for me they fall into two camps: "Typeful" languages and "(effectively) Typeless" languages.
The former have a "good" typesystem, and expose it as a tool the programmer can use to help themselves solve problems. In the latter case, there may or may not be a type system lurking in the language somewhere, but the language either doesn't expose that system as a tool for the programmer to use, or it does so inconsistently.
Personally I think python is in the later camp, it has types but it doesn't really give the programmer much help when solving problems.
Clojure is another example. The Java type-system is down there somewhere in the murk, but idiomatic Clojure code doesn't use it, and it's not really a tool for the programmer to use, more an accident of the platform.
[+] [-] wyldfire|11 years ago|reply
> type hinting will dampen the effect of type inconsistency I feel
I'm not sure that I see type hinting being that advantageous. I kinda hope that it could allow PyPy and other VMs to make assumptions that would otherwise require heuristics.
Perhaps type hinting could make static checkers/linters for Python more effective, though.
[+] [-] unknown|11 years ago|reply
[deleted]
[+] [-] kyllo|11 years ago|reply
This is why I like dynamic languages that have immutable-by-default data structures, like Clojure.
[+] [-] mox1|11 years ago|reply
In python I would suggest wrapping your call in a try / expect block or using "isinstance()".
If you are using a publicly available library or popular piece of code that can return None or an Integer, I would argue that piece of code was written incorrectly. Newbie python users might do this, but I think experienced python devs would see the problem.
Finally, some documentation or justification for the reasoning behind this decision at the top of the function or on a web page somewhere would help as well.
As with most powerful , full featured languages it is pretty easy to shoot yourself in the foot if you are not careful, I don't think this is a python specific problem...
[+] [-] robmccoll|11 years ago|reply
Not saying it's not possible and doesn't work well for a lot of companies, but I do think you take on technical debt to get up and running fast when you choose Python. It's a choice I would personally think twice about.
[+] [-] jammycakes|11 years ago|reply
The main problem isn't anything to do with technical failings of the language, but a misconception that it's an obscure niche language that nobody uses outside of academia, with an ecosystem bordering on the nonexistent. People seem to think that if you start using Python you're going to end up with an incomprehensible codebase that's impossible to maintain because you can't hire developers who know it. They're generally quite surprised when I tell them how widely used it is and what all it gets used for.
[+] [-] meddlepal|11 years ago|reply
[+] [-] solve|11 years ago|reply
I can almost see these guys' angle, of why they might see this as a possible threat that could undermine their career. .NET coding pays well, and is usually semi-exclusive to those who previously had an employer pay for them to use the tools, even though that's not theoretically necessary.
[+] [-] robwilliams|11 years ago|reply
[+] [-] frownie|11 years ago|reply
Python2 to 3 migration is not easy. There are tools but the problem is that they don't see everything and therefore, you end up with 95% of your code converted. Then it's up to you to figure out the last 5%, which is an order of magnitude harder that the first 95%... So I ended up having a fairly long transition of migrating Python2 code to Python2+3 code.
For bothe these issues, the common discourse is : have proper test coverage. But well, we live in a real world, and maintaining a code coverage strong enough to allevaite the problems (around 90%) is just very hard. If you're coding alone, that may be just too much (my case). In a team setting, with money to spend, that may be possible, but you'd need a very disciplined team.
But anyway, AFAIC, working with Python is just super productive (I compare to Java). It also feels much more battle tested than, say, Ruby.
For me python is not a scripting language but a "glue" language set in the middle of a huge libraries ecosystem.
Now, I didn't do XA transaction stuff, high performance stuff, etc. For me it's more alike a well done Visual Basic : you can achieve a lot very quickly. Contrary to VB, the language and its ecosystem are really clean.
I'm lovin' it
[+] [-] jeffasinger|11 years ago|reply
Looking back, I don't think even for a personal project, I would ever do something that wasn't a one-off without good test coverage. It's essentially taking on technical debt, as it makes you much more afraid of fixing anything.
[+] [-] hogu|11 years ago|reply
There is also another library out there which approaches the problem a bit differently
https://andreacensi.github.io/contracts/
[+] [-] crdoconnor|11 years ago|reply
I do as well, and I find that while we occasionally run into typing bugs, the tests nearly always catch it and they catch it quickly. Moreover, these are tests that we'd write in any language, and static typing would, in most other languages, mean more verbose code. Overall we still come out ahead.
>However, when you end up with meta code, then it can become really difficult to track the types down in the n-th level of recursion...
Yea, I try to avoid that. If there's a library that does meta-code that's unit tested to hell and back, maybe. If I have the time to write one and unit test it, maybe. But, I still try and keep regular projects clear of it.
[+] [-] jpolitz|11 years ago|reply
Statements like "Python is more strongly-typed than Java" can mean too many different things without a precise definition of what "strongly-typed" means. The Wikipedia page linked from the article even supports the position that there isn't an accepted definition for the strong vs. weak! (https://en.wikipedia.org/wiki/Type_system#.22Strong.22_and_....)
These terms are not very illuminating, and I don't understand the post's argument about types as a result, especially w.r.t None vs. null.
One argument that the post might be making, and I'd like to see fleshed out, is "Python's expressions and built-in operators check their inputs at runtime in a way that gives useful and effective error messages in practice." That seems like a lesson from experience that could be backed up with anecdotes and provide some useful feedback on the language. That avoids the terminology debate about types, which is more about picking definitions than about the quality of the language for certain purposes.
It does require defining "useful and effective" for error messages, but I'm more interested in that debate :-)
[+] [-] bmoresbest55|11 years ago|reply
[+] [-] kzhahou|11 years ago|reply
You have to understand that these articles are written for a number of reasons:
* The company (here, paypal/ebay) needs to recruit people.
* Often, the language or technology is under siege internally, and these external posts strengthen its position.
* The company wants to get some message out there, but it doesn't actually have any interesting technology of its own, so it goes on and on about some known tech.
[+] [-] cdnsteve|11 years ago|reply
I was using the Requests library last night to work with some APIs, it's an absolute gem, it's so easy to use. The Python library manager (pip) just works and the idea to isolate your dev environment with virtualenv is fantastic. PEP8 for a universal style guide is underestimated in large projects.
[+] [-] kreshikhin|11 years ago|reply
It's controversial statement.
Generators, greenlets, deferreds, and futures, it's all not great concurrency primitives definitely. It's okay for typical "python" tasks, but many applications have needs in more powerful solutions like golang channels for example.
[+] [-] raymondh|11 years ago|reply
"Our most common success story starts with a Java or C++ project slated to take a team of 3-5 developers somewhere between 2-6 months, and ends with a single motivated developer completing the project in 2-6 weeks (or hours, for that matter)."
[+] [-] sago|11 years ago|reply
[+] [-] jerf|11 years ago|reply
As for PyPy's "faster than C" performance, people really really really need to stop believing anything about a JIT running a single tight loop that exercises one f'ing line of code! Follow that link and tell me if your code even remotely resembles that code. In practice, PyPy is, I believe, "faster than CPython" with a lot of caveats still, but "faster than CPython" isn't a very high bar.
(Similarly, though another topic, Javascript is not a "fast language". People seem to believe this partially because of JIT demonstrations in which integers being summed in a tight loop runs at C speed. But this is easy mode for a JIT, the base level of functionality you expect from one, not proof that the whole language can be run at C speeds.)
There is no version of Python that will run you at anything like C or C++ or Java or Go or LuaJIT speeds on general code. It can't; for one thing you have no choice but to write cache-incoherent code in Python, to say nothing of the numerous other problems preventing Python from going fast. (Copious hash lookups despite the various optimizations, excessive dynamicness requiring many things to be continuously looked up by the interpreter or verified by the JIT, etc.)
I've drilled down on this one, but there several other "debunkings" here that are equally questionable. Python does not have a "great" concurrency story... it has a collection of hacks of varying quality (some really quite good, though; gevent is awesome) that get your around various issues, at the cost of some other tradeoff. The definition of "strongly typed" that Python conforms to is almost useless, because everything is a "strongly typed" language by that definition. In practice, it's a dynamically typed language, and yes, that can cause problems. Another collection of hacks are available to get around that, but they're add-ons, which means the standard library and other libraries won't use or support them. Yes, Python is a scripting language, it's just that it turns out "scripting languages" are a great deal more powerful that was initially conceived.
Wow, I must hate Python, huh? Nope. It's a fantastic language, certainly in my top 3, and still probably underutilized and underrespected despite its general acceptance. When I hear I get to work with it, I generally breath a sigh of relief! It is suitable for a wide variety of tasks and should certainly be in consideration for a wide variety of tasks you may have to solve.
But, it is always bad advocacy to gloss over the problems a language has, and all languages have problems since no one language can solve all problems perfectly. If you are doing something really performance sensitive, stay away from Python. If you've got something highly concurrent, think twice... your problem needs to comfortably fit on one CPU using one of the existing solutions or there's hardly any reason to prefer Python. (Yes, Python can sort of use more than one CPU but if you're going to do that you'll probably be happier elsewhere. "Just throw lots of processors at the problem" isn't generally a good solution when you're starting from a language that can easily be ~50-100x slower than the competition... that's still an expensive answer, even today.) Yes, it is dynamically typed and there are situations where that is OK and situations where it is counterindicated. In the long term, you don't help a language by trying to minimize the problems... you help by making it clear exactly what it is good for, when you should use it, and when you shouldn't. Otherwise, you convince a hapless programmer to pick up your solution, pour a year or two into discovering it doesn't actually do what you said it did, and now you've made an enemy for your language. Better for them to never pick it up because you truthfully told them it wasn't suitable.
That said, though, be sure your task is performance sensitive before just writing Python off... modern machines are really hard to understand the performance of and most people's intuitions are pretty bad nowadays. Dynamic typing has its problems, but so does static. Etc. etc. No easy answers, alas.
[+] [-] njharman|11 years ago|reply
I've had projects fail cause of not getting done, being buggy, but never for not being fast enough.
Finally, slowness, until you get really low-down is relatively easy problem with many solutions.
[+] [-] geekam|11 years ago|reply
[+] [-] hogu|11 years ago|reply
For builds: - build a conda environment for your project - export your conda environment using conda list -e, and then take all the conda packages for those and put them into a single tarball
For deployment - bootstrap a base python environment using miniconda http://conda.pydata.org/miniconda.html - install that tarball of packages with conda install <mytarball.tar>
It's not as simple as a jar, but it's reasonably close
[+] [-] vetinari|11 years ago|reply
[+] [-] teleyinex|11 years ago|reply
Our stack in Crowdcrafting is fairly simple in terms of hardware (we've two servers with 2 GB of RAM each and only 2 cores, while the DBs have 4 GB of RAM with 4 cores). You can read more about it here: http://daniellombrana.es/blog/2015/02/10/infrastructure.html
In my experience the problems that we've had are always related about how we develop, not the technologies. One of our ways of solving problems is avoiding increasing the resources of the hardware as it will "hide" the real issues of your software. Thanks to this approach we were able to store more than 1.5 records per second over a period of 24 hours in 2014 with a DB running on less than 1GB of RAM and our servers with 2GB of them. It worked really well! (Actually after that our provider called us to expend more on our hardware).
Our platform is designed to scale horizontally without problems. We use a very simple set of technologies that are widely used, and we try to use very simple libraries all the time. This has proven to us to be very efficient and we've managed to scale without problems.
We heavily test and analyze our software. This is mandatory. We've almost 1000 tests written covering 97% of our code base. We also analyze the quality of our code using a third party service and we've an score of 93%, and this helps people to help sending patches and new developers joining the platform.
Right now our servers are responding in less than 50ms in average, and we are not even using Varnish (check out this blog post: http://daniellombrana.es/blog/2015/03/05/uwsgi.html). Thus, yes Python is a good language, it can scale, and if you usually have a problem it will be basically in your own code. Hence debug it!
Cheers,
Daniel
[+] [-] rubiquity|11 years ago|reply
And for the record, by Enterprise I'm not talking Silicon Valley Enterprises. These are strictly non-tech Enterprises that traditionally view anything software related as a cost center.
[+] [-] slantedview|11 years ago|reply
We're engineers here so let's be real and recognize that different tools are suited to different tasks. Limiting oneself to concurrency without OS threads is, to be polite, not necessary. Obviously you CAN build pretty much anything you want with pretty much any tool, as the examples in this article show, but that doesn't mean you should.
(written by someone who writes Python daily)
[+] [-] thearn4|11 years ago|reply