Python on Wheels

[shadowmint]

Wow, great article. I've heard of wheels for months, but I've never really understood why I should remotely care about them before (it always seemed to me they solved a problem that didn't exist; namely, binary distributions for pure python packages, since you can't support c-plugin packages cleanly with wheel anyhow...).

Turns out the answer is:

1) Yes, you can kind of support packages with c-plugins, if you're specific and careful.

2) They make server deployments significantly faster.

That's actually pretty neat.

...but damn, they still look like a massive pain to work with.

[brabram]

I confirm about the pain. They should be transparent and you shouldn't need to care about them but that's absolutely not the case. Also, it's the responsibility of the pypi maintener to push wheels on it, but none do this, so you end up building your own wheels while this should have already been done on pypi and it's stupid. And one last thing: https://twitter.com/mitsuhiko/status/426700148409135104 "It turns out, Python binary wheels on Python 2 are rejected by PyPI, I suppose because of the lack of UCS tag in the filename."

Apart from all of this pain, I'm very happy to be able to use them.

I'm really waiting for someone to build a "wheel on demand" website that would be a proxy to pypi. Or to turn pypi into a wheel farm (I don't know how realist this idea is).

[themartorana]

After being bummed by the removal of 'bundle' in pip and writing an article about how we moved to using wheel for faster deployments[1], we moved to using Packer for server builds for our cloud infrastructure. Now we can backup to using a simple 'requirements.txt' file and not worrying about install and compile speeds.

Removing 'bundle' was painful for us, and I even contributed a bunch of code to 'pip2pi' [2] to make it easy to set up an S3 bucket as a "local" pip mirror for our EC2 infrastructure, but it was all bandaid after bandaid.

So far, Packer has felt much more elegant (despite the effort it took to put it in to our workflow) than wheel, pip2pi, or any other solution.

[1] http://tech.flyclops.com/replacing-pip-bundle-374

[2] https://github.com/wolever/pip2pi

(Edited for grammar.)

[girvo]

For a language that prides itself on simplicity, this seems really painful...

[pak]

Yes, it is. Coming to this mess from the joy of bundler/gems, npm, or hell, even CPAN will ruin your afternoon.

It is just an unfortunate circumstance of the community not efficiently converging on well designed standards quickly enough, and so, the rocky soil has sprouted strange twisty crops. There are articles on how the setuptools/distutils frankenhack pretty much set off the whole trainwreck that you see today, e.g.: http://blog.startifact.com/posts/older/a-history-of-python-p...

Armin (the OP) has also written about this sad history previously: http://lucumr.pocoo.org/2012/6/22/hate-hate-hate-everywhere/

[sdegutis]

Managing dependencies correctly is probably the most important long-term problem to solve in programming, but for some reason it always seems to be given a low priority by language authors and communities, until one day it's become a tangled mess that isn't logistically possible to get out of. I don't get it, and it's frustrating.[1]

[1]: Somehow, all my lines aligned perfectly in this comment box at its default size! https://www.dropbox.com/s/slq8x0rqkj7ydzn/aligned.png

[donaldstufft]

A great deal of the pain that Armin experienced is mostly due to the fact that Wheels are very new and aren't very polished yet. Additionally there is a vast wealth of technical debt in the packaging tools so it's quite easy to introduce these kinds of problems because of the general architecture (Something that we're trying to fix long term!).

[gbog]

You know that simplicity of use is very complex to achieve, and requires complex tools behind, right?

Actually it is even true for music instruments: A piano is very complex inside and can be played relatively easily, while a violin is much more simple in the fabric but much harder to play.

[Too]

It is like a farce on the "there shall only be one way to do it" mantra. Not to mention that some of the package managers has to be installed through one of the other packet manager, creating a very long dependency chain if you want to install a package through the former.

[sashk]

Spent some time this week to move deployment of venvs to use wheels (we install numpy and matplotlib). Now, instead of close to 10 minutes deployment time, it's done in about 30 seconds. I can live with the fact, that I'll need to make changes to the deployment process now and then, while wheel traveling towards 1.0, but I'm happy to go that route. And you should to.

[gtaylor]

Isn't there a way to make a relocatable virtualenv? I'd be inclined to just do that, compress it up and distribute the entire virtualenv (our deploy environment is homogenous to the extreme).

[e12e]

A great introduction to wheels. I'm still not sure if wheels really is a good idea though. For simple use-cases traditonal pip seems fine, for more complex use-cases buildout seems better? Having tried to install numpy and pygame in a virtualenv I do see a need for something better -- but I'm not convinced wheels will solve the problem of complex c-library dependencies...?

[travisoliphant]

This is a nice post, and the Python packaging community is making strides in the right direction.

However, it's not there yet. Armin could bring himself to say this in this post: "It's there, it kinda works, and it's probably worth your time."

Conda is not perfect (it needs more people building conda binaries as well as a few pull requests to add https improvements) --- but I will confidently say "it is there, it does work now, and it is worth your time"

This is especially true with a package repository like exists at http://repo.continuum.io and the many personal repositories at http://binstar.org.

People that have used conda for deployment (especially of complex C-dependencies in the NumPy and PyData stack) are saving time and effort today.

[mariocesar]

I use wheels to speed up the tests for sorl-thumbnail https://github.com/mariocesar/sorl-thumbnail/blob/master/.tr... Initially I was into fully use wheel to install the environment but I found that some packages didn't work, now I just use it for just some critical packages and reduce the 12 minutes test to 3 minutes

Will love to see wheels getting more attention, really feels like a great direction for python packaging.

[travisoliphant]

A lot of people in the PyData ecosystem using NumPy and Pandas are using conda and conda packcages (http://conda.pydata.org). Conda packages are easy to build and many exist at repo.continuum.io and on channels at http://binstar.org

[jpdlla]

Love sorl-thumbnail, thanks for all the great work!

[beggi]

Page is currently down, so here: http://webcache.googleusercontent.com/search?q=cache:http://...

[overgard]

Ugh. I'm so sick of the open source communities' disdain for binaries. There's no reason we should be recompiling shit everywhere. It's a waste of time, headspace, and processing power.

[Pxtl]

When you're trying to support multiple architectures and OSes and your ABI took a decade or two to settle down, and all your tools are open-source so you have the source anyways, I can see the instinct to just say "I'll just compile it against the target machine" even if it's absurdly wasteful.

[raptium]

I thought it was another web framework ...

[r0muald]

Just like Ruby on Rails :)

[_jb]

Google cache link: http://webcache.googleusercontent.com/search?q=cache:0rftZaF...

[pippy]

I've had the exact same experience as the author of this post. Eggs are great if you just want to install a python library. Horrible if you want to modify the library or distribute a program.

I personally use a mixture of setuptools and distutils2, and it's a confusing mix.

[asmeurer]

"What the command is supposed to do is to collect all the dependencies and the convert them into wheels if necessary..."

How does it convert the non-wheel packages into wheels?

[amouat]

On a slight tangent; does anyone else think that Docker is probably a better solution than virtualenv in most cases? I never really got on withe virtualenv.

[travisoliphant]

You can also try conda's environments which are a system-level concept and is an even lighter-weight approach to some of the problems people are using Docker to solve. If you just need an independent software environment, conda gives you that while sharing files that can be shared. http://conda.pydata.org

[dagw]

The subset of problems that virtualenv solves that are also solvable by Docker is pretty tiny. But sure if you have problem that can easily be solved by Docker, then Docker is probably a better solution.

[DrJ]

what about pex?