I'm not saying pyenv is not a useful tool, but it is not a tool for beginners fighting with python packaging problems. It's a specialist tool to normalize your setup.
Very often, I see people tell me they don't have problems with pyenv, but later on have other unrelated problems with their dependencies. Analysis then prove it was because of pyenv, they just didn't know it. The cost is not obvious.
Actually, it only builds it locally if it can't find a pre-packaged version for your system/arch. Admittedly that's most of the recent ones on a Mac, but there is a difference (I've been using pyenv for nearly ten years[1] now).
The big advantage for me is that I can match whatever runtime and standard library a target has (and yes, that's needed more times than not, even in this new age of Docker).
Additionally, you can build an _optimized_ Python. I have this set for my builds:
If you don’t care which specific version of python you are using, do not use pyenv.
You will know when you care. These days, doing common tasks, the constraint solver in poetry will often tell you:
“Hey! I can’t find a version of sentencepiece with metadata that lets me combine this with python 3.12.2. Sorry! I give up!”
Now, if you aren’t concerned with using your bare metal for CUDA or fancy new MPS or AMD stuff. Just ignore this and use containers. I’d use podman compose.
However, I use pyenv on every machine. Because it compiles specific versions I actually need, even to create various virtual environments. If compiling python automatically sounds tough, you probably don’t need to anyway.
To describe the problem you’d see. I try to use poetry by default, though I think it became popular before it was PEP-compliant or useful in devops. It is impossible to control the behavior of other package managers, and poetry is/was strict about that. Which means you can’t force deploy in many cases. (Better lately.)
For the problem pyenv helps to solve, I back-up my pyproject.toml setuptools backend with pip and requirements.txt. These days, requirements-cuda.txt and requirements-mps.txt.
The landscape is still a disaster for binary compatibility, but it can be done lol. (I’ve been doing python packaging professionally since prom, which was python 2.6 give or take.)
So, what is "the tool for beginners fighting with python packaging problems"?
That is, the pattern seems to be that someone mentions a solution, then a zillion responses as to why it sucks.
Is there any tool or pair that sucks least for most cases and beginners? I get that every case is different, but perhaps there are some useful starting points?
If you use Homebrew: you can use ‘brew pyenv-sync’ to use Homebrew’s pythons with pyenv. Similar commands are available for rbenv/nodenv (which always feel like it is missing an ‘e’ to me)
I think that you are painting a bit of an unreasonably bleak view of pyenv. I think that one can easily get value out of it without being a Python expert. I’m not sure how I can refute your “yes, but you’ll eventually run into trouble. You just haven’t clocked enough hours yet”. I can’t prove a negative. But I’ll say that I’ve been writing Python in my day job for a decade.
But, to add to the list of problems, these Python versions IIRC do not compile with optimisation turned on, so they’re by default quite a bit slower than they need to be.
When the God Of Programming made Python, all other languages were jealous of its elegance, simplicity and intuitive beauty. When the other programming languages went to complain he said..."Wait until you see what package systems I will give them...They will never get an environment properly setup..." :-)
Oh man, you just confirm that I’m not crazy. For every single time I use pyenv it just downloads source and build. I got tangled into so many issues such as TLS headers in my AL2 boxes. I thought the whole thing is about getting the official binary and install and not compile from source even for a 2 year old release.
Can I ask - why the dislike for virtualenvwrapper? If you are saying that it can have occasional problems, I agree with you. But it makes the process so much simpler. The occasional problems I’ve had (had an issue once or twice deleting a venv) pales in comparison to the advantage I get in remembering a couple fast commands. Is there something better or am I relegated to “source what/directory/was/it/again/bin/activate?
Just curious: what are the downsides of poetry installed with pipx? The article mentions having to install poetry in another venv, but that's hardly an issue with pipx (you just add an 'x' after 'pip'), and installing pipx is as straight-forward as it can be.
What's the problem? The repo clearly states that you need to install such and such build prerequisites.
Also, the issues the article you link are things that damn near every programmer will eventually need to know. Things like PATH are IMO the basics -- if you don't understand this or how to `$packageManager install` something, you're gonna have a rough time programming in general, regardless of language.
Tools you can use to make sure the Python program you wrote keeps working: requirements.txt, pip, pipenv, pyenv, virtualenv, pyenv-virtualenv, virtualenvwrapper, pyenv-virtualenvwrapper, venv, pyvenv, conda, miniconda, poetry, docker, nix.
Which ones did I miss? Which of them actually ensure your program always works the same as when you first wrote it, without asterisks?
You've missed: pdm, uv, pip-tools, pipx, rye, and probably some others.
Only pdm and poetry generate cross-platform lock files by default as far as I know, but there are a lot of people trying to solve this problem right now.
It's not an easy problem to solve. Python's package management predates package managers from most other programming languages and Python itself predates Linux. There is a lot of baggage so change is very slow.
I am always curious how many people, outside of those building code for third party clients, actually hit this problem? In the 10+ years of using Python I have never had a problem using the core tools. The ecosystem is far from perfect but it has never cause me a problem.
Edit: Wow y'all are some sour people for voting down this question. I truly wonder how often people run into this problem compared to just complaining about it.
You didn’t miss Poetry, but I have to say up until I started using Poetry python in large projects was a pain in the tooling department to setup and maintain over longer periods of time.
It’s no panacea, but feels more stable and usable (especially from onboarding new team members PoV) than other tooling I’ve tried
What package managers are people using in other languages to make sure that software "always works the same as when you first wrote it, without asterisks"? I'd like to understand how they solve the "package no longer exists in a central registry" problem.
> Which of them actually ensure your program always works the same as when you first wrote it, without asterisks?
There aren't such tools. Python not being a standard and heavily reliant on the OS that runs it and on third-party components that are also not standard leaves you with no choice by to "be at the wheel" all the time. Virtually anything written in Python will go stale in a mater of few years. In other words, you need to constantly update and test as the environment changes just to stand still.
I really find it quite astounding just how much absolute nonsense python programmers are willing to put up with.
Needing more than one, MAYBE two versions of your language installed is insanity. If I build a Python widget and send it to my non-programmer coworker, there is exactly zero chance it will work until I walk over and manually set up the correct language. Instead I use a language that natively compiles to an exe.
Python is pretty neat, but the concept of a program that just works anywhere is so utterly alien that I genuinely cannot find any practical use for it.
I can't just install a pip package to use globally, I have to set up some goddamn virtual environment because everything is so brittle that nothing works without the exact correct language version.
It's like NPM all over again. Dependency hell is so bad that your package manager is now a dependency so we need a package manager manager to manage installing the correct package manager version to then install the correct packages.
Every single time I've come up to some Python thing or other, I spend about fifteen minutes fucking with it before giving up and using a tool built in a sane language.
The fact that I can't just send any random person a program and expect it to run at all is just lunacy. And no, compiled Python does not count because that's even more brittle tooling on top of all the other bullshit I have to deal with. If it doesn't work out of the box, it's a bad tool and I have far better things to do with my very valuable time.
Plug for Mise (https://mise.jdx.dev/) which is like asdf (which is like pyenv, but for any language you might want all in one tool) but written in Rust and without shims (by default) such that it’s much faster at running the activated runtime. It installs runtimes in parallel and will even download the correct plugin if you don’t already have it installed. It can also replace direnv, which I haven’t done because I’m using it as a drop-in replacement for asdf with the .tool-versions files my team already has in every repo (it supports .python-version as well if you’ve been using pyenv). It works great as a “better asdf”/“better pyenv” for my purposes.
https://asdf-vm.com/ ASDF is better because it works with many more languages, other than only Python, like Rust, Go, Node, etc, and other tools, such as AWS/Google/Firebase/Azure CLIs.
After some trial and error I have now settled on the following "stack": 1)I manage python versions with pyenv 2)For each new project I create a new virtualenv with venv "PYENV_VERSION=3.10 python -m venv .venv" 3)Then I start jump into the venv and initiate the project with Poetry ("poetry init -n") and manage dependencies with Poetry.
If I'm keeping the projects under Dropbox, then I'll just add the .venv folder to the ignore list ("attr -s com.dropbox.ignored -V 1 .venv") to reduce the amount of files that need to be synced. If I need to get back to old project, I simply recreate the venv and install dependecies using Poetry.
A good habit would be to add a .pythonversion file on the project folder. Pyenv can pick that up and it is then obvious which version was used for the project.
I do development on Windows, but run all the Python stuff on WSL2 and use VS code for notebooks and everything. One thing I haven't looked at is using pipx to manage Poetry. Now I simply have one Poetry version installed.
Pyenv has worked fine once I managed to collect all the necessary dependencies to Ubuntu. For me it's not just to switch between Python 3.x. I like the fact that you can easily have different patch versions and match exactly the version I would use in production (via containers).
OP - I'm curious why you've submitted this today, you seem experienced enough to guess this is not the first time you've used it. Has there been a major change or something happened in the community I need to know about? Or are you just spreading the love for a great piece of open source software?
I used pyenv in my previous company. At first it was fine, but after a while with multiple versions of Python installed, things stopped working. The right virtual env was not activated, etc. (It could have also been because our software updates like OS and security upgrades were pushed by desktop support). I removed all of it, and just resorted to installing multiple versions of Python the old way (downloading from python.org) and then use `python<version> -m venv /my/virtual/env` to manage my virtual environments. Things are more stable, and I don't feel like its magic. I am not going back to pyenv.
I've given up on trying to manage runtime dependencies with language specific tooling. Instead, I've moved to mise which handles the majority of them (Python, Node, Ruby, Terraform, etc.) the same way. It will also activate Python virtual envs:
There has been tons of churn in the Python project management space, and I feel like I've been blissfully unaware of all of it with this workflow. Can't recommend enough.
Why regular Python binaries and not pyenv? I've had enough pyenv snafus and meltdowns that I started looking for alternatives, and it turned out I could just install whatever Python binary I wanted and specify it wherever I wanted. What could be easier?
python -m venv is fine, I've just gotten used to virtualenv.
pyproject.toml is the future and almost everything supports it pretty well; it also has fewer implicit weirdnesses than setup.py or setup.cfg.
pip-tools are really simple, and they let you have hashed dependencies which are super important IMO.
I have been using pyenv for years and it's the best tool for managing Python versions. You can install any version, switch between them, use different versions for different projects, have multiple versions installed at the same type without problems. It really solved all my Python version managament problems, I never looked for any other tool like this since I started using it.
I've been using python on and off since Django 0.96, which was released in 2007.
I don't recall ever needing anything other than virtualenvwrapper and pip—and even some of the annoyances these tools had early on have been solved by now...
If you really need different versions of python, you can just `mkvirtualenv -p python3 venvname`
I feel like every other tool out there has to explain what problem they solve that virtualenvwrapper doesn't
Don't install anything globally, creates lots of envs, and feel free to have different versions of python installed side-by-side with some "main" version preferably symlinked as `python` and `python3`
I used to use this a lot when working on a Windows machine. Worked pretty well. But nowadays Nix solves the same problem in a fully general way, for all software rather than a single language. You can have whatever versions of whatever you want, side by side, without interfering with each other.
This is the way. I moved everyone I work with at over to the pyenv, virtualenv, poetry stack. Once you get set up it’s pretty smooth sailing. It is a shame that so many tools are needed to work with python, but I was happy to finally find something that works.
This is a truly, truly terrible idea. It adds several failure modes, some subtle so you can go a long way in a state of error, just so beginners can type `python` instead of e.g. `python3.10`.
Many developers, not just me, have a similar setup: we use virtual environments everywhere, and if you aren't in one, `python` doesn't even resolve to a symbol.
If I want to write a quick script with no dependencies, I directly call `python3.xx` on it. Otherwise, I create a virtualenv.
Yes, it's a bit harder for beginners, but from a huge amount of experience helping people who are starting up in programming, people have little issue in following a few more instructions. What demolishes beginners is getting into a bad state where nothing works and you don't know why.
[+] [-] BiteCode_dev|1 year ago|reply
The number of possible modes of failure in this situation is huge.
See also: "Why not tell people to "simply" use pyenv, poetry or anaconda"
https://www.bitecode.dev/p/why-not-tell-people-to-simply-use
I'm not saying pyenv is not a useful tool, but it is not a tool for beginners fighting with python packaging problems. It's a specialist tool to normalize your setup.
Very often, I see people tell me they don't have problems with pyenv, but later on have other unrelated problems with their dependencies. Analysis then prove it was because of pyenv, they just didn't know it. The cost is not obvious.
[+] [-] rcarmo|1 year ago|reply
The big advantage for me is that I can match whatever runtime and standard library a target has (and yes, that's needed more times than not, even in this new age of Docker).
Additionally, you can build an _optimized_ Python. I have this set for my builds:
[1]: https://taoofmac.com/space/blog/2015/10/03/1245[+] [-] b33j0r|1 year ago|reply
You will know when you care. These days, doing common tasks, the constraint solver in poetry will often tell you:
“Hey! I can’t find a version of sentencepiece with metadata that lets me combine this with python 3.12.2. Sorry! I give up!”
Now, if you aren’t concerned with using your bare metal for CUDA or fancy new MPS or AMD stuff. Just ignore this and use containers. I’d use podman compose.
However, I use pyenv on every machine. Because it compiles specific versions I actually need, even to create various virtual environments. If compiling python automatically sounds tough, you probably don’t need to anyway.
To describe the problem you’d see. I try to use poetry by default, though I think it became popular before it was PEP-compliant or useful in devops. It is impossible to control the behavior of other package managers, and poetry is/was strict about that. Which means you can’t force deploy in many cases. (Better lately.)
For the problem pyenv helps to solve, I back-up my pyproject.toml setuptools backend with pip and requirements.txt. These days, requirements-cuda.txt and requirements-mps.txt.
The landscape is still a disaster for binary compatibility, but it can be done lol. (I’ve been doing python packaging professionally since prom, which was python 2.6 give or take.)
[+] [-] mwexler|1 year ago|reply
That is, the pattern seems to be that someone mentions a solution, then a zillion responses as to why it sucks.
Is there any tool or pair that sucks least for most cases and beginners? I get that every case is different, but perhaps there are some useful starting points?
[+] [-] mikemcquaid|1 year ago|reply
[+] [-] bmitc|1 year ago|reply
What is the alternative? Every solution that I know of on Linux requires you to build Python on the machine: asdf, official Python downloads, etc.
[+] [-] cqqxo4zV46cp|1 year ago|reply
But, to add to the list of problems, these Python versions IIRC do not compile with optimisation turned on, so they’re by default quite a bit slower than they need to be.
[+] [-] whalesalad|1 year ago|reply
[+] [-] wodenokoto|1 year ago|reply
I’m sure he is trying to promote some sort of work flow in that article but I don’t understand which.
[+] [-] belter|1 year ago|reply
When the God Of Programming made Python, all other languages were jealous of its elegance, simplicity and intuitive beauty. When the other programming languages went to complain he said..."Wait until you see what package systems I will give them...They will never get an environment properly setup..." :-)
[+] [-] mrbonner|1 year ago|reply
[+] [-] d0mine|1 year ago|reply
I guess, I'm in the "expert" category. I'm saying it so that people won't be afraid to try it.
[+] [-] guappa|1 year ago|reply
[+] [-] Vaslo|1 year ago|reply
[+] [-] lb4r|1 year ago|reply
> https://www.bitecode.dev/p/why-not-tell-people-to-simply-use
Just curious: what are the downsides of poetry installed with pipx? The article mentions having to install poetry in another venv, but that's hardly an issue with pipx (you just add an 'x' after 'pip'), and installing pipx is as straight-forward as it can be.
[+] [-] Capricorn2481|1 year ago|reply
> It doesn’t support Windows. That’s game over right there, for half of the community.
What do you mean by this? I use pyenv on windows all the time.
[+] [-] hot_gril|1 year ago|reply
[+] [-] rewgs|1 year ago|reply
Also, the issues the article you link are things that damn near every programmer will eventually need to know. Things like PATH are IMO the basics -- if you don't understand this or how to `$packageManager install` something, you're gonna have a rough time programming in general, regardless of language.
[+] [-] nlitened|1 year ago|reply
Which ones did I miss? Which of them actually ensure your program always works the same as when you first wrote it, without asterisks?
[+] [-] anze3db|1 year ago|reply
Only pdm and poetry generate cross-platform lock files by default as far as I know, but there are a lot of people trying to solve this problem right now.
It's not an easy problem to solve. Python's package management predates package managers from most other programming languages and Python itself predates Linux. There is a lot of baggage so change is very slow.
[+] [-] bvrmn|1 year ago|reply
[+] [-] infecto|1 year ago|reply
Edit: Wow y'all are some sour people for voting down this question. I truly wonder how often people run into this problem compared to just complaining about it.
[+] [-] otabdeveloper4|1 year ago|reply
Nix does, if you want to actually invest the effort into the "without asterisks" part.
[+] [-] spacephysics|1 year ago|reply
It’s no panacea, but feels more stable and usable (especially from onboarding new team members PoV) than other tooling I’ve tried
[+] [-] _dain_|1 year ago|reply
[+] [-] fbuilesv|1 year ago|reply
[+] [-] crabbone|1 year ago|reply
There aren't such tools. Python not being a standard and heavily reliant on the OS that runs it and on third-party components that are also not standard leaves you with no choice by to "be at the wheel" all the time. Virtually anything written in Python will go stale in a mater of few years. In other words, you need to constantly update and test as the environment changes just to stand still.
[+] [-] diggan|1 year ago|reply
Using python and python libraries only from your package manager (like APT) for a specific OS version.
Micromamba as well, lightweight version of conda/miniconda.
[+] [-] 2wrist|1 year ago|reply
The other one is Docker, which some people hate. I don't mind it, in some cases I prefer it.
[+] [-] neon5077|1 year ago|reply
Needing more than one, MAYBE two versions of your language installed is insanity. If I build a Python widget and send it to my non-programmer coworker, there is exactly zero chance it will work until I walk over and manually set up the correct language. Instead I use a language that natively compiles to an exe.
Python is pretty neat, but the concept of a program that just works anywhere is so utterly alien that I genuinely cannot find any practical use for it.
I can't just install a pip package to use globally, I have to set up some goddamn virtual environment because everything is so brittle that nothing works without the exact correct language version.
It's like NPM all over again. Dependency hell is so bad that your package manager is now a dependency so we need a package manager manager to manage installing the correct package manager version to then install the correct packages.
Every single time I've come up to some Python thing or other, I spend about fifteen minutes fucking with it before giving up and using a tool built in a sane language.
The fact that I can't just send any random person a program and expect it to run at all is just lunacy. And no, compiled Python does not count because that's even more brittle tooling on top of all the other bullshit I have to deal with. If it doesn't work out of the box, it's a bad tool and I have far better things to do with my very valuable time.
[+] [-] jolux|1 year ago|reply
[+] [-] delduca|1 year ago|reply
[+] [-] jpalomaki|1 year ago|reply
If I'm keeping the projects under Dropbox, then I'll just add the .venv folder to the ignore list ("attr -s com.dropbox.ignored -V 1 .venv") to reduce the amount of files that need to be synced. If I need to get back to old project, I simply recreate the venv and install dependecies using Poetry.
A good habit would be to add a .pythonversion file on the project folder. Pyenv can pick that up and it is then obvious which version was used for the project.
I do development on Windows, but run all the Python stuff on WSL2 and use VS code for notebooks and everything. One thing I haven't looked at is using pipx to manage Poetry. Now I simply have one Poetry version installed.
Pyenv has worked fine once I managed to collect all the necessary dependencies to Ubuntu. For me it's not just to switch between Python 3.x. I like the fact that you can easily have different patch versions and match exactly the version I would use in production (via containers).
[+] [-] blowski|1 year ago|reply
[+] [-] cgopalan|1 year ago|reply
[+] [-] rsyring|1 year ago|reply
https://mise.jdx.dev/lang/python.html
It has other nice helpers for development environments (tasks, env variables, etc.).
[+] [-] camgunz|1 year ago|reply
- Download whatever Python binaries I need from python.org
- virtualenv --python=$PYTHON_VERSION ~/.virtualenvs/$PROJECT_NAME
- pyproject.toml
- pip-compile --generate-hashes
There has been tons of churn in the Python project management space, and I feel like I've been blissfully unaware of all of it with this workflow. Can't recommend enough.
Why regular Python binaries and not pyenv? I've had enough pyenv snafus and meltdowns that I started looking for alternatives, and it turned out I could just install whatever Python binary I wanted and specify it wherever I wanted. What could be easier?
python -m venv is fine, I've just gotten used to virtualenv.
pyproject.toml is the future and almost everything supports it pretty well; it also has fewer implicit weirdnesses than setup.py or setup.cfg.
pip-tools are really simple, and they let you have hashed dependencies which are super important IMO.
[+] [-] Capricorn2481|1 year ago|reply
[+] [-] kissgyorgy|1 year ago|reply
[+] [-] airstrike|1 year ago|reply
I don't recall ever needing anything other than virtualenvwrapper and pip—and even some of the annoyances these tools had early on have been solved by now...
https://virtualenvwrapper.readthedocs.io/en/latest/
If you really need different versions of python, you can just `mkvirtualenv -p python3 venvname`
I feel like every other tool out there has to explain what problem they solve that virtualenvwrapper doesn't
Don't install anything globally, creates lots of envs, and feel free to have different versions of python installed side-by-side with some "main" version preferably symlinked as `python` and `python3`
The end
[+] [-] jmathai|1 year ago|reply
It also ensures that my requirements.txt is sufficient for the code to run.
[+] [-] _dain_|1 year ago|reply
[+] [-] chasenjohnson|1 year ago|reply
[+] [-] Hackbraten|1 year ago|reply
The shortest ritual I could figure out so far is:
and I wonder what might be an easier incantation that still works.[+] [-] ktosobcy|1 year ago|reply
[+] [-] NikkiA|1 year ago|reply
[+] [-] TomSwirly|1 year ago|reply
Many developers, not just me, have a similar setup: we use virtual environments everywhere, and if you aren't in one, `python` doesn't even resolve to a symbol.
If I want to write a quick script with no dependencies, I directly call `python3.xx` on it. Otherwise, I create a virtualenv.
Yes, it's a bit harder for beginners, but from a huge amount of experience helping people who are starting up in programming, people have little issue in following a few more instructions. What demolishes beginners is getting into a bad state where nothing works and you don't know why.
[+] [-] RamiAwar|1 year ago|reply
Just install several pythons with different binary names.
ex. python3.8, python3.9, python3.11
No need to complicate things.
If using Poetry, you can just do `poetry env use python3.8`.
[+] [-] nxpnsv|1 year ago|reply
[+] [-] claytonjy|1 year ago|reply