This makes me so happy. Back when we had Jenkins slaves, one of our devops guys set a pipeline up that pip installed different versions over the top of system packages causing weird intermittent failures everywhere. Different pipelines would be running in different requirements files. I revoked sudo privs immediately for Jenkins (I didn't add them in the first place) and reprovisioned the whole build cluster resulting in pipelines breaking consistently where they should have been breaking: trying to do stupid stuff.
Personally I only ever use the system python packages on Linux if I can get away with it. Saves a whole world of problems.
Mixing pip with another package manager has always seemed weird to me. You're just asking for things to conflict and break.
I noticed with Homebrew that there was no way to untangle packages installed through pip and ones installed through Homebrew. After dealing with that mess once, I now make sure to use pip install --user. It can still cause things to break, but if that does happen it's at least easy to nuke the packages installed to my home directory.
Good. Now we just need to get pip itself updated so it refuses to run outside of a venv, and refuses to run unless invoked with "python -m pip" and we'll finally have something at least half decent.
And don't even get me started about how much better npm is at publishing packages, versus pip's refusal to add the same user friendliness.
PEP 704 is a recent proposal to require a virtual environment by default for any package installer - https://peps.python.org/pep-0704/. Again, you can opt-out if you want.
> There’s no shortage of package management alternatives available for Python [...]
> How someone is meant to pick between these as a new developer is a mystery.
This.
Every time I get booked to look at some Python project hours are usually wasted initially figuring out what dependency mgmt solution was used how.
And with what 'special sauce' the resp. developers deemed to be 'the right way' (or some library required because ... it just does)
As the author wrote: it seems common to omit the dependency setup in the Readme for Python projects.
I can understand why one would not mention this 'step' in a Rust or Node project but for Python it seems very much necessary.
I outright look for alternatives for something when the search comes with something written in python; the well-accepted strategy for deploying Python seems to be "abandon all hope of deploying it yourself in a way where updating is easy and hope docker container someone did that dealt with this mess will be enough.
I'm convinced that there are very few python libraries that Just Work if you follow their installation instructions. I've never found one that didn't come with issues myself.
Complain about this to a Python dev and you'll be "Well actually"ied to oblivion and each and every one will have their own opinion-as-fact on the best practice for managing these -- totally unaware how antithetical Python development has become from The Zen of Python.
Yeah, the well-actually's are a problem. It's not all of us though.
Python dev's know we have a problem, it's just hard to fix because "people developing apps and worrying about dependencies" is a rather small part of the python community. It's not like Java or something where everybody writing the language is a developer. Most are scientists or business people or students working in places like anaconda or Jupyter. So it's really hard to get momentum behind an all-together-now solution.
I've slowly been gravitating toward Nix flakes so I can use it to pin to a project versions of all of the things you can't reliably install with pip alone (like python itself, or numpy, or postgres or whatever) and then have it read deps from poetry (via poetry2nix) for everything that "just works," but that's never gonna fly with the non-developer Python community. Hell, it probably won't even fly with half of the developers either, but it works well for me.
I think my situation is typical of python developers, which is why we have this problem. I think it'll stick around for a while because it's not like "just use a different language" is gonna fly with the non-dev crowd. They're going to expect somebody else to solve these problems for them.
(I may have a bias because my company offers OSS python apps in a SaaS form factor, so our support folk are the ones solving these problems--typically by either handling the virtualenv behind the scenes or by ensuring that users with conflicting dependencies are using different images).
I wanted to run guarddog on source packages. Only then build them locally and install. Turns out, `pip download` triggers code execution in fetched packages.
Somewhat surprising and in this day and age worth spreading awareness of.
Put your package names in requirements.txt and run `make update-frozen`. To reinstall everything from frozen state, `make clean frozen`. (And replace the first space with a tab; HN is stripping my tabs out)
I know Pythonistas like to use Python for everything, but there are other tools out there that will make your life much simpler.
The article talks about installing Python packages for development, but if you find yourself using `pip` to install Python tools/scripts then you should use `pipx` - it will properly sandbox those tools so they don't break (or be broken by) the system or other Pythons:
If anyone's interested in a pipx clone with excellent tab completion, I would appreciate any feedback on pipz, a function of my zsh plugin for python environment and dependency management: zpy
While I agree with the author to not do global pip installs for every new project, I also don’t want to see text in every git repo README explaining Python package managers.
The lack of one true package management approach is a failure of the language. OP is advocating for a saner default like npm, instead of the current venv + pip mess.
As a non-python person who has hair-pulling with python pip / pip3 / python2 / python3 python-is-python2-or-python3, this was a relevation.
pipenv looks like what pip should have been.
Another story on HN is "what happened to Ruby" and that really crystallized what I don't like about python. I'm not a ruby programmer, but I have to admit how much fantastic software came out of Ruby.
Ruby was always fighting Java for some reason, it should have been fighting Python. If only Ruby had won THAT war.
This is where npm gets it right. It's so much simpler to have the default install in a local folder, and then have an option to install globally if you like.
I appreciate the concern for new developers, but I really don't think it's a good solution to have every project readme describe pip, poetry, pipenv, and whatever other new hotness there is in the package management world. There's a reason that all the readmes describe pip installation: it's the lowest common denominator, present with every standard python install, and along with virtualenv (also standard) it can do most of the requirements for package management.
I think to help new developers, we could encourage documentation to briefly point to the official PyPA documents on the variety of options available. It would be better to focus on making that more accessible, rather trying to throw the burden onto package maintainers to describe using their package with every new tool.
Omg this is so true! I installed a package globally, but then my interpreter was using another version of python, which doesn't have the installed package. It took me an hour to find out about this. What a waste of time.
It's also interesting how things like AWS Lambdas, Graviton, etc, are exposing all the shortcomings of the various pip install, venv, poetry, etc, approaches.
It's not impossible to figure it out, but you end up spending a lot of time to come up with something that works locally, within containers, inside a CI/CD system, and then deployed out across things like Lambdas, or non x64 machines.
Then, after it's all working, upgrading the Python version, or an extension that has C code, etc, repeats some of the hard bits.
At least with Lambda it really is easy, just use Serverless Application Model and when you do “sam build” choose “--use-container”. It will look at your CloudFormation template where you are referring to the local directory containing your source code and requirements.txt and download and build in the appropriate Docker container for your language, version and architecture.
When I switched to Arch Linux, I learned that pip has a --user option to install Python packages in the home dir of the current user. This is essential to not interfere with the system install from the system package manager. I had really trouble with that in former times.
Furthermore, as I now be used to bleeding edge packages, I update at least once a week all the outdated Python packages of my >450 installed ones.
When some packages get downgraded because of requirements, I ask:
Do I need the package that caused the downgrade more often or with more of the packages in the main environment, or is this true for one or some of the downgraded packages?
According to the answer, I put the 'problematic' package(s) in a new or existing venv, and update the downgraded ones in the main environment, if necessary.
This work cannot be done by a package manager!
Costs me <10 minutes every week to keep the main environment up to date, a bit more if I want that for some or all venvs.
If I didn't have A installed and then I install B which transitively installs A, then I expect that uninstalling B will also uninstall A. If only one system is managing the packages, then it is able to do this. It will have a record of the things I've explicitly installed so it knows what dependencies are safe to uninstall.
apt handles this by marking packages as manually installed. You and the author could both be happy with that solution but afaik pip doesn't currently store such information.
This post points out one of my struggles with python.
I am not a python developer but I use python heavily for some tooling. So all I need to do is to “distribute” my tools to other servers in a replicable and consistent matter, isolated from global packages.
Can you please help me understand two points?
1. If I use venv+pip to install some python app, do I have to “activate” that specific virtual environment before executing that tool or can I just simply call it by its path on the file system?
2. Are there any official guide rails for making venv-wrapped app accessible to other users on a server? Or just as simple as placing links to /usr/local/bin/ for example?
1: usually you can just run the binary by its path. tbh I don't fully understand why it doesn't always work, but it's fairly rare, and most of the ones I can kinda-remember may have been during install time.
2: due to 1, symlinks often work. It's how I've installed all of my custom python binaries. Otherwise you'll very frequently see python "binaries" installed by e.g. Homebrew that are actually ~5 lines of minor environment prep and then running the actual binary - that's the only reliable way afaik.
> Lets say you use the same package again, but theres been a new release with some additional features. When you upgrade your global Python to use it, you now need to ensure every project you’ve done now works with it. That’s annoying and unlikely to happen, what you’ll be left with is a broken build.
Wait, what?
Don't python packages generally use `semver` versioning, and ensure that upgrades in the same major version are backwards-compatible?
And that different major versions are co-installable?
I saw this Twitter thread the other day (https://twitter.com/fchollet/status/1617704787235176449?s=46...) about similar problems, and some comments suggest using Docker. I couldn’t find any guides or ways to do this for a Python project; anyone here know more or has done this before?
poetry breaks once a while for me, so I am not using it these days.
pipenv used to be my first choice but it became inactive, seems it is actively under development again?
a few weeks ago there is a recommendation for PDM but I have not really used it.
For now I am using the pip+venv approach.
By the way, you better do: `python -m pip install` instead of `pip install`, don't remember why anymore but I did read somewhere that explained the difference and I agreed on then to prefer 'python -m pip install'
I use Python for research. If I need some package, I simply want the latest version; pip install is usually fine.
If something depends explicitly on the fixed (old) version, that's when problems happen and I grudgingly remember how to use pyenv. But I like to use the most recent versions and most recent Python, and I like packages that share this bleeding edge approach.
Article conflates global installation into the system python with global installation in general. Not everything is a project dependency. If you want, say, ipython, available everywhere, global installation is appropriate. You can get this without clobbering my system python by simply not using the system python for my projects.
I've been a happy user of pipenv for several years (at work, in production) and still recommend it. You lock the versions you want independently of the requirements.txt so you can update just the packages you want without worrying about sub-dependencies. 10/10 recommend.
You probably want conda if you're in this situation, as it basically solves for these issues (but doesn't have great docs for actually adding packages to it, unfortunately).
npm certainly has a number of problems (at the end the article compares pip to npm) -- but after reading this article I didn't realize pip was so problematic. I also didn't realize it installed things globally.
> I also didn't realize it installed things globally
It doesn't. It's a subtle distinction but the 'blame' doesn't lie with pip. When you do a pip install it does it in the context of the python interpreter you're using.
If you use your global python you get an installation in a global context from pip. If you use a non-global python you get a non-global installation from pip. And this is what venv etc give you; a local interpreter, which means the associated pip installs in a local context (a separate one for each venv).
I don't understand why pip doesn't do it like npm. Admittedly, I don't write Python code much, but "npm install xyz@1.2.3" simply installs to a node_modules/ folder in the current directory. Very easy to parse and nuke if I need to. I don't really understand how venv and its weird shell prompt are a better solution.
Is there a canonical example of how python projects should manage dependencies and sandboxing such that other developers can just clone, install, and get to work?
Put everything in a docker container/OCI image and have someone own managing and babysitting the build of that image for everyone else.
There really is no single tool or workflow for everything in the python world. What works for a simple source only python package can break horribly if you try using sophisticated scientific computing packages with numerous native dependencies (and then you realize you need conda or a whole other set of tools).
I'm not sure if I get the point of this article. So basically the author has learnt that there are a different ways of managing packages in Python? I'm aware that this might be a problem in Python, but let's be serious guys, you only need to spend 5 mins to learn about venv/conda and you will never face any problem in a basic Python project. You don't have to write an article about that.
lazka|3 years ago
You can still force it via `pip install --break-system-packages ...` if needed.
unxdfa|3 years ago
Personally I only ever use the system python packages on Linux if I can get away with it. Saves a whole world of problems.
wheelerof4te|3 years ago
On Linux, you either use the system packages via "apt install", or you use venvs.
EDIT: For context, I've meant "managed" distros like Debian and Ubuntu.
Groxx|3 years ago
For anyone on other systems who wants this kind of protection right now, pip has had this available for a few years at least:
I absolutely recommend doing it. Immediately.ziml77|3 years ago
I noticed with Homebrew that there was no way to untangle packages installed through pip and ones installed through Homebrew. After dealing with that mess once, I now make sure to use pip install --user. It can still cause things to break, but if that does happen it's at least easy to nuke the packages installed to my home directory.
TheRealPomax|3 years ago
And don't even get me started about how much better npm is at publishing packages, versus pip's refusal to add the same user friendliness.
ianwootten|3 years ago
meitham|3 years ago
unknown|3 years ago
[deleted]
virtualritz|3 years ago
> How someone is meant to pick between these as a new developer is a mystery.
This.
Every time I get booked to look at some Python project hours are usually wasted initially figuring out what dependency mgmt solution was used how. And with what 'special sauce' the resp. developers deemed to be 'the right way' (or some library required because ... it just does)
As the author wrote: it seems common to omit the dependency setup in the Readme for Python projects.
I can understand why one would not mention this 'step' in a Rust or Node project but for Python it seems very much necessary.
ilyt|3 years ago
ploxiln|3 years ago
Spivak|3 years ago
The game of, “blank project immediately needs 12 different tools I read about on a blog post” is silly.
wheelerof4te|3 years ago
Just create a virtual environment and install your packages there.
Done.
davidmurdoch|3 years ago
Complain about this to a Python dev and you'll be "Well actually"ied to oblivion and each and every one will have their own opinion-as-fact on the best practice for managing these -- totally unaware how antithetical Python development has become from The Zen of Python.
__MatrixMan__|3 years ago
Python dev's know we have a problem, it's just hard to fix because "people developing apps and worrying about dependencies" is a rather small part of the python community. It's not like Java or something where everybody writing the language is a developer. Most are scientists or business people or students working in places like anaconda or Jupyter. So it's really hard to get momentum behind an all-together-now solution.
I've slowly been gravitating toward Nix flakes so I can use it to pin to a project versions of all of the things you can't reliably install with pip alone (like python itself, or numpy, or postgres or whatever) and then have it read deps from poetry (via poetry2nix) for everything that "just works," but that's never gonna fly with the non-developer Python community. Hell, it probably won't even fly with half of the developers either, but it works well for me.
I think my situation is typical of python developers, which is why we have this problem. I think it'll stick around for a while because it's not like "just use a different language" is gonna fly with the non-dev crowd. They're going to expect somebody else to solve these problems for them.
(I may have a bias because my company offers OSS python apps in a SaaS form factor, so our support folk are the ones solving these problems--typically by either handling the virtualenv behind the scenes or by ensuring that users with conflicting dependencies are using different images).
aflukasz|3 years ago
I wanted to run guarddog on source packages. Only then build them locally and install. Turns out, `pip download` triggers code execution in fetched packages.
Somewhat surprising and in this day and age worth spreading awareness of.
throwaway892238|3 years ago
I know Pythonistas like to use Python for everything, but there are other tools out there that will make your life much simpler.
jasonpeacock|3 years ago
https://pypa.github.io/pipx/
KptMarchewa|3 years ago
AndyKluger|3 years ago
https://github.com/andydecleyre/zpy
mark_l_watson|3 years ago
throwboatyface|3 years ago
AtlasBarfed|3 years ago
pipenv looks like what pip should have been.
Another story on HN is "what happened to Ruby" and that really crystallized what I don't like about python. I'm not a ruby programmer, but I have to admit how much fantastic software came out of Ruby.
Ruby was always fighting Java for some reason, it should have been fighting Python. If only Ruby had won THAT war.
robertlagrant|3 years ago
brenns10|3 years ago
I think to help new developers, we could encourage documentation to briefly point to the official PyPA documents on the variety of options available. It would be better to focus on making that more accessible, rather trying to throw the burden onto package maintainers to describe using their package with every new tool.
https://packaging.python.org/en/latest/key_projects/
alanng|3 years ago
savingsPossible|3 years ago
then the 'pip' is running the same version as the 'python' command (I believe, can you check and comment latter?)
(you'd still have to check your IDE if you are not running python from the CLI)
savrajsingh|3 years ago
sammy2255|3 years ago
lrobinovitch|3 years ago
jackhoy|3 years ago
Link for anyone not familiar with pyenv/virtual env usage: https://www.jackhoy.com/web-applications/2017/02/12/setting-...
tyingq|3 years ago
It's not impossible to figure it out, but you end up spending a lot of time to come up with something that works locally, within containers, inside a CI/CD system, and then deployed out across things like Lambdas, or non x64 machines.
Then, after it's all working, upgrading the Python version, or an extension that has C code, etc, repeats some of the hard bits.
scarface74|3 years ago
It works great when you have native dependencies.
rgavuliak|3 years ago
mharig|3 years ago
Furthermore, as I now be used to bleeding edge packages, I update at least once a week all the outdated Python packages of my >450 installed ones. When some packages get downgraded because of requirements, I ask: Do I need the package that caused the downgrade more often or with more of the packages in the main environment, or is this true for one or some of the downgraded packages?
According to the answer, I put the 'problematic' package(s) in a new or existing venv, and update the downgraded ones in the main environment, if necessary.
This work cannot be done by a package manager!
Costs me <10 minutes every week to keep the main environment up to date, a bit more if I want that for some or all venvs.
kgwgk|3 years ago
Why would i expect that? If one day I install A and another day I install B, which depends on A, I wouldn’t expect to lose A of I were to uninstall B.
ziml77|3 years ago
ZGDUwpqEWpUZ|3 years ago
404mm|3 years ago
I am not a python developer but I use python heavily for some tooling. So all I need to do is to “distribute” my tools to other servers in a replicable and consistent matter, isolated from global packages.
Can you please help me understand two points?
1. If I use venv+pip to install some python app, do I have to “activate” that specific virtual environment before executing that tool or can I just simply call it by its path on the file system?
2. Are there any official guide rails for making venv-wrapped app accessible to other users on a server? Or just as simple as placing links to /usr/local/bin/ for example?
Groxx|3 years ago
2: due to 1, symlinks often work. It's how I've installed all of my custom python binaries. Otherwise you'll very frequently see python "binaries" installed by e.g. Homebrew that are actually ~5 lines of minor environment prep and then running the actual binary - that's the only reliable way afaik.
Bonus answer to 2: pipx looks pretty decent.
okasaki|3 years ago
aflukasz|3 years ago
Hopefully, in 2024, we will be able to say same thing about signing via sigstore ecosystem.
Karellen|3 years ago
Wait, what?
Don't python packages generally use `semver` versioning, and ensure that upgrades in the same major version are backwards-compatible?
And that different major versions are co-installable?
alexchantavy|3 years ago
rgavuliak|3 years ago
synergy20|3 years ago
pipenv used to be my first choice but it became inactive, seems it is actively under development again?
a few weeks ago there is a recommendation for PDM but I have not really used it.
For now I am using the pip+venv approach.
By the way, you better do: `python -m pip install` instead of `pip install`, don't remember why anymore but I did read somewhere that explained the difference and I agreed on then to prefer 'python -m pip install'
savingsPossible|3 years ago
If there are 2 installed, then "python" can refer to (say) python 3.10 and pip to python 3.9
using python -m makes you pip with 3.10
atemerev|3 years ago
If something depends explicitly on the fixed (old) version, that's when problems happen and I grudgingly remember how to use pyenv. But I like to use the most recent versions and most recent Python, and I like packages that share this bleeding edge approach.
dissent|3 years ago
yboris|3 years ago
switch007|3 years ago
disgruntledphd2|3 years ago
hungryforcodes|3 years ago
So the solution is?
kcartlidge|3 years ago
It doesn't. It's a subtle distinction but the 'blame' doesn't lie with pip. When you do a pip install it does it in the context of the python interpreter you're using.
If you use your global python you get an installation in a global context from pip. If you use a non-global python you get a non-global installation from pip. And this is what venv etc give you; a local interpreter, which means the associated pip installs in a local context (a separate one for each venv).
wildrhythms|3 years ago
zokier|3 years ago
Always use venv.
bentaber|3 years ago
qbasic_forever|3 years ago
There really is no single tool or workflow for everything in the python world. What works for a simple source only python package can break horribly if you try using sophisticated scientific computing packages with numerous native dependencies (and then you realize you need conda or a whole other set of tools).
bayesian_horse|3 years ago
bjd2385|3 years ago
realitysballs|3 years ago
sam_goody|3 years ago
nntwozz|3 years ago
kkthxbb|3 years ago
vlunkr|3 years ago