This is great, but sometimes I think that python needs a new package manager from scratch instead of more tools trying to mix and mash a bunch of flawed tools together in a way that's palatable by most of us. Python packaging sucks, the whole lot of it. Maybe I'm just spoiled by rust and elixir, but setuptools, distutils, pip, ez_install, all of it is really subpar. But of course everything uses pypi and pip now, so it's not like any of it can actually be replaced. The state of package management in python makes me sad. I wish there was a good solution, but I just don't see it.
Edit: I don't mean to disparage projects like this and pipfile. Both are great efforts to bring the packaging interface in line with what's available in other languages, and might be the only way up and out of the current state of affairs.
I think python packaging has gotten LOTS better in the last few years. I find it quite pleasurable to use these days.
From binary wheels (including on different linux architectures), to things like local caching of packages (taking LOTS of load off the main servers). To the organisation github of pypa [0], to `python -m venv` working.
Also lots of work around standardising things in peps, and writing documentation for people.
I would like to applaud all the hard work people have done over the years on python packaging. It really is quite nice these days, and I look forward to all the improvements coming up (like pipenv!).
I'd suggest people checkout fades [1] (for running scripts and automatically downloading dependencies in a venv), as well as conda [2] the alternative package manager.
Okay - maybe I'm missing something, but pip is the only Python package manager I've ever used. And it's basically "pip install xxx", "pip install --upgrade xxx", "pip show xxx", "pip list", "pip uninstall xxx"
I'm curious what I've been missing about pip that makes it problematic - I've never used the other tools you mentioned (setuptools/distutils/ez_install) - so I can't comment on them, but, on the flip side, I've never had to use them, so maybe my requirements are somewhat more simple than yours.
I strongly agree but this can work now and is a big improvement over what we currently have. So while I would literally pay to see somebody work on a better package manager (which can generate exe, deb and use a conf file indead of .py), this is a good filler.
All you've said here is "python packaging sucks", with no explanation why, and with no alternative. Not a substantial comment, and I'm disappointed that it's been voted to the top.
I'd ask for your reasoning but it seems sametmax has done a good job of that for you:
"This is great, but sometimes I think that python needs a new package manager from scratch"
Ha, that's what I came here to say!
Or better - a new packaging paradigm.
Maybe it's extremely uncool to say ... but I think Java still has the best packaging paradigm of all languages. Jars rule. Of course 'gradle' is kind of a confusing mess so they don't have dependencies worked out very well ...
Nevertheless I do feel that Python's packaging and dependency/versioning woes create a much bigger systematic problem than many realize.
I gave up populating requirements in setup.py. I just use multiple requirements.txt. This article has been debated for years already and there is absolutely no right / wrong.
Hey everyone. I'm Kale, currently the lead developer on the conda project. It's been mentioned a few times in this thread, and I just want to make sure that any questions about it are answered accurately. Feel free to ask me anything about product conda. Thanks!
Often people have a requirements.live.txt, or other packages depending on the environment. Is that handled somehow? Can we use different files or sections? [ED: yes, different sections]
Still wondering to myself if this is worth the fragmentation for most people using requirements.txt ? Perhaps the different sections could have a "-r requirements.txt" in there, like how requirements.dev.txt can have "-r requirements.txt". [ED: the pipfile idea seems to have quite some people behind it, and pip will support it eventually. Seems it will be worth it to standardise these things. requirements.txt is a less jargony name compared to Pipfile though, and has a windows/gui friendly extension.]
Other tools can set up an environment, download stuff, and run the script. Will pipenv --shell somescript.py do what I want? (run the script with the requirements it needs). ((I guess I could just try it.)) [ED: doesn't seem so]
Why Pipfile with Caps? Seems sort of odd for a modern python Thing. It looks like a .ini file? [ED: standard still in development it seems. TOML syntax.]
With a setup.py set up, all you need to do is `pip install -e .` to download all the required packages. Or `pip install somepackage`. Lots of people make the setup.py file read the requirements.txt. Do you have some command for handling this integration? Or is this needed to be done manually? [ED: seems no considering about this/out of scope.]
It's a very similar set of tools. I use pip-compile which allows me to put all of my dependencies into a `requirements.in` file, and then "compile" them to a `requirements.txt` file as a lockfile (so that it is compatible with pip as currently exists).
This looks great, though, I'm excited to check it out!
We use pip-tools on all our Python projects and it works great. I believe the requirements.in compiled to requirements.txt approach is much more sane and less error-prone.
I think an app should not expose end users to its dependencies. That leaves the end user with a lot of pain figuring out versions of dependencies and god forbid you need to compile some dep then you need a build environment and its dependencies any of which can fail in this chain leaving a very unpleasant and even hostile end user experience.
Ruby and Node apps are particularly guilty of this pulling in sometimes hundreds of packages some of which need compilation. Compare that to a Go binary which is download and use. These things can get very complicated very fast even for developers or system folks let alone end users who may not be intimately familiar with that specific ecosystem.
Very similar. I think Pipenv improves on Bundler by leveraging Virtualenv and Ruby doesn't have a per project equivalent to Virtualenv that I'm aware of. You can set the path config variable of Bundler to not place the project Gems in a central location which I think is cleaner and try to remember to always do now.
It would be _super_ interesting if the Python and Ruby communities got together to harmonize every last detail of their packaging toolchain. Who is in?
Seems to be a python specific nix-shell like tool?
With nix[OS] you just run `nix-shell -p python[2,3] python[2,3]Pacakges.numpy ...` to get an environment with the required packages.
Of course this requires that the python library is packaged in nix, but in my experience the coverage is quite good, and it's not very hard to write packages once you get the hang of it.
It also possible (but currently a bit clumsy in some ways) to set up named and persistent environments.
I will definitely be trying this out. Python version and package management is a dumpster fire that wastes gobs of my time on the regular. I'll try anything that promises to end the pain.
Finally someones does it!
I was using:
pip -t .pip
in my code, avoiding virtual-env completely, but that was not enough and incomplete.
As this is not cross platform and it would be nice to switch between Linux/Windows while coding to maintain platform compatibility, can the virtualenv envs be created with a os platform & subsystem prefix ?
for example, having multiple envs at once:
I always wonder if this could be done once and for all languages, instead of Ruby making bundler, Haskell Cabal sandboxes or stack, Perl Brew, etc. Is this where Nix is going?
I don't know, different languages and environments need different things. I'm not sure that "package management" is as generic as it seems at first glance.
You make it one tool, and sysadmins will instantaneously lock it down. These package managers, in most cases, are developer tools built to get around system-wide locks on libraries; the more you centralize them, the more likely it is they will get locked down, and then someone will build tools to get around that, and so on and so forth.
You can easily get a nice isolated python environment with some packages in nix without using pip, pyenv, etc. `nix-shell -p python pythonPackages.numpy ...`
So far I think it works quite well for most languages as long the needed packages are in nixpkgs.
Some of the tooling could be better, but the underlying model seems sound.
I'm not really convinced language-specific package managers are needed. Nix isn't perfect yet, but it has come a long way.
normally I use virtualenvwrapper and that makes a virtualenv directory for all virtualenvs you create with it. before that, I always create my projects' venvs inside my project hierarchy.
I had a dilemma about it. But after all, you can not move your venv directory unless you use `--relocatable` option. So, anyone have a strong argument about creating venvs inside your project directory?
I've always found the whole virtualenv stuff so superfluous. Do we really need all the machinery with shell scripts and static paths?
We just use a directory where we keep our dependencies. It's a matter of:
mkdir libs
pip install -t libs <package>
# then to run
PYTHONPATH=libs python app.py
From what I can tell, this accomplishes everything a venv does (except bringing the Python interpreter itself along) without requiring any extra tools or conventions to learn.
I just keep venvs in the project folder and add them to .gitignore. --relocatable never worked well for me, I just keep my requirements.txt up to date so it's trivial to blow away a venv and recreate it if necessary (python3 -m venv env && source env/bin/activate && pip install -r requirements.txt).
I find tools like virtualenvwrapper and this one from Kenneth tend to solve issues I don't really have. A little bit of repetitive typing here and there is ok to burn knowledge into my mind; and less leaky abstractions I have to deal with, the better.
I was really not a fan of the last "made in Reitz" project Maya. But this, I really can get along.
The whole things make it way easier to get started for a beginner. Now more activate. No more wondering about virtualenv. Automatic lock files are great since no project I know of use them since they are not well understood.
It's like node_packages (easy and obvious), but cleaner (no implicit magic).
LinkedIn has a similar open source project that is much more mature. It builds on Gradle features to manage complex dependencies and build Python artifacts. If you include this LinkedIn Gradle Plugin [1] you can automatically run tests in a virtual env and source a file to enter the project's virtual env.
PyGradle [2]: "The PyGradle build system is a set of Gradle plugins that can be used to build Python artifacts"
> --three / --two Use Python 3/2 when creating virtualenv.
I use Python 2.7, 3.4, and 3.5 on various projects. Is there a way to choose between 3.4 and 3.5 using Pipenv? I'm using something like this with virtualenv:
First, you need to take care of the virtualenv manually. This tool avoid that. You can completly ignore the virtualenv. You don't even have to populate requirements or learn about pip freeze.
Secondly, this tool allow you to freeze your requirement list at specific versions. So in your req file, you have the name of the packages you depend on. Bon on the req lock, you get all the pulled dependencies recursively with the version you are using right now. The first one let you dev more easily. The second one deploy more easily.
[+] [-] cderwin|9 years ago|reply
Edit: I don't mean to disparage projects like this and pipfile. Both are great efforts to bring the packaging interface in line with what's available in other languages, and might be the only way up and out of the current state of affairs.
[+] [-] renesd|9 years ago|reply
From binary wheels (including on different linux architectures), to things like local caching of packages (taking LOTS of load off the main servers). To the organisation github of pypa [0], to `python -m venv` working.
Also lots of work around standardising things in peps, and writing documentation for people.
I would like to applaud all the hard work people have done over the years on python packaging. It really is quite nice these days, and I look forward to all the improvements coming up (like pipenv!).
I'd suggest people checkout fades [1] (for running scripts and automatically downloading dependencies in a venv), as well as conda [2] the alternative package manager.
[0] https://github.com/pypa/
[1] https://fades.readthedocs.io/en/release-5/readme.html#what-d...
[2] http://conda.pydata.org/docs/intro.html
[+] [-] ghshephard|9 years ago|reply
I'm curious what I've been missing about pip that makes it problematic - I've never used the other tools you mentioned (setuptools/distutils/ez_install) - so I can't comment on them, but, on the flip side, I've never had to use them, so maybe my requirements are somewhat more simple than yours.
[+] [-] sametmax|9 years ago|reply
[+] [-] ploggingdev|9 years ago|reply
[+] [-] barbs|9 years ago|reply
I'd ask for your reasoning but it seems sametmax has done a good job of that for you:
https://news.ycombinator.com/item?id=13460490
[+] [-] unknown|9 years ago|reply
[deleted]
[+] [-] ellisv|9 years ago|reply
[+] [-] edblarney|9 years ago|reply
Ha, that's what I came here to say!
Or better - a new packaging paradigm.
Maybe it's extremely uncool to say ... but I think Java still has the best packaging paradigm of all languages. Jars rule. Of course 'gradle' is kind of a confusing mess so they don't have dependencies worked out very well ...
Nevertheless I do feel that Python's packaging and dependency/versioning woes create a much bigger systematic problem than many realize.
Kudos to the author though ...
[+] [-] shakna|9 years ago|reply
> It harnesses Pipfile, pip, and virtualenv into one single toolchain. It features very pretty terminal colors.
For a weekend project, this has some very nice things.
Which removes the need for me to run my own project that basically does these things... In more or less, a worse way.
Everything I've come to expect from Reitz, and hopefully it'll gain some decent ground like other projects of the same author.
[+] [-] therealmarv|9 years ago|reply
[+] [-] yeukhon|9 years ago|reply
[+] [-] kalefranz|9 years ago|reply
[+] [-] renesd|9 years ago|reply
Often people have a requirements.live.txt, or other packages depending on the environment. Is that handled somehow? Can we use different files or sections? [ED: yes, different sections]
Still wondering to myself if this is worth the fragmentation for most people using requirements.txt ? Perhaps the different sections could have a "-r requirements.txt" in there, like how requirements.dev.txt can have "-r requirements.txt". [ED: the pipfile idea seems to have quite some people behind it, and pip will support it eventually. Seems it will be worth it to standardise these things. requirements.txt is a less jargony name compared to Pipfile though, and has a windows/gui friendly extension.]
Other tools can set up an environment, download stuff, and run the script. Will pipenv --shell somescript.py do what I want? (run the script with the requirements it needs). ((I guess I could just try it.)) [ED: doesn't seem so]
Why Pipfile with Caps? Seems sort of odd for a modern python Thing. It looks like a .ini file? [ED: standard still in development it seems. TOML syntax.]
With a setup.py set up, all you need to do is `pip install -e .` to download all the required packages. Or `pip install somepackage`. Lots of people make the setup.py file read the requirements.txt. Do you have some command for handling this integration? Or is this needed to be done manually? [ED: seems no considering about this/out of scope.]
Is there a pep? [ED: too early it seems.]
[+] [-] conradev|9 years ago|reply
It's a very similar set of tools. I use pip-compile which allows me to put all of my dependencies into a `requirements.in` file, and then "compile" them to a `requirements.txt` file as a lockfile (so that it is compatible with pip as currently exists).
This looks great, though, I'm excited to check it out!
[+] [-] FabioFleitas|9 years ago|reply
[+] [-] throw2016|9 years ago|reply
Ruby and Node apps are particularly guilty of this pulling in sometimes hundreds of packages some of which need compilation. Compare that to a Go binary which is download and use. These things can get very complicated very fast even for developers or system folks let alone end users who may not be intimately familiar with that specific ecosystem.
[+] [-] choxi|9 years ago|reply
[+] [-] igravious|9 years ago|reply
It would be _super_ interesting if the Python and Ruby communities got together to harmonize every last detail of their packaging toolchain. Who is in?
[+] [-] uranusjr|9 years ago|reply
[+] [-] gourneau|9 years ago|reply
[+] [-] olejorgenb|9 years ago|reply
With nix[OS] you just run `nix-shell -p python[2,3] python[2,3]Pacakges.numpy ...` to get an environment with the required packages.
Of course this requires that the python library is packaged in nix, but in my experience the coverage is quite good, and it's not very hard to write packages once you get the hang of it.
It also possible (but currently a bit clumsy in some ways) to set up named and persistent environments.
[+] [-] command_tab|9 years ago|reply
I'm glad to see Python getting the same attention as other modern package managers. This is all great work!
[+] [-] caconym_|9 years ago|reply
[+] [-] istoica|9 years ago|reply
As this is not cross platform and it would be nice to switch between Linux/Windows while coding to maintain platform compatibility, can the virtualenv envs be created with a os platform & subsystem prefix ? for example, having multiple envs at once:
[+] [-] zoul|9 years ago|reply
[+] [-] HeyImAlex|9 years ago|reply
[+] [-] toyg|9 years ago|reply
[+] [-] olejorgenb|9 years ago|reply
You can easily get a nice isolated python environment with some packages in nix without using pip, pyenv, etc. `nix-shell -p python pythonPackages.numpy ...`
So far I think it works quite well for most languages as long the needed packages are in nixpkgs.
Some of the tooling could be better, but the underlying model seems sound.
I'm not really convinced language-specific package managers are needed. Nix isn't perfect yet, but it has come a long way.
[+] [-] jaybuff|9 years ago|reply
[+] [-] nejdetckenobi|9 years ago|reply
I had a dilemma about it. But after all, you can not move your venv directory unless you use `--relocatable` option. So, anyone have a strong argument about creating venvs inside your project directory?
[+] [-] icebraining|9 years ago|reply
We just use a directory where we keep our dependencies. It's a matter of:
From what I can tell, this accomplishes everything a venv does (except bringing the Python interpreter itself along) without requiring any extra tools or conventions to learn.[+] [-] toyg|9 years ago|reply
I find tools like virtualenvwrapper and this one from Kenneth tend to solve issues I don't really have. A little bit of repetitive typing here and there is ok to burn knowledge into my mind; and less leaky abstractions I have to deal with, the better.
[+] [-] sametmax|9 years ago|reply
The whole things make it way easier to get started for a beginner. Now more activate. No more wondering about virtualenv. Automatic lock files are great since no project I know of use them since they are not well understood.
It's like node_packages (easy and obvious), but cleaner (no implicit magic).
Like.
[+] [-] georgeaf99|9 years ago|reply
PyGradle [2]: "The PyGradle build system is a set of Gradle plugins that can be used to build Python artifacts"
[1] https://github.com/linkedin/pygradle/blob/01d079e2b53bf9933a...
[2] https://github.com/linkedin/pygradle
[+] [-] helb|9 years ago|reply
I use Python 2.7, 3.4, and 3.5 on various projects. Is there a way to choose between 3.4 and 3.5 using Pipenv? I'm using something like this with virtualenv:
[+] [-] iheredia|9 years ago|reply
[+] [-] jjawssd|9 years ago|reply
[+] [-] dvl|9 years ago|reply
[+] [-] sametmax|9 years ago|reply
Secondly, this tool allow you to freeze your requirement list at specific versions. So in your req file, you have the name of the packages you depend on. Bon on the req lock, you get all the pulled dependencies recursively with the version you are using right now. The first one let you dev more easily. The second one deploy more easily.
All that, for less complexity. Win win.