Python Packaging, One Year Later: A Look Back at 2023 in Python Packaging

[infecto]

I always struggle with these discussions. I have been using Python a long time and while there ahve definitely been a handful of gotchas over the years, it was never an experience that held me up long enought to think twice about. I am sure there is a better way but what holds me up is how often it impacts other people but not myself. Perhaps my usage is not advanced enough but it always leaves me wondering.

Edit: Just as an additional thought, is one of the main issues when distributing to client machines to what is essentially python executables and the clients could be using different OS? My mind jumps to probably not using Python at that point unless there was a specific dependency/library to using it.

[jonawesomegreen]

I've encountered frustration with Python packaging in two main areas:

1) Installing applications within Docker containers. While wheels have improved this situation, I was surprised that building a package for easy copying into a container and running without the need for installing build tools and extensions in the final container image was not straightforward, especially coming from other languages.

2) Distributing Python utilities to end users across various platforms in an easily installable manner without requiring them to follow lengthy instructions to set up all the dependencies has been another challenge.

We use Poetry and have largely "solved" #1 with a somewhat complicated Docker build. It works well now, so no one has to think about it much. That made deploying Python server-side fairly easy. However, #2 has been much more of a challenge, and I wonder if that is where other folks in this thread are feeling the most pain.

[bbojan]

Same here. I always see people complaining about Python packaging, but rarely run into issues myself. Maybe it's an OS issue? I use Linux.

Some of the projects at work use gigabytes of computer vision dependencies and again I don't remember the last time I had any issues.

Could some of the people who do have issues give an example or two of the problems they run into?

[volker48]

The only time I've ever experienced a memorable amount of pain was when I was working on a project that used one of the newer options like Poetry. Using a virtualenv, a requirements.txt file, and pip I have not had serious issues in 15 years of using Python.

[coldtea]

>it was never an experience that held me up long enought to think twice about

The pain is when you're doing more complex stuff than creating a local virtualenv with some basic deps though.

[nerdponx]

I'm less pessimistic about packaging than most. No language that I know of has ever attempted a standardization effort like this. I think it will pay off in the long term. But it's taken years to get here and it will take more years to wrap up the project.

> An attempt at a specification was rejected due to “lukewarm reception”, even though there exist at least four implementations which are achieving roughly the same goals, and other ecosystems also went through this before.

Python has been weird lately.

We got structural pattern matching, which is entirely new syntax that is essentially just syntactic sugar and only kind-of solves a problem that relatively few users ever had.

But then we reject __pypackages__ and null-coalescing/safe-navigation operators, which solve problems that everyone has and are unambiguous improvements and modernizations of the language. Even if the PEPs had problems and needed to be rewritten from scratch, there is now approximately zero chance that will ever happen.

I love Python but sometimes I feel like the decisions are arbitrary and do not reflect what is best for the language.

It drives me nuts because I'm a captive user. There's too much momentum for data science and machine learning to switch and be taken seriously in a non-solo workplace. Also I just like Python.

[PurpleRamen]

Which problem or usecase would null-coalescing/safe-navigation operators solve, which are not already solved with conditional expression or walrus-operator?

[ngrilly]

> No language that I know of has ever attempted a standardization effort like this.

What about Ruby, Rust, or Go, for example?

[cassianoleal]

Python packaging is really bad.

These days I just use virtualenv since at least it's simple to use and reason about. It's bad, just not nearly as bad as the others I've tried.

[Hackbraten]

You might like Relieving your Python packaging pain [1]. The article seems to come to a similar conclusion as you do.

[1]: https://www.bitecode.dev/p/relieving-your-python-packaging-p...

[ehutch79]

Why is venv bad though?

[hasty_pudding]

poetry is awesome. i recommend it

[dannyz]

From a user point of view Python packaging has probably never been better, but it is still a huge mess from anyone having to maintain these projects which might explain some of the dichotomy in the comments.

In my experience most of these packaging tools work fine for pure Python packages, it is when you try and bundle extensions in a cross-platform way that things get really messy. For better or worse I think third party package managers somewhat outside the Python ecosystem, i.e. conda + conda-forge, are the only tools that get this right.

[bspammer]

I really enjoyed this too (linked in the article) https://gregoryszorc.com/blog/2023/10/30/my-user-experience-...

[bvrmn]

Yes, it's a great read. It validated my frustration with python packaging.

[smitty1e]

Seeing the Poetry support on the thread, maybe just

1. pick the best of breed,

2. have the PSF sponsor a Packaging kickstarter for those of us who'd toss in some money but lack time/skill to do more than whine, and

3. get this mess fixed up for 3.13 this Fall or 3.14 (pi-thon?) in 2025?

I get it that the feature Venn across the scene has many disjoint parts. The management seems a greater challenge than the code itself.

[jjgreen]

get this mess fixed up for 3.13 this Fall or 3.14 (pi-thon?) in 2025?

I admire your optimism but don't share it. Crap packaging will be present in the Python experience for the lifetimes of multiple dogs. 2075 at the earliest.

[t43562]

I'm more worried about commercial entities gaining power - by simply employing enough people to work on something that the alternatives cannot compete.

People who want "one" anything - I can sympathise with the desire for simplicity (as long as it's a kind that suits you) but I hope it never happens.

[oblio]

This is a niche concern. There are no commercial package managers that I know of, in any mainstream ecosystem. Even Gradle, which probably comes closest, is perfectly usable without the commercial add-ons.

[bvrmn]

I have a dozens of packages created since 2009 and never had issues with distutils, setuptools also worked fine for me. Even C extensions weren't a problem.

Recently I've tried setup.py-less approach. Oh boy. What a mess "modern" python packaging is. Multiple backends. PEP-517. pip<23 could not install your package. `build` tool includes explicitly excluded files.

At least you can include markdown as a README for the package.

[nerdponx]

Setuptools is supposed to work with PEP 517 just fine with no changes, other than adding the build-backend declaration in pyproject.toml. Old versions of Pip will find your setup.py and use it directly, and new versions will find it via pyproject.toml and invoke it via `build`. Maybe you hit a bug? Or were doing something unusual and complicated. I never had a problem with the transition in a Setuptools pronect.

[lysecret]

Nowadays I am usually using micromamba inside Docker. I honestly never had any issues. I need docker anyway for deployment. And mamba/conda is almost needed for scientific stuff (working with some super niche weather stuff which is only on conda).

IMO the most important thing is to stay flexible, all of them have their tradeoffs pick one that works and move to a different one if it becomes painful.

Docker is very very useful though. Can you imagine the joy I felt when I spun up a 6 month old very complex project and it immediately worked.

[boxed]

The loss of Guido as the BDFL was not good for this. And honestly, even before that he was too much hands off to resolve this situation sadly.

[hasty_pudding]

Use Poetry. Python packaging problems solved.

[drcongo]

I don't know why the PyPA seems to be stubbornly ignoring Poetry, most Python devs I know have switched to it and are very unlikely to switch again to something else. It does what you need it to do, it's stable and reliable these days, and the ergonomics are good.

[bbojan]

Also Poetry integrates well with pip/venv and vice-versa, so you can transition to it step-by-step if your project consists of multiple packages across a multirepo or a monorepo. You don't have to switch everything all at once.

[BerislavLopac]

No, use PDM. ;)

The main problem with Poetry is that it still does not support PEP 621. [0]

[0] https://github.com/python-poetry/poetry/issues/3332

[apple4ever]

That's what I found. It just works. I never have to worry about anything.

Python should just adopt it.

[PurpleRamen]

Are there relevant use cases, which poetry is not supporting, because of which people prefer other solutions too?

[bvrmn]

Yes a great tool which breaks existing project tomls on update. Before poetry you had one issue and two with.

[guappa]

Poetry is just a wrapper on pip… might as well just use pip.

[ggregoire]

I use pip inside docker and it just works.

[coldtea]

If by "just works" you mean "now I also need to have and maintain a container image for isolation", yes.