top | item 42423771

(no title)

thingification | 1 year ago

That might be fine in your context. People's problems are real, though. What they're almost always missing is separating the source code from the compiled output ("lock files"). Pick a tool to help with that, commit both files to your ("one's") project, problem solved.

People end up committing either one or the other, not both, but:

- You need the source code, else your project is hard to update ("why did they pick these versions exactly?" - the answer is the source code).

- You need the compiled pinned versions in the lock file, else if dependencies are complicated or fast-moving or a project goes unmaintained, installing it becomes a huge mindless boring timesink (hello machine learning, all three counts).

Whenever I see people complaining about python dependencies, most of the time it seems just that somebody lacked this concept, or didn't know how to do it with python, or are put off by too many choices? That plus that ML projects are moving quickly and may have heavy "system" dependencies (CUDA).

discuss

order

thingification|1 year ago

To be more concrete:

In the source code - e.g. requirements.in (in the case of pip-tools or uv's clone of that: uv pip compile + uv pip sync), one lists the names of the projects one's application depends on, with a few version constraints explained with comments (`someproject <= 5.3 # right now spamalyzer doesn't seem to work with 5.4`).

In the compiled output - i.e. the lock files (pip-tools or uv pip sync/compile use requirements.txt for this) one makes sure every version is pinned to one specific version, to form a set of versions that work together. A tool (like uv pip compile) will generate the lock files from the source code, picking versions that are declared (in PyPI metadata) should work together.

My advice: pip-tools (pip-compile + pip-sync) does this very nicely - even better, uv's clone of pip-tools (uv pip compile + uv pip sync), which runs faster. Goes nicely with:

- pyproject.toml (project config / metadata)

- plain old setuptools (works fine, doesn't change: great)

- requirements.in: the source for pip-tools (that's all pip-tools does: great! uv has a faster clone)

- pyenv to install python versions for you (that's all it does: great! again uv has a faster clone)

- virtualenv to make separate sandboxed sets of installed python libraries (that's all it does: great! again uv has a faster clone)

- maybe a few tiny bash scripts, maybe a Makefile or similar just as a way to list out some canned commands

- actually write down the commands you run in your README

PS: the point of `uv pip sync` over `uv pip install -r requirements.txt` is that the former will uninstall packages that aren't explicitly listed in requirements.txt.

uv also has a poetry-like do-everything 'managed' everything-is-glued-together framework (OK you can see my bias). Personally I don't understand the benefits of that over its nice re-implementations of existing unix-y tools, except I guess for popularizing python lockfiles - but can't we just market the idea "lock your versions"? The idea is the good part!