Things I've learned about building CLI tools in Python

[guessmyname]

I’ve noticed that I never quite feel at ease with the Python programs I write.

I’ve been using Go to create projects, both big and small, since 2013.

Almost every time I attempt to build something even remotely complex with Python, I end up regretting it, especially when other people besides myself start using these programs. The main problem is the lack of assurance that the same program will function correctly on another person’s computer. With Go programs, it’s as simple as having a statically linked binary, and given the ease of cross-compilation, I’m very confident that what works on my machine will work on my coworker's or customer's computer as well.

You know how some people suggest that Shell scripts should not exceed a certain number of lines, because beyond that point, it’s better to create a Python, Ruby, PHP, or similar script? I experience a similar sentiment when working with Python. A few hundred lines may be acceptable, but anything larger than that, I believe, is better suited to be written in a compiled language.

[athrun]

I feel the same way.

Python has been my goto language for a long time, but lately I've been noticing that I've been holding off on writing new tools with it because on the back of my mind I have this nagging feeling that making them robust and portable will take too much work—and so I don't even bother getting started.

It's this trap of yes you get to ~99% pretty fast, but the last 1% (packaging/distribution) then take forever.

But I'm still looking for a good alternative... Golang does the job—no question, but it doesn't spark joy for me.

[Jare]

My rule of thumb used to be shell scripts past 100 lines get converted to Python, and Python scripts past 1000 lines should get converted to something else. But in practice, the Python has stayed almost always.

[pjmlp]

I know Python since version 1.6, and Go is such a downgrade in productivity that I would only use it when not given an option, like on some DevOps tools.

As someone that has experience with static binaries since 1990, way before dynamic loading was a common option in modern computing, yeah it works on the other computer, provided the distribution is exactly the same, and all required files and network configurations are exactly the same.

[rjzzleep]

I can't say I can relate at all. If you do things from scratch that might be true, but there is a pretty popular python tool called cookiecutter that allows you to generate the basic skeleton of the app. I usually pick something that contains poetry, click(I guess there is typed now) and some linting choices.

For fun I just googled a template and tried: https://github.com/radix-ai/poetry-cookiecutter

And the result is quite good.

Your comment assumes that python cli scripts need to be single liners, but IIRC there are several tools that allow you to bundle a package into a single file like pex, shiv, and zipapp.

[rtpg]

There are packaging tools for Python, and if your tooling is targetting people already using Python, just relying on `pip` + writing a proper pyproject.toml is a good solution nowadays (protip for people with virtualenv issues: direnv solves so much of this it's not funny).

But I have been looking around for a while for something that's more certain than `pip`, and unfortunately everything I've found (like Bazel or Buck) suffers from having to do a lot of futzing to use dependencies.

[neuromanser]

> I experience a similar sentiment when working with Python. A few hundred lines may be acceptable, but anything larger than that, I believe, is better suited to be written in a compiled language.

Python, IMO, has no niche anymore. A few hundred lines of Python is a hundred lines of Zsh, or the same few hundred lines of C++, and to top it off, there's the shit show of Python tooling for deployment. setup.py, requirements.txt, pyproject.toml… Fifteen files with overlapping contents in twelve different grammars (mild exaggeration), with new ones added every other year. Setuptools can't find your entrypoint…

[abdusco]

Fingers crossed for vlang[0]. It's like golang with better types and more syntactic sugar. Feels like a proper upgrade from Python.

I really hope they succeed.

[0]: https://vlang.io/

[switch007]

For me Python is addictive.

You know the tooling is bad and in the long term it will hurt, but the standard library and third party packages are just phenomenally productive and that’s a huge draw.

[ShadowBanThis01]

I was going to learn Python for the same reason: to create utilities that would run on most any computer. Mostly to do things like file-parsing and data-format conversion.

But the Python ecosystem seems to be such a disappointing mess that I just gave up on the whole idea. I'm learning JavaScript/TypeScript now and you can build CLI programs with Deno.

[DanielHB]

If you distribute any CLI tool you should include the runtime and any attached dependencies, but with dynamic languages that can easily put your distributable in the tens of megabytes in size which is a bit of a pain.

I mean for the longest time the AWS CLI used the python/pip installed in your own machine and it probably caused thousands of man-hours of wasted time.

[xen0]

The equivalent to static linking in Python would be bundling all code into an archive (including transitive dependencies), along with an interpreter. Some shell script can be used to unpack and run.

It's possible, just not the norm.

[samsquire]

I wrote a tool once that would do healthchecks before doing anything it would format it in a lovely table.

It would clone repositories (microservices) and configure LXC containers.

[hiAndrewQuinn]

I build little CLI tools in Python non-stop. ChatGPT and some basic knowledge of how the `click` library works has made it almost completely trivial to get the ball rolling for whatever need I have for it, `--help` text included.

The fact that the barrier for creation is so low means I'm even willing to do them to solve very niche problems in generalizable ways. [1] is common enough that a few people have starred it. [2] is niche enough that other Anki folks haven't used it AFAICT. [3] is likely something I'll never personally need again, even though Azure VM reservations not letting you customize your reminders for when they're about to expire is probably a costly mistake for a great many firms.

All started with this same starting methodology, because what I wanted was just a little too fiddly to want to hack together with my shell toolkit.

[1]: https://github.com/hiAndrewQuinn/finstem

[2]: https://github.com/hiAndrewQuinn/table2anki

[3]: https://github.com/hiAndrewQuinn/AzureReservations2ICS

[thrdbndndn]

I'm sure click has its advantage if your CLI is particularly complex, but for me the built-in argparse is more than enough, it has almost all the common things you need.

By the way, argparse (and I assume click too) by default allows having positional arguments and switches in any order, i.e., both:

    mycli pospara0 --switch --option A
    mycli --switch --option A pospara0 

work. This seems like nothing but I've encountered many CLI utilities written in other languages (particularly, go and node.js) that force you to have switches at the beginning. and I really hate that.

I don't know if it's caused by their corresponding default/popular CLI library or what, someone could enlighten me.

(Of course, in some cases like things like FFMPEG, the order absolutely matters; but it's not the case for 99% of utilities.)

[omgmajk]

Agreed. I, and we (at work) use argparse and it works as intended. I don't know why I would ever switch at this point. Also I feel like arguments should not be ordered unless absolutely necessary, just feels like a head ache to me.

[deniscepko2]

Same here argparse does everything a cli tool would ever need. Looked at click lib and actually don't even find it more readable

[apple4ever]

> This seems like nothing but I've encountered many CLI utilities written in other languages (particularly, go and node.js) that force you to have switches at the beginning. and I really hate that.

Same here. Why I am force to remember or look up which order arguments should go in? There is no reason for that and they should be able to go in any order.

[raffraffraff]

I hate it when a cli forces ordering of args when there's no reason to! It's mitigated somewhat by decent tab-completion that only completed what is allowed.

[atoav]

blender is also one of those where order matters. »Oh, you want me to render after loading the file, then you should have told me«

[crabbone]

> I'm sure click has its advantage if your CLI is particularly complex

None whatsoever. Argsparse is better all around. Click is just a worthless piece of software that nobody should be using.

As for the order of options / arguments. I think, the reason is the historical implementations and use of getopt that would be used in a switch inside a loop, which (maybe unintentionally) made the order irrelevant. It's likely that other libraries implement parsers in the way that is sensitive to the order. Whether that's deliberate it's hard to tell. There are definitely advantages to this approach too, but it's hard to know whether authors sought out those advantages deliberately.

For instance, when options can take arguments (especially when they can take multiple arguments) they can be confused with sub-commands or the arguments to commands. Imposing ordering restrictions helps to resolve ambiguities as to what argument is being processed. On the other hand, you may claim that not imposing ordering on arguments prevents CLI authors from creating confusing interfaces where users can accidentally mix arguments to options with sub-commands or arguments to commands.

[jdoss]

I have been using Typer on every one of my CLI projects which uses Click under the hood. The documentation is fantastic, the CLI app it produces looks great and Typer lets you create things quickly. I high recommend it.

https://typer.tiangolo.com/

[stevenrj]

I've been using docopt to handle CLI arguments for years now.

http://docopt.org/

[wedn3sday]

>> Flags with single character shortcuts can be easily combined—symbex -in fetch_data is short for symbex --imports --no-file fetch_data for example.

I pretty much use argparse for making all my CLI tools, but I dont know of an easy way of doing this single character flag thing. Is it possible/easy with argparse?

[jmholla]

`argparse` does it by default:

    >>> import argparse
    >>> p = argparse.ArgumentParser()
    >>> p.add_argument("--foo", "-f", action="store_true")
    >>> p.add_argument("--bar", "-b")
    >>> p.parse_args(["-fb", "baz"])
    Namespace(foo=True, bar='baz')

[m463]

I use argparse too, and it's one of the best python libraries (and my most-used)

you can do short (one character) or long arguments with argparse directly:

  parser = argparse.ArgumentParser(argument_default=None)
  parser.add_argument('-d', '--debug', action='store_true', help='debug flag')

I also do lots of other things, like long help with no args like this:

  if len(sys.argv) == 1:
      parser.print_help(sys.stderr)
      sys.exit(1)

[reassembled]

In my experience building large applications in Python becomes delicate due to the lack of static typing, as well as overlooking issues of scope in variable usage. It can be avoided with diligence but I’ve definitely shot myself in the foot and let errors slip through in Python programs I’ve written for the above reasons, which ended up compromising the validity of the program (mainly automated test scripts that were used to test other software and hardware).

I’ve only been programming for about 5 years in earnest. I held on to Python for dear life in the first days of my career, but have since transitioned to full-time C/C++ development, primarily in embedded and hardware interfacing applications. I feel like my large programs are much more manageable and maintainable now. Some of this is of course due to having grown as a programmer as well.

[nylonstrung]

Could one not just use a tool like Mypy that strongly enforces static typing in Python?

It seems like you get a lot of the benefit of static typing if you adopt it as a self-imposed constraint?

https://breadcrumbscollector.tech/mypy-how-to-use-it-in-my-p...

[spearo77]

The folks at Textualize have taken it one step further with https://github.com/Textualize/trogon

It's a neat way to make powerful CLIs more accessible to less-technical users.

[renewiltord]

This rules. Thank you for sharing!

[ArcHound]

I've came to the same conclusion as the author some time ago, my cookiecutter template is more opinionated https://github.com/ArcHound/python_script_cc . Best for use-cases when you need to do some automated API calls. Will checkout Typer and Textualize too, thanks HN!

[quickthrower2]

Is there a way to compile a python CLI script, and it’s dependencies and python itself into an executable.

That makes the tool nicer to use. To me a CLI tool should stand alone ideally. Obviously that is not the trend as many things that are CLI are installed via node or npm.

I guess docker could solve most of the issues here

[crabbone]

The answer is "sometimes".

Python can be relatively easily embedded in a C program and its source code can be compiled to C. The problems come from Python modules that are built to use shared libraries. It's not impossible to solve, but it means that you'd have to find the source code for those modules and recompile them to link statically with those libraries. This could be quite an undertaking, and is probably not worth it, unless you want to learn more about build systems and build tools in general.

Finally, in some cases it's impossible due to the licensing. I.e. you may have a Python module that relies on a shared library with license that prohibits redistribution. In that case it's not a technical, but a legal problem. This, however, isn't unique to Python, and you'd face similar issues no matter the language you chose to use.

Re' Docker: in most cases this is not a solution to making command-line interfaces. I actually struggle to think in what case it is. You'd have to write a program with the command-line interface and then put it in the image for Docker to create a container from (which will usually make it very inconvenient to use due to the Docker containers by default running such programs in separate filesystem, user and network interfaces.) This would make things like user identity, user's data and, well, obviously, network hard to access for the program while gaining you noting of substance.

[zinodaur]

Yeah! pyinstaller is an example. They do it by bundling a standalone python interpreter (x-platform ones too) with the necessary python libraries bundled in, just like you suggested

[xavdid]

I recommend pipx (https://pypa.github.io/pipx/) for this to get the same basic result. While it's not a pre-compiled binary, it is a standalone installation that takes care of dependencies and virtual virtual environments in a way that the user never has to think about them. As far as they're concerned, they `pipx install ...` and it "just works".

[simonw]

There are a few tools that can do this. I've had good initial results from PyInstaller: https://til.simonwillison.net/python/packaging-pyinstaller

[archargelod]

You better off with using a compiled language.

If you interested in a language that's compiled, fast, but as easy and pleasant as Python - I'd recommend you take a look at Nim: https://nim-lang.org.

Nim has cligen library to generate and parse arguments: https://github.com/c-blake/cligen

And to prove what Nim's capable of - here's a cool repo with 100+ cli apps cligen author wrote in Nim: https://github.com/c-blake/bu

[tgmux]

Containers are the strategy I've used in the past for this purpose. For my needs, I've found any extra runtime to be negligible.

[raffraffraff]

It's the main reason for Go's popularity imho. I loved the fact that all the Hashicorp stuff i used (consul, packer, vault, terraform) were just binaries.

[synthos]

Maybe one day, Mojo

[d4rkp4ttern]

I’ve used Tyler and Fire and like them both but recently I’ve been in search for a Python Lib that gives user numerical choices and allows arrow navigation, like the “gh” (GitHub) CLI. I wasn’t able to find one. Anyone has a rec? Thanks

[jackblemming]

Simon is a well of knowledge and good advice!

[thowafasdflkj]

I use clap and embed cpython

[tbrockman]

This is the way.

clap is a much better developer experience (IMO) and you end up with performant (no terrible cold starts) and strongly-typed code (where possible) without having to deal with building and distributing a Python CLI.

I will never forget falling in love with Python when I first started learning to program, but experiencing internal CLIs written in Python at scale is an experience I would encourage everyone to avoid unless UX and maintenance aren’t concerns.

[psd1]

No mention of completions.

How does HN provide tab-completion for CLI commands?

[crabbone]

Saw "Click" being used. Didn't read further. This is worthless.

For those who don't know. Python has argsparse package that ships with every Python distribution. It's much better in terms of organizing command-line arguments, easier to debug, easier to extend (which is very rarely necessary).

Click is a third-party dependency. It's not solving any real problems. It's not like argsparse had a problem and Click came to solve those. It's just that author had too much spare time on their hands and decided to learn how to do something new. The author made some rooky mistakes along the way. He totally misunderstood how locales and encodings work and for a while Click was a source of errors related to that. Maybe still is, but fewer packages are using it? -- I don't know.

If anyone chooses to use Click over argsparse, it only means lack of research. Following fads w/o any sort of independent thinking. Not someone I'd encourage to take advice from.

[oefrha]

click an alternative argparse API and then some (progress bar, for instance). While I prefer argparse to click, saying it’s worthless because argparse exists is like saying requests is worthless because urllib.request exists.

Btw, mitsuhiko created Flask, simonw created Django. Total rookies, I know.

[apple4ever]

Thank you for this. I was wondering why Click is around when argsparse works great and has everything needed (and does not enforce positional arguments).