The Trouble with __all__

[trainfromkansas]

__all__ is only relevant for * imports.

And please, just don't use * imports. It really doesn't save you much time at the cost of implicit untraceable behavior. If you don't worry about * imports, you don't need to add the __all__ boilerplate to every module.

This article is more about advertising a package called tach, that I suppose tries to add "true" private classes to Python.

But it doesn't actually enforce anything, because you still need to run their tool that checks anything. You could just easily configure your standard linter to enforce this type of thing rather than use a super specialized tool.

[claxo]

A direct benefit of using `__all__` at module A is better intellisense while editing files that imports A, if A has a small intended public API and many internal usage symbols.

[matsz]

Python's imports are the worst I've seen in any mainstream programming language by far.

Relative path imports are unnecessarily difficult, especially if you want to import something from a parent directory. There's no explicit way to define what you'd like to export either.

The syntax is inconsistent, too:

    from X import Y
    import Z

vs. (modern JS)

    import { Y } from 'X';
    import * as Z from 'Z';

Even C/C++ make more sense here.

[Spivak]

> Relative path imports are unnecessarily difficult, especially if you want to import something from a parent directory.

This is never explained well to beginners but this is because Python imports deal with modules and packages, any relationship with directories is accidental. With namespace packages foo.bar and foo.baz might live on totally separate parts of the filesystem. They might not live on the filesystem at all and instead be in a zip file.

    somedir/
    otherdir/
      __init__.py
      foo.py
    scripts/
      __init__.py
      myscript.py

    # in myscript.py
    from .. import otherdir.foo

If you `python scripts/myscript.py` then it won't work because myscript.py is run with a __name__ of __main__ which doesn't have any parent module.

When you run python -m scripts.myscript it sets __package__ to scripts.myscript so now it does have a parent module and somedir and otherdir are importable under the root module because '.' is added to the search path by default.

[simonw]

It took me far longer than I'd like to admit for "import *" in modern JavaScript to make sense to me.

Python imports made sense straight away - but then I've been a Python programmer for a long time so I'm far removed from a beginner's perspective.

[drhagen]

I always thought Scala had nice imports [1]:

    import users.*
    import users.User
    import users.{User, UserPreferences}
    import users.UserPreferences as UPrefs

[1] https://docs.scala-lang.org/tour/packages-and-imports.html#i...

[BurningFrog]

The worst thing is that it can't handle circular imports, unlike every other language, AFAIK.

I've been told it's because of some deep part of Python interpretation that just can't be changed. The mutable default argument madness is also caused by that.

Anyone know more?

[plasticeagle]

Python is just plain unsuitable for any project larger than a couple of files.

Every Python project contains a hidden and deadly complexity that will grow over time - and will eventually destroy it. There's no way around it, it creeps in no matter what you do. The imports situation is only part of it - it wasn't what killed our simulation tools, or our build scripts, or our test framework - and required that we rewrite them all in different and more suitable languages.

Python's performance, global modules, whitespace, untyped-by-default code are all killers. You pretty much have to use virtual environments to permit isolation between the multiple differently-versioned sets of dependent packages that you'll need for any project of any complexity, which are a cumbersome and painful solution to a problem that simply shouldn't exist.

It may technically be possible to write clean and maintainable code in Python if you try hard enough, but you're always skating so close to the edge that eventually somebody is going to get in there tip the whole fragile mess into the abyss.

[sevensor]

I’ve led the maintenance and development of a medium sized (100s of kloc) Python codebase over a period of five years. As it matured, the code became significantly less fragile and more maintainable. Using private members of modules (or any other object) turned into way less of a problem as we improved our design. Turns out the problems were mostly around I/O and mutable state. We fixed those problems by pushing side effects and mutation as far out to the edge of the codebase as possible. If your core data processing routines don’t have side effects and don’t mutate global state, who cares who the caller is?

[nick238]

What's the one true language we should all be using?

[Daishiman]

Looks like somebody isn't running a linter on CI.

[aatarax]

Don't know about using __all__ for introspection, but I have found it immensely useful for organizing, reading, and communicating code. When a package has a bunch of files inside of it, but only a handful of names exposed in __all__ it helps a lot with orienting yourself around the package.

[InfoSecErik]

The author seems to expect someone to be patrolling imports with a gun rather than a strongly-worded "we're not liable if you hurt yourself" sign.

[klyrs]

Tell me you've never violated pep8 in public without saying pep8

[zokier]

If you think this sort of hack is going to keep your python codebase clean I got some bad news for ya.

Also: https://docs.astral.sh/ruff/rules/import-private-name/

[anamexis]

That's the point of the article.

[gorgoiler]

It would be interesting to compare this with an alternative based on static analysis.

The Python ecosystem has many standard tools nowadays to enforce consistent style, including how modules import each other. The ast and libcst modules are very fast and can quickly identify any imported symbols beginning with an underscore:

  from a import _naughty

It’s also quite possible to build a list of symbols that were imported and ensure that their underscore-prefixed attributes are not accessed:

  import a

  a._naughty()

You could get creative I suppose…

  import a
  f = next(
    getattr(a, f“_{s}”)
    for s in synonyms(“cheeky”)
  )
  f()

…but at that point one hopes that, as a last resort, ones reviewers would cry foul.

[nuttingd]

  import my_module

This is compatible with `__all__` if you have your code broken down into smaller sub-modules and collect them in the main module as follows:

  # my_module/__init__.py
  from .submodule import *
  from .another_submodule import *

[jph00]

Yup that's what I do too. Works great. And nbdev auto-generates __all__ so I don't have to think about anything -- it all just works.

(Besides which, when you do have __all__ just regular wildcard imports work well anyway. I've been using them for >10 years without trouble. I think people just repeat the claim that they're a problem without having 1st-hand experience of using them with __all__ correctly.)

[kstrauser]

Eh. Python does indicate public and private APIs. "Name" is public. "_Name" is internal, but you can mess with it if you need to. "__Name" is private, and if you go there anyway and the thing explodes and gives you a bad haircut, you were warned.

Python's position is that we're all adults here. Don't touch "_Name". Really don't touch "__Name". You can if you're an expert and willing to take responsibility for your actions.

Addendum: And in this ModuleWrapper thing, I can still access `core._module.PrivateApi` so we're back to square one.

[0x63_Problems]

In my experience, Python devs rarely use the underscore.

Package authors are pretty good about it. But when the only people using your code are your coworkers, people don't seem to put as much thought into having a clean interface.

[VeejayRampay]

"we're all adults here" is self-justification for the fact that the language is unable to enforce anything

[efilife]

This is a problem with the language

[VeejayRampay]

once again, python being the absolute worst amongst all widely used languages