top | item 41788026

PEP 760: No more bare excepts

164 points| ayhanfuat | 1 year ago |discuss.python.org | reply

385 comments

order
[+] kristjansson|1 year ago|reply
My concern, and IMO what should be the overwhelming concern of the maintainers, is not the code that is being written, or the code that will be written, but all the code that has been written, and will never be touched again. A break like this will force lots of python users to avoid upgrading to 3.17, jettison packages they may want to keep using, or deal with the hassle of patching unmaintained dependencies on their own.

For those Python users for whom writing python is the core of their work that might be fine. For all the other users for whom python is an foreign, incidental, but indispensable part of their work (scientists, analysts, ...) the choice is untenable. While python can and should strive to be a more 'serious', 'professional' language, it _must_ have respect and empathy for the latter camp. Elevating something that should be a linter rule to a language change ain't that.

[+] isoprophlex|1 year ago|reply
Exactly this. I totally agree. It's incredible to think that some people still run python 2 scripts; something unpalatable to the point of being nauseating for a day-to-day python programmer, but totally understandable in the context of incidental usage by a scientist dealing with legacy systems.

If these things start happening to python 3 on a larger scale, might as well throw in the towel and go for python 4.

[+] kragen|1 year ago|reply
The Python maintainers have switched from considering backward compatibility useful but costly to considering it actively harmful; there was a campaign a few years ago to convince library maintainers to stop making their code Python-2-compatible:

> The Python 3 statement was drawn up around 2016. Projects pledged to require Python 3 by 2020, giving other projects confidence that they could plan a similar transition, and allowing downstream users to figure out their options without a nasty surprise. We didn’t force people to move to Python 3, but if they wanted to stick with Python 2, they would stop getting new versions of our projects.

(https://python3statement.github.io/)

I know this sounds like a joke, and you probably think you're misreading, but no. Projects pledged to require Python 3 by 2020. They made a promise to break backward compatibility. Not just a few minor projects, either; TensorFlow, Spark, IPython, Pandas, NumPy, SymPy, Hypothesis, etc.

Since that happened, everyone who considers backward-compatibility good (if costly), rather than evil, has abandoned Python.

The "other users for whom python is an foreign, incidental, but indispensable part of their work (scientists, analysts, ...)" would have to fork Python, but it's probably too late for that; they can hardly hope to fork TensorFlow, Pandas, etc., as well.

[+] rurban|1 year ago|reply
major changes with breaking backward compatibility would require a major bump. Fine for python 4 I would say.
[+] m463|1 year ago|reply
Honestly 2->3 has been a HUGE mess, and python should be learning from these sorts of mistakes.
[+] o999|1 year ago|reply
s/except:/except Exception:/g
[+] Narhem|1 year ago|reply
Disagree, I’m so disappointed in companies who do sprint type development refusing to use Python. It works well with the “Silicon Valley startup ecosystem”.

That being said, as far as workplace differences I’d say Java shops would be the ideal, slower, less long term problems but so much more initial investment.

[+] cuuupid|1 year ago|reply
If I’m understanding this correctly this proposal would fully break compatibility with many (most?) codebases, actually remove syntactic sugar and force more characters for the same functionality. I fail to see how this is even being considered.

I don’t understand the idiomatic viewpoint either here, I understand the author personally finds it confusing when excepts aren’t verbose but I think you would be hard pressed to find many python developers who agree. Even outside the ecosystem, most languages have been adding more support for bare excepts (like js with bare catch) so this feels like a step backwards.

But maybe I’m just not understanding this proposal!

[+] phkahler|1 year ago|reply
>> I fail to see how this is even being considered.

To me it stinks of an ego-centric person thinking they're a "language developer" and knowing better than the actual users of the language what's best for them. Just because something can be misused doesn't mean you have to take it away.

I haven't noticed, but since Rust came along is there a trend among languages to enforce "safer" programming at the language level? I could see that kind of thinking getting way out of hand. If that's the case, I would see this one as "I'm going to save the world with this dumb little change that breaks things for a bunch of people!"

I would hope a PEP like this came about from frequent user requests but that doesn't seem to be the case.

[+] chefandy|1 year ago|reply
PEP contains lots of best practices that aren't enforced by the interpreter though, doesn't it? e.g. PEP 8. It's been a while so maybe PEP 8 is more unique than I realize? It seems like a pretty sensible recommendation that wouldn't necessarily need to change the way exceptions are handled by Python. Right there in PEP 8 it says in big text "A Foolish Consistency is the Hobgoblin of Little Minds." I imagine that enforcing this in the interpreter would fall under that, but it seems like a good piece of advice for folks new to the language, or more likely, new to coding.
[+] 01100011|1 year ago|reply
I don't know, but it seems like, as a language grows, the type of people working on improvements changes. In the beginning, the contributions come from people trying to solve application problems. In the end, the committees and contributors seem to be less connected to reality and more internal and isolated. They make changes that seem conceptually sound but aren't grounded in what the users of the language actually care about.
[+] Larrikin|1 year ago|reply
Python feels like they are fixing the language the best they can by slowly and properly adding explicit types. Character count is not a valid argument when we have hard drives that are multiple terabytes and IDEs and LLMs that will gladly auto complete a full word with the import. Foo, bar variable names and i, n incrementers should be banished to freshman level undergrad tests meant to intentionally confuse the student.

I'm hoping Python 4 will be a big breaking change similar to the previous one and full support for explicit types will be one of the reasons.

[+] dataflow|1 year ago|reply
> force more characters for the same functionality

This is actually a good thing in some cases (possibly this one). Risky stuff should inherently be harder to do than safer stuff, otherwise people will reach for the risky alternatives when they don't need to, just to save time - or because they don't realize the risk.

Or at least, that's often the case. What's lacking here is evidence that this is actually happening. I can believe it, but evidence is necessary for breaking the language.

[+] zahlman|1 year ago|reply
>actually remove syntactic sugar and force more characters for the same functionality. I fail to see how this is even being considered.

Python is not APL. Getting at the functionality in fewer characters is not a design goal - it's just a usually consequence of the actual design goal.

This is being considered because "Explicit is better than implicit." and because it helps avoid a common class of error (e.g. `except: continue` prevents aborting a loop with Ctrl-C, which is often not intentional).

Or as the PEP puts it:

> While this syntax can be convenient for a “catch all” handler, it often leads to poor coding practices: > > 1. It can mask important errors that should be propagated. > 2. It makes debugging more difficult by catching and potentially hiding unexpected exceptions. > 3. It goes against the Python principle of explicit over implicit.

Python isn't any other language, either. It certainly isn't taking design guidance from JavaScript (which runs in an environment where the page is expected to show something coherent, and not a loud screaming error, no matter how absurd the input data and/or code).

As for how much code it would break, you made me curious:

    ~/Desktop/dev$ find . -name "*.py" | wc -l
    49433
    ~/Desktop/dev$ grep --include='*.py' -rnw . -e 'except.*:' | wc -l
    109801
    ~/Desktop/dev$ grep --include='*.py' -rnw . -e 'except:' | wc -l
    5692
    ~/Desktop/dev$ grep --include='*.py' -rnw . -e 'except[ \t]*:' | wc -l # just to make sure
    5692
But drilling down further, over 2/3 of those bare excepts are in local copies of Python itself (i.e., multiple versions of the standard library) that I built from source. Probably all the rest are in dependencies. I don't write code like that myself if I'm even remotely paying attention. (Of course, that only tells me how many occurrences there were, not how many files have them. But the first two results imply an average of about 2 `except`s per file, so.)
[+] thiht|1 year ago|reply
I love using the bare except in small 1-file scripts, it just does the job elegantly
[+] amelius|1 year ago|reply
I do have to say that this fun little scripting language is starting to look more and more like a serious compiled language.
[+] williamsmj|1 year ago|reply
I have mixed feelings about this.

There are two "problems" this PEP is trying to solve.

One is that bare excepts are permitted. The argument against this is that explicit is better than implicit. A matter of taste, but I don't find this convincing.

The other problem is what bare excepts mean. Bare excepts are syntactic sugar for `except BaseException`. This means that an application containing a bare `except` followed by the vast majority of real-world error handling will continue to run even if SystemExit or KeyboardInterrupt is raised. This is almost always a bug.

I do find this second argument convincing, and I wish Python did not contain this design wart.

If I could go back in time and change Python syntax, it would make it hard for people to silently treat these special interrupts as "handleable" like regular errors. The tiny set of applications that really can and should handle them (e.g. TUIs or the mailman example discussed in the final section of the PEP) can explicitly do so with e.g. `except KeyboardInterrurpt` or even `except BaseException`.

But I agree with the consensus here that this does not rise to the level of something being worth a backwards-incompatible change.

[+] theamk|1 year ago|reply
Disagree. I do this kind of code all the time:

   try:
      something()
   except:
      log_tons_of_debug_info()
      raise
and I am very glad that I get my debug info works even if I press Ctrl-C or someone calls sys.exit().
[+] gmueckl|1 year ago|reply
Java solved the problem by having Throwable as the root of all exceptions and not advertising that fact loudly. The derived Exception class is the root of all safely catchable exceptions. When someone catches a Throwable, something strange is going on.
[+] rectang|1 year ago|reply
> Bare excepts are syntactic sugar for `except BaseException`.

I'm guessing that a `3to4` script would be provided which replaces bare `except:` with `except BaseException:`. We have the experience of `2to3` to draw on with regards to how that might play out.

EDIT: Haha, I now see that this PEP proposes a change without advancing the major version. That surprises me.

[+] MantisShrimp90|1 year ago|reply
I think Rich Hickeys advice of not breaking people applies here.

The anger from this potential change is that really all you are doing is taking something away that was working, and now people will need to review their code or keep python on a previous version which sucks.

I think that people who propose these kinds of changes don't appreciate the importance of the programming language being at the bottom of the stack so there's really never a good reason to break people even if you think it's nicer as you really can't appreciate how much work you are creating for people.

[+] kej|1 year ago|reply
The PEP has been withdrawn based on the poll results, which mostly echo the comments here: https://discuss.python.org/t/pep-760-no-more-bare-excepts/67...

>Hi everyone!

>Thanks a lot for voicing your opinions and concerns! After reading carefully all the arguments, the poll and the different positions we have decided that the best course of action is to withdraw the PEP as there is clear agreement that the breakage doesn’t justify the benefits here.

>Thanks a lot!

[+] aftbit|1 year ago|reply
One must imagine Sisyphus happy. Python just loves to break working code on a regular basis with its new releases. If your code is protected from untrusted user data and the internet, Python 2 is actually a really nice language that doesn't constantly force rewrites.

Oh, you want to know the naive UTC datetime in Python, to interface with something like PostgreSQL that recommends naive times? Back in the old days, a simple datetime.datetime.utcnow(). Now days, you need something like:

    try:
        from datetime import UTC as tz_UTC
    except ImportError:
        from pytz import UTC as tz_UTC
    dt = datetime.datetime.now(tz_UTC).replace(tzinfo=None)
[+] instig007|1 year ago|reply
> Oh, you want to know the naive UTC datetime in Python, to interface with something like PostgreSQL that recommends naive times?

Postgres never recommended naive datetimes. A TZ-aware datetime is semantiacally the same as a tuple of (<location/agreed offset>, <time in the moment since unix epoch defined in terms of UTC>). Those who recommended dropping the knowledge of the first part from that pair did it because they didn't know better.

[+] bigstrat2003|1 year ago|reply
> If your code is protected from untrusted user data and the internet, Python 2 is actually a really nice language that doesn't constantly force rewrites.

If Python 2 is acceptable for your use case, then you could stay on an old version of Python 3 just fine as well.

[+] hauntsaninja|1 year ago|reply
You don't need to use pytz, you can use the following on all Python 3: tz_UTC = datetime.timezone.utc
[+] joshkel|1 year ago|reply
Interesting.

The way I was taught Python, you really, really don't want to use bare `except:`, because it catches _everything_: Ctrl-C interruptions, system exit, etc. Instead, you really ought to use `except Exception:` (where `Exception` is the base class for any "normal" runtime error).

So I definitely understand the rationale, but it's hard to say it's worth the pain of backward incompatibility - we have linters, style guides, etc. that can catch this.

[+] wild_pointer|1 year ago|reply
Yes, I was bitten by it in the past. Still, it'd better be a lint, or at least a very very long deprecation period... like, deprecated and removed in Python 4 or something.
[+] dimator|1 year ago|reply
absolutely, this should not be done at the language level. the language should not enforce "best practices", that's what the ecosystem is for.
[+] sph|1 year ago|reply
This would be a backward-incompatible change, no?

In other languages, such a change would only be possible with a major version bump, though I imagine that because of the Python 3 collective trauma, the language designers now are OK with breaking older code without calling it Python 4. Anything goes, as long as it's called Python 3.

(Python lost me in the 2->3 migration and I haven't used it in a decade, so correct me if I'm wrong)

[+] mananaysiempre|1 year ago|reply
I don’t think there have been significant backwards-incompatible changes in Python 3 the language. There have been some, e.g. async and await were first introduced as soft keywords and then switched to being actual ones. But that’s not all that different from the treatment of yield in Python 2.

(Recent stdlib changes have been much more destructive, but I’m assuming that, like in the original thread, we’re drawing a distinction between those and changes to the actual language.)

Full disclosure, I welcomed Python 3, because for me that was the first time (since 2.4 on Windows XP) that I could count on my programs not randomly shitting their pants upon encountering Cyrillic in files or filenames, which for a native speaker of Russian you can imagine is quite important. (The csv stdlib module in Python 2 did that, IIRC. Perhaps I was holding it wrong, but experience shows that absolutely everybody did.)

[+] letmeinhere|1 year ago|reply
Python doesn't use semantic versioning. The number after the first period is a major (annual) release and can and does contain breaking changes (though so far never on the scale of the 2->3 upgrade).

We may never see a 4.0 because of the scar tissue, but the language continues to evolve.

[+] stackskipton|1 year ago|reply
This is what not having sane BDFL does to organization.

Only proper response is private email from sane BDFL similar to Linus email of "WE DON'T BREAK USERSPACE"

[+] graemep|1 year ago|reply
This reeks of "our users are idiots and we need to keep them away from sharp edges".

A bare except is something to be flagged up by tools, not disallowed by the language. It is definitely not worth a backward-incompatible change.

I am slowly going off Python.

[+] forgottofloss|1 year ago|reply
"our users are idiots and we need to keep them away from sharp edges" is exactly what keeps driving me away from Python and pip. It's why I wrote https://pip.wtf -- Python package management would be so simple if they'd just stop adding more and more seatbelts and cushions to Python.
[+] OscarCunningham|1 year ago|reply
One example of a time I used a bare except was when I wanted a program to retry three times if it failed for any reason. I just wrapped everything in a for loop with a catchall except.

The problem occurred when our scheduling program (Airflow) noticed the program was taking too long to run and decided to kill it. It sent a kill signal to Python, which dutifully caught the exception, retried and continued to run. I had to add a special case to allow Airflow to kill the program.

This PEP just forced me to look up the difference between the classes Exception and BaseException. It turns out that BaseException includes every exception, whereas Exception excludes those that are trying to exit the program (like SystemExit and KeyboardInterrupt).

[+] xorcist|1 year ago|reply
Valid reasons for backwards incompatible changes to language syntax:

1. The language guarantees have become inconsistent, and the syntax breaks security boundaries or realtime guarantees that the language explicitly promises, and it is unfixable.

2. The universe of all written code is small enough that there are guarantees the syntax is unused, or we can change all instances of the syntax in an atomic manner.

Invalid reasons for backwards incompatible language changes:

1. Everything else.

[+] jmyeet|1 year ago|reply
I'm not a fan of this for two reasons.

First, consider this motivation from the PEP:

> Requiring specific exception types makes the programmer’s intentions clear and encourages thinking about what exceptions might occur.

This reeks of the same rationale for checked exceptions in Java. That was a failed experiment. You can't force people to deal with exceptions. They end up just swallowing them instead. It's better to propagate an exception than do that almost all the time.

Like will we see except: replaced with except object:? I don't even know if that's valid. It's never come up.

Second, this would be a breaking change. I really feel like this is where Python 3 went off the rails. In the Python 2 days making breaking changes was essentially verboten. But ever since Python 3 decided breaking changes were OK< it's like there's little restraint now and minor releases seems to be far too comfortable with this.

[+] bee_rider|1 year ago|reply
They explicitly describe the PEP as evil, is there a tradition in the Python community for having obviously terrible PEPs, just to document the reasons for not doing something? Because that would make this a lot more understandable.
[+] zmnayt|1 year ago|reply
As many people have observed here, this is a couple of Steering Council members showing activity. Getting one's PEPs accepted has a totally inflated weight in the Python "community". The more, the better (by contrast very few people care about perfect and bug-free code).

So, if this thing is accepted, it pads the resume of certain people even more. And many software orgs will have one additional week of job security by rewriting existing code. It's a win-win situation.

Ever since the walrus operator coup Python has descended into madness and make-work initiatives.

[+] bunderbunder|1 year ago|reply
I get that lately Python has decided it wants to be an industrial-grade enterprise programming language. But there's a part of me that misses when the Python community retained a "we're all adults here" ethos.
[+] berdario|1 year ago|reply
A bunch of people are mentioning the bugbear of the Python3 migration, but there's an important difference that makes the migration a lot simpler for a backward-incompatible change like this one proposed in PEP760:

You can just write code that is compatible with Python runtimes both before and after the change.

That means that you can use the same test suite, gradually getting the code more and more compatible with the new version, and you can switch your production runtime, without having to worry about a more complicated and involved rollback process.

Notably, in the Python 2->3 migration, it was not really possible (at first[*]) because "" (and b"") literals became -> b""

while u"" literals became -> ""

So, there was no way to write literals that would mean the same thing, and have the same type across the two versions

This is also the reason why libraries like six offered a `six.u` function (https://six.readthedocs.io/#binary-and-text-data) but that required banning use of non-ASCII codepoints in your string literals

[*] This was eventually addressed with PEP 414 (and of course, even with with PEP 414, the migration was not trivial)

https://peps.python.org/pep-0414/

[+] move-on-by|1 year ago|reply
I just completed upgrading a monolith from Python 3.8 to 3.11 - no doubt many others in the same position with 3.8 going EOL. It was a monumental effort. I will say the huge majority of work was upgrading libraries that hadn’t been updated in years. I won’t go into the specifics of why we had chosen not to update these libraries earlier (unless there is interest), but I will say Python being as backward compatible as possible has huge real world value. More for the community and ecosystem than the language itself. For the people who care about PEP 760, they have their choice of linting tool to enforce this requirement.
[+] tln|1 year ago|reply
-1000

Why would you ever consider breaking everyone's throwaway scripts?? For what is already a universal linter rule?