[programd]

The worst part is that they marked the incident as resolved after 32 minutes, but didn't mention that mirrors for the packages on security.ubuntu.com have huge queues. OK fine, we can wait until the mirrors sync and you can choose another mirror to do your update - eventually. You can also work around this while updating Ubuntu 24.04 by manually installing the deb file it wants.

But wait, there's more! You can't install new instances of Ubuntu 24.04.2 because the installer connects to security.ubuntu.com by default (probably for good reasons) and will bail out while formatting and writing the disk when it gets 500 Internal Server Error from security.ubuntu.com for a specific deb file. There's no option around it that I'm aware of if you're doing an install connected to the network. I'm told that things should work if you try to install without networking connected. But that's not working for me, possibly due to some drivers it needs to pull for my hardware that are not in the default installer.

Ran into this while trying to install a fresh instance on an old mac hardware.

All in all, not a good look for Canonical, especially given how long this is taking to resolve and the lack of any status indication that this is still a problem. Lots of people are being bitten by this in the last 24 hours.

[e2le]

The craziest thing I've discovered is that unattended-upgrades does not timeout after failing to download pkgs from security.ubuntu.com AND will NEVER release "dpkg/lock-frontend". It will happily keep failing to download new pkgs, NEVER printing any error messages that I could see to the journal or a log file ("/var/log/unattended-upgrades"), and preventing the user from using apt because it holds a lock that it refuses to give up.

The process doesn't even respond to "systemctl stop unattended-upgrades" or SIGTERM. Only "kill -9" ends the titan grip it has over my systems.

Edit:

Out of curiosity I ran a packet capture, during the 8 minutes it was running, unattended-upgrades (apt) received 4MB and sent 182KB of packets. Given the unattended-upgrades package is installed by default on Ubuntu and the "apt-daily-upgrade" timer will run every 24 hours ((archive|security).ubuntu.com has being down for longer), I can only imagine that there must be millions of machines reaching out, repeatedly and uselessly, attempting to download new pkgs without any timeout over and over again.