Good time to note that Buypass offers free certificates over ACME. I have a few of my domains configured to use them instead of LetsEncrypt, just for redundancy and to ensure I have a working non-LE cert source in case LE suffers problems like this over a longer time period.
Example OpenBSD /etc/acme-client.conf:
authority buypass {
api url "https://api.buypass.com/acme/directory"
account key "/etc/acme/buypass-privkey.pem"
contact "mailto:youremail@example.com"
}
domain example.com {
domain key "/etc/ssl/private/example.com.key"
domain full chain certificate "/etc/ssl/example.com.pem"
sign with buypass
}
This is neat. Does cert-manager have facilities to automatically use a fallback ACME provider, so I could automate using this? I'd also accept a pool of ACME providers, but a priority ordering seems ideal. I don't see the functionality listed anywhere, maybe there's some security argument that this is a bad idea?
Cheers! They look like decent chaps and also outside the US for some additional certificate diversity. Are there other trustworthy Acme issuers out there?
A pity that acme-client(1) does not allow for fallbacks, but I will add a mental note about it being an easy enough patch to contribute if I ever find the time.
Let's Encrypt stopped its certificate expiration email notification service a while ago, and I hadn't found a replacement yet. As a result, I didn't receive an expiration notice this time and failed to renew my certificate in advance. The certificate expired today, making my website inaccessible. I logged into my VPS to renew it manually, but the process failed every time. I then checked my cloud provider's platform and saw a notification at the top, which made me realize the problem was with the certificate provider. A quick look at Hacker News confirmed it: Let's Encrypt was having an outage. I want to post this news on my website, but I can't, because my site is down due to the expired certificate.
They have been communicating the ending of the email notices for quite a while and have been telling users that you should have some other monitoring in place to avoid just this situation
> Let's Encrypt stopped its certificate expiration email notification service a while ago, and I hadn't found a replacement yet.
This sounds like an easy problem to identify root cause for.
I think I received about 15 'we're disabling email notifications soon' emails over the past several months - one of which was interesting, but none were needed, as I'd originally set this up, per documentation, to auto-renew every 30 days.
Perhaps create a calendar reminder for the short term?
Haven't they always, from day one, insisted that their primary goal was to encourage (force) automation of certificate maintenance, as a mechanism to make tls ubiquitous (mandatory everywhere)?
Because you're not supposed to rely on emails. You should have an automated certificate renewal in place. I'm under the impression that Let's Encrypt wants to reduce certificate validity even further from the current 90 days.
Oof, you're right, that's rough that it's so soon after they discontinued their email service!
I wrote this blog post a few weeks ago: "Minimal, cron-ready scripts in Bash, Python, Ruby, Node.js (JavaScript), Go, and Powershell to check when your website's SSL certificate expires." https://heiioncall.com/blog/barebone-scripts-to-check-ssl-ce... which may be helpful if you want to roll your own.
(Disclosure: at Heii On-Call we also offer free SSL certificate expiration monitoring, among other things.)
If it's a personal website you should consider HTTP+HTTPS. It offers the best of both worlds and your website would always be accessible even if some third party CA is not (or if there's some local issue, or if the HTTP client connecting has cert issues). MITM attacks on personal websites are extremely, extremely rare.
whoa whoa whoa.. slow down! you dont just leap to "It's DNS"... you have to try to blame everything else first before you get to DNS. it's like foreplay!
(Or, as I recently encountered, it can also be a McAfee corporate firewall trying to be helpful by showing a download progress bar in place of an HTTP SSE stream. I was sure that was being caused by MTU, but alas no.)
Mostly this should be a non-event due to renewal long before expiration? Although huge deal I suppose for services that require issuing new certifications constantly; Let's Encrypt would be major failure mode for them.
I encountered this while trying to issue a new certificate for a service. As a temporary fix, started using ZeroSSL which conveniently also supports the ACME protocol. While not a big problem, if you have something like `cert-manager` being used on Kubernetes, then it requires quite a bit of reconfiguration, and you may spend a couple hours trying to figure out why a certificate hasn't been issued yet.
That said, I'm unbelievably grateful for the great product (and work!) LetsEncrypt has provided for free. Hope they're able to get their infrastructure back up soon.
Subscribers will be able to optin to short-lived certificates via a certificate profile mechanism being added to our ACME API.
We hope to make short-lived certificates generally available by the end of 2025.
The earliest short-lived certificates we issue may not support IP addresses, but we intend to enable IP address support by the time short-lived certificates reach general availability.
I think I am going to become a fan of shorter certificate lifetimes because as soon as the chuckleheads in the CAB truly break the Internet on the level they are pushing for, the sooner we get to discard the entire PKI dumpster fire.
We're seeing a lot of downstream effects of this at StatusGator. Of course any provider that relies on LetsEncrypt to issue certs (such as Heroku) is affected.
One notable exception is Cloudflare: They famously no longer rely solely on LetsEncrypt.
The browsers and security people have been pushing towards shorter certs, not longer ones. Knowing how to rotate a cert every year, if not shorter, helps when your certificate or any of your parent certs are compromised and require an emergency rotation.
Long expiration times = compromised certs that hang around longer than they should. It's bad.
Note that you can make your own self-signed CA certificate, create any server and client certificates you want signed with that CA cert, and deploy them whenever and wherever you want. Of course you want the root CA private key securely put somewhere and all that stuff.
The only reason it won't work at large without a bit of friction is because your CA cert isn't in the default trusted root store of major browsers (phone and PC). It's easy enough to add it - it does pop up warnings and such on Windows, Android, iOS and hopefully Mac OS X, but they're necessary here.
No, it's not going to let the whole world do TLS with you warning-free without doing some sort of work, but for small scales (the type that Let's Encrypt is often used for anyway) it's fine.
You've always been able to do this. Whether its useful to your clients has always been the problem.
In a practical sense you likely wouldn't like the alternatives, because for most people's usage of the internet there's exactly one authority which matters: the local government, and it's legal system - i.e. most of my necessary use of TLS is for ecommerce. Which means the ultimate authority is "are you a trusted business entity in the local jurisdiction?"
Very few people would have any reason to ever expand the definition beyond this, and less would have the knowledge to do so safely even if we provided the interfaces - i.e. no one knows what safety numbers in Signal mean, if I can even get them to use Signal.
The bigger question that's going unasked: what the hell is the point of an expiration date if it keeps getting shorter? At some point we will refresh the cert every second.
The whole point of the expiration is in case a hacker gets the private key to the cert and can then MITM, they can keep MITMing successfully until the cert the hacker gives to the clients expires (or was revoked by something like OCSP, assuming the client verifies OCSP). A very long expiration is very bad because it means the hacker could keep MITMing for years.
The way things like this work with modern security is ephemeral security tokens. Your program starts and it requests a security token, and it refreshes the token over X time (within 24 hrs). If a hacker gets the token, they can attack using it until 1) you notice and revoke the existing tokens AND sessions, or 2) the token expires (and we assume they for some reason don't have an advanced persistent threat in place).
Nobody puts any emphasis on the fact that 1) you have to NOTICE THE ATTACK AND REVOKE SHIT for any of these expirations to have any impact on security whatsoever, and 2) if they got the private key once, they can probably get it again after it expires, UNLESS YOU NOTICE AND PLUG THE HOLE. If you have nothing in place to notice a hacker has your private key, and if revocation isn't effective, the impact is exactly the same whether expiration is 1 second or 1 year.
How many people are running security scans on their whole stack every day? How many are patching security holes within a week? How many have advanced software designed to find rootkits and other exploits? Or any other measure to detect active attacks? My guess is maybe 0.0001% of you do. So you will never know when they gain access to your certs, so the fast expiration is mostly pointless.
We should be completely reinventing the whole protocol to be a token-based authorization service, because that's where it's headed. And we should be focusing more on mitigating active exploits rather than just hoping nobody ever exploits anything. But that would scare people, or require additional work. So instead we let like 3 companies slowly do whatever they want with the entire web in an uncoordinated way. And because we let them do whatever they want with the web, they keep introducing more failure modes and things get shittier. We are enabling the enshittification happening in front of our eyes.
Hopefully the thundering herd when service is restored doesn't knock things offline again. I know LE designs for huge throughput (something like 3X total outstanding certificates in 24 hours, at one point) and the automated client recommendations for backoff are pretty good, but there will be a lot of manual applications/renewals I'm sure.
That ship has sailed. DNSsec is not liked even a little bit.
Given that control over DNS is how domain validated certs are handed out, it would make a lot of sense to cut out the middle man.
But DNS does not have a good reliable authenticated transport mechanism. I wonder if there was a way to build this that would have worked.
greyface-|7 months ago
Example OpenBSD /etc/acme-client.conf:
CGamesPlay|7 months ago
ninjin|7 months ago
A pity that acme-client(1) does not allow for fallbacks, but I will add a mental note about it being an easy enough patch to contribute if I ever find the time.
Kholin|7 months ago
mixdup|7 months ago
Jedd|7 months ago
This sounds like an easy problem to identify root cause for.
I think I received about 15 'we're disabling email notifications soon' emails over the past several months - one of which was interesting, but none were needed, as I'd originally set this up, per documentation, to auto-renew every 30 days.
Perhaps create a calendar reminder for the short term?
tbrownaw|7 months ago
mattbillenstein|7 months ago
zerof1l|7 months ago
compumike|7 months ago
I wrote this blog post a few weeks ago: "Minimal, cron-ready scripts in Bash, Python, Ruby, Node.js (JavaScript), Go, and Powershell to check when your website's SSL certificate expires." https://heiioncall.com/blog/barebone-scripts-to-check-ssl-ce... which may be helpful if you want to roll your own.
(Disclosure: at Heii On-Call we also offer free SSL certificate expiration monitoring, among other things.)
attentive|7 months ago
You should have it on auto-renewal anyway.
You can grab a cert from ZeroSSL and probably some others.
You can also get 1year cert from aws for like $15 though I'd stick with auto renews.
yjftsjthsd-h|7 months ago
Shouldn't that happen automatically a bit beforehand?
mr_toad|7 months ago
superkuh|7 months ago
rfv6723|7 months ago
It can send alerts to multiple alerting providers.
https://github.com/TwiN/gatus
pgporada|7 months ago
Titan2189|7 months ago
There's no way it's DNS
It was DNS
senectus1|7 months ago
adamcharnock|7 months ago
(Or, as I recently encountered, it can also be a McAfee corporate firewall trying to be helpful by showing a download progress bar in place of an HTTP SSE stream. I was sure that was being caused by MTU, but alas no.)
woleium|7 months ago
jasonthorsness|7 months ago
As they move to shorter-lifetime certs (6 days now https://letsencrypt.org/2025/01/16/6-day-and-ip-certs/?utm_s...) this puts it in the realm of possibility that an incident could impact long-running services.
kenshaw|7 months ago
That said, I'm unbelievably grateful for the great product (and work!) LetsEncrypt has provided for free. Hope they're able to get their infrastructure back up soon.
unknown|7 months ago
[deleted]
pepa65|7 months ago
Subscribers will be able to optin to short-lived certificates via a certificate profile mechanism being added to our ACME API.
We hope to make short-lived certificates generally available by the end of 2025.
The earliest short-lived certificates we issue may not support IP addresses, but we intend to enable IP address support by the time short-lived certificates reach general availability.
sugarpimpdorsey|7 months ago
ocdtrekkie|7 months ago
phillipseamore|7 months ago
https://zerossl.com/ (90 days)
https://www.buypass.com/ (180 days)
colinbartlett|7 months ago
One notable exception is Cloudflare: They famously no longer rely solely on LetsEncrypt.
keysdev|7 months ago
Especially something that needed to be renewed every 90 or is it 40 days now. How about issuing 100 years certificates as a default?
kyrra|7 months ago
https://cloud.google.com/certificate-manager/docs/public-ca-... (EDIT: Google is their own CA, with https://pki.goog/ )
The browsers and security people have been pushing towards shorter certs, not longer ones. Knowing how to rotate a cert every year, if not shorter, helps when your certificate or any of your parent certs are compromised and require an emergency rotation.
RiverCrochet|7 months ago
Note that you can make your own self-signed CA certificate, create any server and client certificates you want signed with that CA cert, and deploy them whenever and wherever you want. Of course you want the root CA private key securely put somewhere and all that stuff.
The only reason it won't work at large without a bit of friction is because your CA cert isn't in the default trusted root store of major browsers (phone and PC). It's easy enough to add it - it does pop up warnings and such on Windows, Android, iOS and hopefully Mac OS X, but they're necessary here.
No, it's not going to let the whole world do TLS with you warning-free without doing some sort of work, but for small scales (the type that Let's Encrypt is often used for anyway) it's fine.
Marsymars|7 months ago
Caddy uses ZeroSSL as a fallback if Let’s Encrypt fails!
dwood_dev|7 months ago
Also, you have days to weeks of slack time for renewals. The only real impact is trying to issue new certs if you are solely dependent on LE.
XorNot|7 months ago
In a practical sense you likely wouldn't like the alternatives, because for most people's usage of the internet there's exactly one authority which matters: the local government, and it's legal system - i.e. most of my necessary use of TLS is for ecommerce. Which means the ultimate authority is "are you a trusted business entity in the local jurisdiction?"
Very few people would have any reason to ever expand the definition beyond this, and less would have the knowledge to do so safely even if we provided the interfaces - i.e. no one knows what safety numbers in Signal mean, if I can even get them to use Signal.
Dylan16807|7 months ago
0xbadcafebee|7 months ago
The whole point of the expiration is in case a hacker gets the private key to the cert and can then MITM, they can keep MITMing successfully until the cert the hacker gives to the clients expires (or was revoked by something like OCSP, assuming the client verifies OCSP). A very long expiration is very bad because it means the hacker could keep MITMing for years.
The way things like this work with modern security is ephemeral security tokens. Your program starts and it requests a security token, and it refreshes the token over X time (within 24 hrs). If a hacker gets the token, they can attack using it until 1) you notice and revoke the existing tokens AND sessions, or 2) the token expires (and we assume they for some reason don't have an advanced persistent threat in place).
Nobody puts any emphasis on the fact that 1) you have to NOTICE THE ATTACK AND REVOKE SHIT for any of these expirations to have any impact on security whatsoever, and 2) if they got the private key once, they can probably get it again after it expires, UNLESS YOU NOTICE AND PLUG THE HOLE. If you have nothing in place to notice a hacker has your private key, and if revocation isn't effective, the impact is exactly the same whether expiration is 1 second or 1 year.
How many people are running security scans on their whole stack every day? How many are patching security holes within a week? How many have advanced software designed to find rootkits and other exploits? Or any other measure to detect active attacks? My guess is maybe 0.0001% of you do. So you will never know when they gain access to your certs, so the fast expiration is mostly pointless.
We should be completely reinventing the whole protocol to be a token-based authorization service, because that's where it's headed. And we should be focusing more on mitigating active exploits rather than just hoping nobody ever exploits anything. But that would scare people, or require additional work. So instead we let like 3 companies slowly do whatever they want with the entire web in an uncoordinated way. And because we let them do whatever they want with the web, they keep introducing more failure modes and things get shittier. We are enabling the enshittification happening in front of our eyes.
bethekidyouwant|7 months ago
jaeh|7 months ago
wnevets|7 months ago
pgporada|7 months ago
unknown|7 months ago
[deleted]
benlivengood|7 months ago
unknown|7 months ago
[deleted]
bravetraveler|7 months ago
If only the same zest applied to probes
adamsiem|7 months ago
phillipseamore|7 months ago
rocqua|7 months ago
But DNS does not have a good reliable authenticated transport mechanism. I wonder if there was a way to build this that would have worked.
skluug|7 months ago
rob_c|7 months ago
burnte|7 months ago
88j88|7 months ago
rob_c|7 months ago