I'm on 12 years of self hosting email and counting. Once every so often, I do end up being blocked, usually by Outlook and once by Yahoo. I'm in their 'sender program' and they still don't actually bother to contact postmaster@, but a few emails is usually enough to unblock the block within 24h.
Agree with a sibling comment that many major providers fail to operate the SPF/DKIM/DMARC tools they insist you do.
Each to their own, but ultimately if we don't hold on to the freedom to operate our own mailservers, it will be taken away through inaction. This means doing some things right: DMARC, DKIM, SPF of course, server maintenance, good password policies and of course IP reputation. The best way I can recommend for IP reputation is to use a dedicated provider or VPS provider that disallows things like VPN endpoints, where it is less likely they'll assign an address with a poor reputation. A good provider might also ask you what you intend to host, and you might be able to discuss IP addresses with them.
I completly agree with you. While Hetzner is my actual neighbor as their headquarters is right in the neighboring town, I still use a server at very small scale provider. I have no problem with my email server. I receive some spam here and there, mostly from Russia. But I immediatly block the according IP addresses for some time.
For years I avoided to use any external service to decide whether its Spam or not. But about 2 years ago I started to rely on some of the external Blocklists.
Till today I have no problem sending Email. Even as I don't use DKIM or DMARC.
To those that still persist, there is a page I found recently that helps you make sure that your outgoing mail is configured correctly: https://appmaildev.com/en/dkim. They generate an e-mail address, you send a mail to them, they do the check and display results (not affiliated, just a happy user).
I've also been self-hosting email for years, and the only deliverability problem I've ever had has been with AT&T. If I try to send something to an AT&T customer, I get an automated "your message has been eaten" notice, and following its directions accomplishes precisely nothing. At this point, I can only guess they're hellbanning the IP block in which my VPS resides, because it does not show up on any public DNSBLs.
Google? No problem. Comcast? No problem. Charter? No problem. AT&T? Problem.
I think we are currently in a phase because of phishing problems. Corporate filters are currently turned up to eleven, but hosting your own server is still very well possible. SPF/DKIM/DMARC helps, but isn't required. You really think the corps that do host their own servers have set that up? It is far more widely spread by enthusiasts that host their own servers.
We also should keep mail for pseudonymous and anonymous user authentication. There are a lot of threats to the free internet right now and user account consolidation is one of it. I agree that people should keep hosting themselves. Someone blocked you? Their loss, let them rant to their internal IT about it. Corporations or institutions that use Google as a provider should be seen with scepticism.
I've been hosting my mail for 20+ years now, with minor issues. I guess I've been lucky.
Reading the comments here makes me incredibly sad. Every answer that tells me to use a provider misses the point. The Internet was created so that there could be many independent nodes, not so that everybody has to rely on one of several blessed providers. I should be able to run my own E-mail.
The real problem is lack of incentives. The big corps do not care about e-mail. It doesn't make money and isn't easily controllable. You can't turn it into a walled garden and lock users in. So, it gets minimal attention, and only defensive measures are developed.
Either we solve the spam problem, or things will get worse. The big tech corps won't solve it for us.
I have also been self-hosting email for 15 years and only had couple of problems at the beginning, mainly until my IP got enough reputation. I have been hosting it on a bare metal Supermicro server in a proper datacenter, though. It has reverse-DNS, SPF, DKIM, TLS, MTA-STS and even DANE with DNSSEC (on a self-hosted BIND but that's another story). It is implemented using Exim, Dovecot, SpamAssasin, DNSBL and Roundcube with OpenLDAP auth. I can recommend this awesome hand-on guide provided by Netherlands Domain Registration Foundation as a basis of a nice configuration https://www.sidn.nl/en/news-and-blogs/hands-on-implementing-...
I had some troubles with IMAP search. I set up CLucene, it was easy and enough for me (no need for Java Lucene). It just took me a long time to figure out why it wouldn't search a domain part of email addresses. It just required to set up the tokenizations in such a way to split words also on @ character, i.e. don't consider a full email address as a word. :P I also had some troubles with OpenLDAP until I finally decided to read the docs and examples there properly. Since then I have been using this setup happily and it appears I will continue to do so! I also share the LDAP with NextCloud btw.
The problem is that collectively we love 'free' (at the point of sale) so much that we'll gladly allow gmail to just walk in and own almost the entirety of the email infrastructure. Then later we realize this gives them the ability to unilaterally make the rules, and we complain. But it's too late.
Also 20+ years, only very minor issues. The spam situation has gotten much better over that time, too. This topic comes up every so often on HN and I feel like the "never self-host, it's guaranteed to be a disaster, you have to use a centralised provider" crowd are just louder. Self-hosting my mail isn't something I think about much or talk about much. It's so obvious to me that it's worth doing, and it's extremely low effort.
I think the problem was this guy was providing email accounts for other people. Other people who probably reused their passwords and had their email in the same DB in the apps they used. That DB was compromised. Hackers got an account to send spam. Then the domain was blacklisted.
He alluded to this in
> At some point your IP range is bound to be banned, either by one asshole IP neighbor sending spam, one of your users being pwned
> The real problem is lack of incentives. The big corps do not care about e-mail. It doesn't make money and isn't easily controllable. You can't turn it into a walled garden and lock users in. So, it gets minimal attention, and only defensive measures are developed.
Is this true? Even for consumer stuff, the only Google product besides search that seems widely popular is Gmail.
For enterprise, Microsoft is pulling in billions from office products. Without digging into their statements I know from my own experience that enterprise email services constitute a core piece of this.
For consumers, switching costs are low (mostly just inconvenient to go through all of your accounts) but not relative to the drawbacks of using corporate email providers which is zero unless you value your privacy.
>the Internet was created so that there could be many independent nodes, not so that everybody has to rely on one of several blessed providers.
any community that grows large enough needs some mechanism to manage trust, this is a universal issue. The early internet was more permissive and less differentiated simply because it was smaller.
The big corps do an alright job at managing spam given the sheer size of the problem, and more importantly you don't just need to solve spam, you need to do so economically, because for your system to stay distributed the nodes need to do the job competitively.
Given that there's intrinsic benefits to managing these things at scale that's not really realistic, in large systems you're always going to have division of labor and stratification for that reason.
1. Require the email receiver to add the sender to their contact list, then put all other emails in "Junk". The UX of this could be quite good nowadays with smartphones and QR codes. The changes to email apps are minimal:
- 1) When a user opens a "mailto:" URL, the email program shows the normal "send email" screen with a "Just add to Contacts" button at the top.
- 2) Email program has a function to show a QR code with the user's "mailto:" URL.
2. Add support to the SMTP protocol to let the receiving server demand that the sender make a small "postage" payment. This will require a privacy-preserving micro-transaction system. I worked on one that doesn't use crypto [0].
Here are some ideas for improving the current system, but not actually solving spam:
3. Create a protocol for automatically reporting emails marked as spam back to their admins. Servers will only accept signed emails. Then servers can do lots of things like:
- Mark sent emails as "Receiver flagged this as spam" and inform the user.
- When an account begins sending spam, rate limit it, disable it pending password reset, or alert the admin.
4. Add SMTP responses that mean "rejecting because your server sends spam" and "rejecting because that user sends spam". Servers can mark sent emails as "Receiver rejected this email" and inform the user.
5. Build a shared spam reporting system that accepts only signed emails and supports searching by email address. Receivers can use it to identify compromised user accounts and reject their email. Senders use it to identify receivers acting in bad faith (reporting ham as spam). A centralized version would be straightforward. A decentralized version would be a challenging project.
6. Add support to the SMTP protocol for reporting rate limits and cooling-off times to email senders. Admins of large shared email systems can feed these metrics into a monitoring system and receive alerts of problems early.
I think the real problem is how bad most e-mail interfaces are (makes you want to use it less) or that there is not a dead simple way to do it other than a provider. It should be easy enough that I can download an app on my phone and be sending messages.
I think matrix might get rid of a lot of the infrastructure barriers, but I don't have much experience with it.
Maybe there needs to be a self-hosting association/union that self-hosters can join? It could advocate for adherence to open standards, and an equal standing for small servers. It could also be a repository of advice for current best practise in small server administration and configuration. Should it be under the auspices of an existing group such as FreedomBox?
The sweet spot for having control over your email while simultaneously minimizing unforseen headaches is to simply own your domain name and point the MX record to whatever hosting provider you want instead of self-hosting a server at home.
Same philosophy for exposing a your personal blog of html files or content like mp4 videos. The sweet spot is to focus on buying a domain name you control. Then let Amazon S3, or Cloudflare, Hezner etc, host your html or mp4 files.
I quit self-hosting email at home over 15 years ago. It's just not something I want to babysit anymore because I have other things to focus on. As long as I control the MX record on my own domain, that's really all that's necessary.
There is also a happy medium. Host your own MX servers but use someone else's SMTP servers. You have complete control over the incoming mail but dodge the filters by using the established business for sending mail.
You are totally missing the simple fact that the number of blessed email providers to choose from is slowly going down. I've seen ISPs with thousands of clients to give up and move the mailboxes to large players simply because their clients' email was ending up in the spam so often that running the support has gotten too expensive.
Yeah, I've been doing exactly this for over 20 years. The only problem I can recall is related to the fact that the hosting provider uses a single SSL cert for the machine that hosts my domain (and many others, presumably), so of course the cert doesn't match my domain name. It's pretty easy to work around, and I only have to deal with it every few years when they do a hardware upgrade, which sometimes means moving my domain to a different machine.
I agree with this assessment, and it's what I do. There is still the single point of failure of losing your domain due to a hostile registrar or mismanagement e.g. allowing registration to lapse.
Ideally there would be some a decentralized permanent domain registry keyed on certs (I know these exist but have not been adopted), or at least a fallback domain you could configure somehow in case you lost control of your main domain.
> simply own your domain name and point the MX record to whatever hosting provider you want
That's not necessarily a sure cure, depending on the hosting provider. RoadRunner (Spectrum / Charter) in the US and Shaw in Canada won't deliver emails from my domain hosted at Runbox.com (or sent directly from the runbox.com domain.) Spectrum's bounce message references an error code that translates to "Spectrum limits the number of concurrent connections from a sender, as well as the total number of connections allowed. Limits vary based on the reputation of the IP address. Reduce your number of connections and try again later."
> point the MX record to whatever hosting provider you want
You can even have a hybrid solution where incoming mail goes directly to your self-hosted server and (some) outgoing mail is relayed through a third party.
"The sweet spot for having control over your email while simultaneously minimizing unforseen headaches is to simply own your domain name"
You would think. The issue no one seems to think about is that you need to make sure to pay for the domain for the duration of your life(at least). Otherwise, as soon as you lose your domain you lose ownership of your email. Any one that has control of the domain has control of your e-mail.
This dawned on me after a place I worked at reactivated the email I used for work when I worked there. It has my name but I have 0 control over it. Lucky for me I never used the email for things other than work so it's not a big deal. It is still bothersome that they can do that and I have no say so on its use.
I like the idea of having a node I can just plug into my network. I run my Urbit on a Mac mini with tailscale (which works great).
The core of what he writes about is correct though, email failed to be truly peer to peer (as imo all non-urbit-like federated systems will) because of the incentives that lead to centralization (spam, difficulty of running nodes, etc.)
We’re suffering the consequences of the local max we’re trapped in currently because of this. The promise of the 90s internet was a bunch of people using decentralized services they controlled - instead we're primarily thin clients connecting to a small handful of powerful ad companies. We're mostly serfs [0] allowed access if we give up our data for ad targeting, follow EULAs nobody reads, and don't say anything the company earls disagrees with.
Controlling your domain/mx is the most valuable thing.
Email reception has not been a problem for me so I enjoy having my mx pointing to my own mail server. It gives me more control and it requires very little maintenance.
If I was going to outsource anything the first choice would be outbound. My email system does everything right for reputation protection. All senders are authenticated on secure connections and the senders are people I trust. Nothing bad gets sent and my static IP is on a reputable server host and is not on any public black lists. I maintain SPF, DKIM etc. If someone decides to block my IP for no reason by accident or on purpose there is very little I can do or care to do anymore.
I have an alternate path for emails setup via a server I host elsewhere ready to go. If I run into a widespread delivery issue due to massive indiscriminate ip blacklisting of my provider I can enable it. That has happened once in 20 years. If delivery gets too hard I will change that policy to send all outgoing emails through a commercial smtp delivery service and let them deal with the problems.
I support the author but let me tell you a counterargument I don’t think he devotes enough to:
Spam is a real issue.
The amount of spam emails which get sent are absurd and likely orders of magnitude more than non-spam. And spammers do a lot to mimic real emails, including just hacking legitimate addresses and adding them to botnets.
Even on gmail, I still get spam sent to my inbox. Fortunately very rarely, but it still happens.
And even if it isn’t bad today, spam has the potential to be much worse in the future with transformer networks and hostile state actors.
And even if it really isn’t that bad and never will be, the big companies and those arguing against self-hosting will claim it is. They don’t want to allow a relative few self-hosted email servers in exchange for much more difficult and less effective spam detection. Forget Gmail and Outlook, why not just use Fastmail or Protonmail?
If you want a legitimate argument for self-hosted emails you need to address the spam. It may be as simple as registering your official email with some organization sponsored by open-source, and all the big companies can trust that one organization. Then the org has to deal with spam registrations but maybe there won’t be much and it will work out. idk much about self-hosting so this org might already exist.
But this article doesn’t mention that org, in fact doesn’t say much at all about spam besides “keep existing spam-prevention because it already works”. But you should at least explain why. Because spam is a legitimate argument for big-co forming an oligarchy that’s not just “so they can make more money”, and it’s the main argument that big-co uses.
It was a huge mistake for email receivers to take on the cost of filtering spam. Of course given the evolution of the internet and email it is easy to see how that mistake happened. Nobody had a crystal ball. But the only solution here is to raise the cost of sending email to the point where spam is no longer profitable.
It seems like one solution is to bcrypt hash (or some similarly expensive algorithm) the email and include the hash in a header. Of course you need to hash per receiver or a spammer can just hash it once and spam away.
The receiving client hashes the email and compares the result with the value in the header and discards emails that don't match.
You'll never get industry buy in though - the FAANG companies don't want to pay that cost for their semi-legitimate email. They prefer to keep that cost externalized.
I believe there have been attempts at something like this, but it clearly never went anywhere.
The address [email protected], which must be used by Australian permanent residents to update their personal details, refuses to accept email messages unless they are from big tech.
Shame on you, Australian Department of Home Affairs.
And shame on Telstra, which provides the service.
---
Remote-MTA: dns; dibp-ibmail2.msng.telstra.com.au
Diagnostic-Code: smtp; 554-mx.msng.telstra.com.au 554 Your access to this mail
system has been rejected due to the sending MTA's poor reputation. If you
believe that this failure is in error, please contact the intended
recipient via alternate means.
E-mail is complicated, sure. But I’ve had it up to here with people who give up running their own server and then go on to vastly exaggerate how infeasible it is, in order to placate their own conscience. It’s not that they’ve gotten tired of doing it, oh no; (they say,) it’s entirely the fault of Google, Microsoft, etc. who’ve made it literally impossible to run your own e-mail server. Except it’s not impossible – lots of us do it, still. And now there’s one fewer of us, so the rest of us have to work that much harder when the next monopolizing standard comes along (BIMI, anyone?). Sure, you don’t owe us anything, but thanks for nothing when making these public rants; you are scaring away people who might still be inclined to help!
Support for client's own domain is currently under works. Our webmail supports PGP and one can use IMAPS/SMTPS or ActiveSync based native email clients too.
All servers self hosted (we run C1/Gentoo!) in our own computing facility in Finland. =)
My fear is that something similar will slowly happen to everything "compute". How long before my bank's website won't let me login if I don't use a computer with secure boot and a browser installed from an app store? McDonalds app on Android won't run on a de-googled or rooted device... At this point one may argue that there will always be computers that you can compile and install your own Linux. Yes, that is true. But just like I am not likely to have un-googled andorid for some apps and googled one for others, the same way it won't be practical to have one computer for some apps and the other one for others. And the one that will win will be the one that lets you login into your bank account, for simple and practical reasons.
I agree with the pains, but the options are not juts Big Tech or self-hosting. There's a myriad of not-big-tech email providers out there, for example there's Posteo, who use open source software and green energy. They are going strong for 13 years now with 400+k accounts.
Sending email out is a royal pain. Trying to deal with a Microsoft ban on my IP even though it’s sparkling clean for several years. DKIM, DMARC, SPF etc all ser up, reverse dns, you name it. Looks like Linode is being blocked as a whole pretty much?
Hate the level of centralization, particularly since there’s still a shit ton of spam still around. Sorry for the rant.
This isn't actually that hard to fix, it's just that for whatever reason, we seem to frequently have this blindspot that we don't seem to have in other industries.
Namely that "do it yourself at home" and "massive oligopolist" aren't the only two options. It's like saying "You can only have hamburgers two ways, cook them yourself or McDonalds."
I do the third and it's been great. I let my paid webhost handle it. (hostdime if you're interested, but I'm sure others do it well also)
I am not a Google employee, but I do work with email anti-spam at scale. There's a lot to critique here but it boils down to three points:
1. Spam filter behavior has changed because spam has increased in volume and sophistication, not because ISPs want to save money, or to eliminate competition. Some techniques that worked well 5 years ago aren't as effective anymore. One of the consequences of this has been a reduction in the value of IP reputation, from a spam signal perspective, particularly for low-volume IPs.
2. IP range reputation does matter. The increase in the value of IP range reputation, as a spam signal, has paralleled the decline in value of low-volume IP reputation. In practice, this means you need to either send enough volume to outweigh the reputation of your IP range (exact quantity varies based on a lot of variables, but as a very rough approximation, 1000 messages a day), or find an IP range with good reputation.
IP range reputation is not easy to assess, sometimes even for email professionals. So you can either gamble with a residential ISP IP, or a VPS IP, or you can find a provider that spends time, effort, and expertise on managing IP range reputation. The practical solution for most senders is the latter. Many of these offer a free tier, and many options are available among providers of all sizes.
3. The filtering behavior reported here is either misunderstood or misrepresented.
First, no, no major ISP (Gmail, Yahoo, Microsoft/Outlook, icloud) is going to permanently block an IP range; filters are designed to be dynamic. In severe, ongoing, high-volume spam scenarios, you could see a 2-week block, maybe occasionally 30 days. But never "one strike".
Mail deletion without a bounce also cam happen, particularly at Microsoft, but again it's almost never seen for legitimate mail - that response is reserved for long-term, severe spam scenarios, where anyone reasonable would agree that a block is warranted. And, again, this is dynamic.
So it looks like OP is either exaggerating, or has been trying to send from IP ranges with unusually bad spam problems.
Have around 100 users on my self-hosted mailserver. Works alright for the most part. Once or twice a year, there are connection issues to small companies with weird settings. I just route those over an external ESP.
Then there is also mxroute.com, which is an indie email provider. He seems to do fine too. Didn't use them yet.
So I think having at least some sending volume is key to running an indie server. You can't do it just for a few mailboxes/users.
I still wouldn't recommend to learn or start with email in 2022. There are better uses of your time.
A person at a company mistakenly created an email list segment (or lack thereof) resulting in an email to the entire email list of hundred of thousands of emails. This combined with inexistent (we were a naive startup without an email specialist role) list hygiene practices meant we were blacklisted by Gmail after some time.
Took a year to get a hold of someone on Gmail's spam team. We found out were on 4+ Gmail blocklists, some of which were ML-based. We couldn't do anything to remove ourselves after we fixed the issues. A $1-2 million revenue channel dried up because we couldn't get out of the Gmail blackhole (short of rebranding completely, rewriting content, and using a different ESP). Fun times.
This thread is quite enlightening. It seems the engineering community predominantly would like to be able to self host email (and presumably other services). By proxy of that I guess that email hosting is not just for one user but maybe also friends and family or some community that you are part of. It's quite clear that managed centralised services accelerated the adoption of the internet, email, etc and the majority of consumers don't really care about self hosting but it also speaks to the fact that we've given up ownership of this aspect of our lives without clear understanding of how it will affect us in the long term. If no one but large corporations own all the services we use then it puts us in a pretty precarious position.
> ... [They use] spam as a scapegoat to nerf deliverability and stifle competition.
Disagree. It was a way too open protocol to begin with. From a time of innocence best suited for places with inherit trust like inside a business. And it's not just spam. Phishing is also a huge issue.
As much as I want to sympathise, Email for the big WWW is unsalvageable IMO. Too many bad actors are out there.
> [Solution:...] * There should be a recourse for legitimate servers
This is the same Big tech story. They want to cut cost, you want a human touch. You can see similar stories here in HN every week. Which is why I think it will never happen.
The topic often comes up. Can't say I share the experience. My servers have never been put on a blacklist in the 7 years they've been running, and one of them operates from my residential DSL connection. Standard postfix+dovecot stack on an Archlinux VPS, I log in once a year to update the packages and make sure there is enough disk space left.
[+] [-] zahllos|3 years ago|reply
Agree with a sibling comment that many major providers fail to operate the SPF/DKIM/DMARC tools they insist you do.
Each to their own, but ultimately if we don't hold on to the freedom to operate our own mailservers, it will be taken away through inaction. This means doing some things right: DMARC, DKIM, SPF of course, server maintenance, good password policies and of course IP reputation. The best way I can recommend for IP reputation is to use a dedicated provider or VPS provider that disallows things like VPN endpoints, where it is less likely they'll assign an address with a poor reputation. A good provider might also ask you what you intend to host, and you might be able to discuss IP addresses with them.
[+] [-] PinguTS|3 years ago|reply
For years I avoided to use any external service to decide whether its Spam or not. But about 2 years ago I started to rely on some of the external Blocklists.
Till today I have no problem sending Email. Even as I don't use DKIM or DMARC.
[+] [-] bornfreddy|3 years ago|reply
To those that still persist, there is a page I found recently that helps you make sure that your outgoing mail is configured correctly: https://appmaildev.com/en/dkim. They generate an e-mail address, you send a mail to them, they do the check and display results (not affiliated, just a happy user).
[+] [-] flyinghamster|3 years ago|reply
Google? No problem. Comcast? No problem. Charter? No problem. AT&T? Problem.
[+] [-] raxxorraxor|3 years ago|reply
We also should keep mail for pseudonymous and anonymous user authentication. There are a lot of threats to the free internet right now and user account consolidation is one of it. I agree that people should keep hosting themselves. Someone blocked you? Their loss, let them rant to their internal IT about it. Corporations or institutions that use Google as a provider should be seen with scepticism.
[+] [-] bullen|3 years ago|reply
To host anything beyond those protocols and/or on more powerful hardware is often counter productive.
The problem is getting the ports opened, you need to fight for that right even if it makes spam worse in the short term.
Fight for external IP, ports and static IP in that order.
[+] [-] LMYahooTFY|3 years ago|reply
Also does it help to use Google/Azure to host with regard to IP reputation with Gmail or Outlook?
[+] [-] yomkippur|3 years ago|reply
[+] [-] 867-5309|3 years ago|reply
this can't exactly be policed
[+] [-] jwr|3 years ago|reply
Reading the comments here makes me incredibly sad. Every answer that tells me to use a provider misses the point. The Internet was created so that there could be many independent nodes, not so that everybody has to rely on one of several blessed providers. I should be able to run my own E-mail.
The real problem is lack of incentives. The big corps do not care about e-mail. It doesn't make money and isn't easily controllable. You can't turn it into a walled garden and lock users in. So, it gets minimal attention, and only defensive measures are developed.
Either we solve the spam problem, or things will get worse. The big tech corps won't solve it for us.
[+] [-] neop1x|3 years ago|reply
I had some troubles with IMAP search. I set up CLucene, it was easy and enough for me (no need for Java Lucene). It just took me a long time to figure out why it wouldn't search a domain part of email addresses. It just required to set up the tokenizations in such a way to split words also on @ character, i.e. don't consider a full email address as a word. :P I also had some troubles with OpenLDAP until I finally decided to read the docs and examples there properly. Since then I have been using this setup happily and it appears I will continue to do so! I also share the LDAP with NextCloud btw.
[+] [-] rootusrootus|3 years ago|reply
[+] [-] jhugo|3 years ago|reply
[+] [-] kagronick|3 years ago|reply
He alluded to this in
> At some point your IP range is bound to be banned, either by one asshole IP neighbor sending spam, one of your users being pwned
[+] [-] LMYahooTFY|3 years ago|reply
Is this true? Even for consumer stuff, the only Google product besides search that seems widely popular is Gmail.
For enterprise, Microsoft is pulling in billions from office products. Without digging into their statements I know from my own experience that enterprise email services constitute a core piece of this.
For consumers, switching costs are low (mostly just inconvenient to go through all of your accounts) but not relative to the drawbacks of using corporate email providers which is zero unless you value your privacy.
[+] [-] Barrin92|3 years ago|reply
any community that grows large enough needs some mechanism to manage trust, this is a universal issue. The early internet was more permissive and less differentiated simply because it was smaller.
The big corps do an alright job at managing spam given the sheer size of the problem, and more importantly you don't just need to solve spam, you need to do so economically, because for your system to stay distributed the nodes need to do the job competitively.
Given that there's intrinsic benefits to managing these things at scale that's not really realistic, in large systems you're always going to have division of labor and stratification for that reason.
[+] [-] 29athrowaway|3 years ago|reply
[+] [-] whiplash451|3 years ago|reply
Email is your identity in many cases (for lack of a better solution).
Email -> identity -> tracking -> advertising revenue
[+] [-] Nelson69|3 years ago|reply
[+] [-] mleonhard|3 years ago|reply
1. Require the email receiver to add the sender to their contact list, then put all other emails in "Junk". The UX of this could be quite good nowadays with smartphones and QR codes. The changes to email apps are minimal:
2. Add support to the SMTP protocol to let the receiving server demand that the sender make a small "postage" payment. This will require a privacy-preserving micro-transaction system. I worked on one that doesn't use crypto [0].Here are some ideas for improving the current system, but not actually solving spam:
3. Create a protocol for automatically reporting emails marked as spam back to their admins. Servers will only accept signed emails. Then servers can do lots of things like:
4. Add SMTP responses that mean "rejecting because your server sends spam" and "rejecting because that user sends spam". Servers can mark sent emails as "Receiver rejected this email" and inform the user.5. Build a shared spam reporting system that accepts only signed emails and supports searching by email address. Receivers can use it to identify compromised user accounts and reject their email. Senders use it to identify receivers acting in bad faith (reporting ham as spam). A centralized version would be straightforward. A decentralized version would be a challenging project.
6. Add support to the SMTP protocol for reporting rate limits and cooling-off times to email senders. Admins of large shared email systems can feed these metrics into a monitoring system and receive alerts of problems early.
[0] https://github.com/mleonhard/hipp
EDIT: tone
[+] [-] zapataband1|3 years ago|reply
I think matrix might get rid of a lot of the infrastructure barriers, but I don't have much experience with it.
[+] [-] femto|3 years ago|reply
[+] [-] jasode|3 years ago|reply
Same philosophy for exposing a your personal blog of html files or content like mp4 videos. The sweet spot is to focus on buying a domain name you control. Then let Amazon S3, or Cloudflare, Hezner etc, host your html or mp4 files.
I quit self-hosting email at home over 15 years ago. It's just not something I want to babysit anymore because I have other things to focus on. As long as I control the MX record on my own domain, that's really all that's necessary.
[+] [-] eikenberry|3 years ago|reply
[+] [-] mordae|3 years ago|reply
It's definitely an anticompetitive practice.
[+] [-] usefulcat|3 years ago|reply
[+] [-] colordrops|3 years ago|reply
Ideally there would be some a decentralized permanent domain registry keyed on certs (I know these exist but have not been adopted), or at least a fallback domain you could configure somehow in case you lost control of your main domain.
[+] [-] redeeman|3 years ago|reply
Dont know about you, but I have setup my mailserver years ago, and outside of regular OS updates, havent had to touch it.
[+] [-] WretchedEarl|3 years ago|reply
That's not necessarily a sure cure, depending on the hosting provider. RoadRunner (Spectrum / Charter) in the US and Shaw in Canada won't deliver emails from my domain hosted at Runbox.com (or sent directly from the runbox.com domain.) Spectrum's bounce message references an error code that translates to "Spectrum limits the number of concurrent connections from a sender, as well as the total number of connections allowed. Limits vary based on the reputation of the IP address. Reduce your number of connections and try again later."
[+] [-] lisper|3 years ago|reply
You can even have a hybrid solution where incoming mail goes directly to your self-hosted server and (some) outgoing mail is relayed through a third party.
[+] [-] WheelsAtLarge|3 years ago|reply
You would think. The issue no one seems to think about is that you need to make sure to pay for the domain for the duration of your life(at least). Otherwise, as soon as you lose your domain you lose ownership of your email. Any one that has control of the domain has control of your e-mail.
This dawned on me after a place I worked at reactivated the email I used for work when I worked there. It has my name but I have 0 control over it. Lucky for me I never used the email for things other than work so it's not a big deal. It is still bothersome that they can do that and I have no say so on its use.
[+] [-] gonehome|3 years ago|reply
I like the idea of having a node I can just plug into my network. I run my Urbit on a Mac mini with tailscale (which works great).
The core of what he writes about is correct though, email failed to be truly peer to peer (as imo all non-urbit-like federated systems will) because of the incentives that lead to centralization (spam, difficulty of running nodes, etc.)
We’re suffering the consequences of the local max we’re trapped in currently because of this. The promise of the 90s internet was a bunch of people using decentralized services they controlled - instead we're primarily thin clients connecting to a small handful of powerful ad companies. We're mostly serfs [0] allowed access if we give up our data for ad targeting, follow EULAs nobody reads, and don't say anything the company earls disagrees with.
[0]: https://zalberico.com/essay/2020/07/14/the-serfs-of-facebook...
[+] [-] shirro|3 years ago|reply
Email reception has not been a problem for me so I enjoy having my mx pointing to my own mail server. It gives me more control and it requires very little maintenance.
If I was going to outsource anything the first choice would be outbound. My email system does everything right for reputation protection. All senders are authenticated on secure connections and the senders are people I trust. Nothing bad gets sent and my static IP is on a reputable server host and is not on any public black lists. I maintain SPF, DKIM etc. If someone decides to block my IP for no reason by accident or on purpose there is very little I can do or care to do anymore.
I have an alternate path for emails setup via a server I host elsewhere ready to go. If I run into a widespread delivery issue due to massive indiscriminate ip blacklisting of my provider I can enable it. That has happened once in 20 years. If delivery gets too hard I will change that policy to send all outgoing emails through a commercial smtp delivery service and let them deal with the problems.
[+] [-] mechanical_bear|3 years ago|reply
[+] [-] armchairhacker|3 years ago|reply
Spam is a real issue.
The amount of spam emails which get sent are absurd and likely orders of magnitude more than non-spam. And spammers do a lot to mimic real emails, including just hacking legitimate addresses and adding them to botnets.
Even on gmail, I still get spam sent to my inbox. Fortunately very rarely, but it still happens.
And even if it isn’t bad today, spam has the potential to be much worse in the future with transformer networks and hostile state actors.
And even if it really isn’t that bad and never will be, the big companies and those arguing against self-hosting will claim it is. They don’t want to allow a relative few self-hosted email servers in exchange for much more difficult and less effective spam detection. Forget Gmail and Outlook, why not just use Fastmail or Protonmail?
If you want a legitimate argument for self-hosted emails you need to address the spam. It may be as simple as registering your official email with some organization sponsored by open-source, and all the big companies can trust that one organization. Then the org has to deal with spam registrations but maybe there won’t be much and it will work out. idk much about self-hosting so this org might already exist.
But this article doesn’t mention that org, in fact doesn’t say much at all about spam besides “keep existing spam-prevention because it already works”. But you should at least explain why. Because spam is a legitimate argument for big-co forming an oligarchy that’s not just “so they can make more money”, and it’s the main argument that big-co uses.
[+] [-] hardwaresofton|3 years ago|reply
https://github.com/foxcpp/maddy
https://blitiri.com.ar/p/chasquid/
These options are much easier to set up, will do things like generate DKIM for you, etc.
I talk about this a lot[0]. There are positively awesome tools for email out there.
[EDIT] - Since I'm repeating myself I've collected all the options into a post[1] I can just link to.
[0]: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
[1]: https://vadosware.io/post/its-never-been-easier-or-harder-to...
[+] [-] jacobsenscott|3 years ago|reply
It seems like one solution is to bcrypt hash (or some similarly expensive algorithm) the email and include the hash in a header. Of course you need to hash per receiver or a spammer can just hash it once and spam away.
The receiving client hashes the email and compares the result with the value in the header and discards emails that don't match.
You'll never get industry buy in though - the FAANG companies don't want to pay that cost for their semi-legitimate email. They prefer to keep that cost externalized.
I believe there have been attempts at something like this, but it clearly never went anywhere.
[+] [-] mastazi|3 years ago|reply
Shame on you, Australian Department of Home Affairs.
And shame on Telstra, which provides the service.
---
Remote-MTA: dns; dibp-ibmail2.msng.telstra.com.au
Diagnostic-Code: smtp; 554-mx.msng.telstra.com.au 554 Your access to this mail system has been rejected due to the sending MTA's poor reputation. If you believe that this failure is in error, please contact the intended recipient via alternate means.
[+] [-] teddyh|3 years ago|reply
[+] [-] intc|3 years ago|reply
Here's EU's JRC-MECSA report on our service: https://mecsa.jrc.ec.europa.eu/en/finderRequest/b5daceffc76e....
Support for client's own domain is currently under works. Our webmail supports PGP and one can use IMAPS/SMTPS or ActiveSync based native email clients too.
All servers self hosted (we run C1/Gentoo!) in our own computing facility in Finland. =)
[+] [-] capdeck|3 years ago|reply
[+] [-] npteljes|3 years ago|reply
https://en.wikipedia.org/wiki/Posteo
[+] [-] gingerlime|3 years ago|reply
Hate the level of centralization, particularly since there’s still a shit ton of spam still around. Sorry for the rant.
https://docs.microsoft.com/en-us/answers/questions/674558/55...
[+] [-] jrm4|3 years ago|reply
Namely that "do it yourself at home" and "massive oligopolist" aren't the only two options. It's like saying "You can only have hamburgers two ways, cook them yourself or McDonalds."
I do the third and it's been great. I let my paid webhost handle it. (hostdime if you're interested, but I'm sure others do it well also)
[+] [-] massaman_yams|3 years ago|reply
1. Spam filter behavior has changed because spam has increased in volume and sophistication, not because ISPs want to save money, or to eliminate competition. Some techniques that worked well 5 years ago aren't as effective anymore. One of the consequences of this has been a reduction in the value of IP reputation, from a spam signal perspective, particularly for low-volume IPs.
2. IP range reputation does matter. The increase in the value of IP range reputation, as a spam signal, has paralleled the decline in value of low-volume IP reputation. In practice, this means you need to either send enough volume to outweigh the reputation of your IP range (exact quantity varies based on a lot of variables, but as a very rough approximation, 1000 messages a day), or find an IP range with good reputation.
IP range reputation is not easy to assess, sometimes even for email professionals. So you can either gamble with a residential ISP IP, or a VPS IP, or you can find a provider that spends time, effort, and expertise on managing IP range reputation. The practical solution for most senders is the latter. Many of these offer a free tier, and many options are available among providers of all sizes.
3. The filtering behavior reported here is either misunderstood or misrepresented. First, no, no major ISP (Gmail, Yahoo, Microsoft/Outlook, icloud) is going to permanently block an IP range; filters are designed to be dynamic. In severe, ongoing, high-volume spam scenarios, you could see a 2-week block, maybe occasionally 30 days. But never "one strike".
Mail deletion without a bounce also cam happen, particularly at Microsoft, but again it's almost never seen for legitimate mail - that response is reserved for long-term, severe spam scenarios, where anyone reasonable would agree that a block is warranted. And, again, this is dynamic.
So it looks like OP is either exaggerating, or has been trying to send from IP ranges with unusually bad spam problems.
[+] [-] m3nu|3 years ago|reply
Then there is also mxroute.com, which is an indie email provider. He seems to do fine too. Didn't use them yet.
So I think having at least some sending volume is key to running an indie server. You can't do it just for a few mailboxes/users.
I still wouldn't recommend to learn or start with email in 2022. There are better uses of your time.
[+] [-] lxchase|3 years ago|reply
A person at a company mistakenly created an email list segment (or lack thereof) resulting in an email to the entire email list of hundred of thousands of emails. This combined with inexistent (we were a naive startup without an email specialist role) list hygiene practices meant we were blacklisted by Gmail after some time.
Took a year to get a hold of someone on Gmail's spam team. We found out were on 4+ Gmail blocklists, some of which were ML-based. We couldn't do anything to remove ourselves after we fixed the issues. A $1-2 million revenue channel dried up because we couldn't get out of the Gmail blackhole (short of rebranding completely, rewriting content, and using a different ESP). Fun times.
[+] [-] asim|3 years ago|reply
[+] [-] yonixw|3 years ago|reply
Disagree. It was a way too open protocol to begin with. From a time of innocence best suited for places with inherit trust like inside a business. And it's not just spam. Phishing is also a huge issue.
As much as I want to sympathise, Email for the big WWW is unsalvageable IMO. Too many bad actors are out there.
> [Solution:...] * There should be a recourse for legitimate servers
This is the same Big tech story. They want to cut cost, you want a human touch. You can see similar stories here in HN every week. Which is why I think it will never happen.
[+] [-] toun|3 years ago|reply