I run a few moderately sized corporate e-mail systems. SPF, DKIM, DMARC are all necessities, but unfortunately they don't guarantee delivery. We have much better success sending e-mail out via Mailgun or Sendgrid. Since Verizon swallowed up Yahoo Mail and @aol.com, it's very easy to get on their shadow ban list, and nearly impossible to get off. Good luck telling all the employees they can't email @yahoo.com and @aol.com for at least 2 weeks.
I see e-mail as another Internet service that is slowly centralizing, and if you operate outside of the big tech companies- for example, if you don't use Google/Yahoo/M365 e-mail, you're probably going to be in for a headache, even if you run perfect e-mail infrastructure.
Not centralizing -- these providers are colluding in a oligopoly. I think there is an argument to be made that these large providers are creating a situation where they set up barriers of trust that only allow email from each other to pass. Small independent senders are locked out simply by not being part of the trusted club, even if their email is valid and passes all of the standard hurdles (SPF, DKIM, etc).
I actively work with and set up ~100 domains in SendGrid, and Yahoo and AOL are still, have have been, THE BANE OF MY EXISTENCE since I started working at this job 8 years ago. It's always been a black hole into any type of support. I always get the same answer back... "If you don't want to be seen as a spam bot, don't act like one".... OK, I'll just tell my client they can't send their 20k emails when they want (with a non insignificant going to Yahoo/AOL still). Honestly I don't know what else we can do better. We don't send newsletters and such, we only send transnational emails.
Which is why it is so vital for anyone with at least minimal tech know-how to run their own email infrastructure. The Internet is only a meaningful concept if it consists of peers running standard decentralized protocols.
If all we had was a few giant corporations with proprietary apps and proprietary protocols, that's not the Internet.
As I post every time on these email discussions, the difficulty of running your own email infrastructure is vastly overstated on HN. The more people do it, the better off the Internet world is for all of us.
Even using SendGrid, it will be a week or two and a few hundred emails to 'warm up' one of their mailserver's IP addresses and get reliable delivery. Like you, I found VZ to be the worst at trusting new servers.
The days of firing off an email from any ol' SMTP server are completely gone.
I feel you 100%. The timing of this post is near perfect with what is happening in my life. I run my own web server with email hosting for clients. I’ve spent the last few years doing everything I can on the deliverability side but with no success. All the checkboxes have been checked but I still hit the spam folders. Just yesterday I gave up. I migrated all my clients to Office 365 and I did it knowing I was killing the soul of the Internet. But I was just too tired of dealing with clients who don’t understand why their emails end up in spam folders all the time. These clients also don’t have the same “values” when it comes to decentralization. They run small businesses and cannot afford this spam issue. I feel like there should be laws againsts what the tech giants are doing. Their spam filter rules are killing competition and the free internet.
Sadly I've found this to be the case both with perfectly configured personal email servers and when using privacy-preserving email providers like Tutanota. Do you think there's anything that can be done to counter this centralizing force?
Same. I run a 10/10 (mail-tester.com) mail server for myself personally on a VPS from Digital Ocean. Its IP address is on Charter's permanent blocklist and has been for years so I cannot email my mom. I set up a SMTP relay via a free-tier sendgrid and it worked for a while but then that relay got added to spamlists and I could basically only email my mom. Presumably the paid sendgrid setups allow a bit more resilience against getting on those lists.
I moved my mail away from Google last year, and can no longer accept people's calendar invites. It seems that the "Yes"/"No"/"Maybe" links in the email body of a calendar invite cause Google's (or whoever's) servers to spoof an email in my name, which predictably fails if my SPF is set strictly. I have not been able to find a good workaround for this.
The sad reality is that you should expect to be delivered into the recipients' spam boxes until you have managed to convince Google/Microsoft/Yahoo that you're not a spammer.
As far as I can tell, it simply is not possible to buy a new domain, pay for a email service and have mails delivered straight into people's inboxes. After all, if you would be able to do that then so could the spammers.
Ramp up volumes slowly. Make sure the recipients actually want the mails you send, and for all that is holy, educate your users that the "Junk" and "Trashcan" buttons are two very different things. People who repeatedly mark your mails as spam need to be blacklisted, no matter if they're paying customers. Period.
Oh, and handle abuse complaints right away. Don't be afraid to tell people they're not supposed to mark your mails as spam and to explain the consequences.
Also, get a private IP from your email service. Don't use their public ips. Or you will cry as the IP they have you on will randomly get put on SpamCop.
Even with my private IPs I've had random anti-spam services just decide to list all Sendgrid IPs.
Just this last weekend I started moving clients off of the mail hosting services I've managed for years and onto Fastmail. I have self-hosted my email since around 2005, and had my first email address around 1994 or thereabouts. It has never been more broken than it is now, and I mean that without even the slightest hyperbole.
The first RBL showed up around 1998, operated by Paul Vixie of DNS fame. I was using a small ISP in the East Bay at the time and they hated the MAPS RBL, especially the idea that they had to deal with some third party to resolve mail transport issues. But, from 1998 until the early 2000s, RBLs proliferated and lots of mail services subscribed to one or more. The one thing they had going for them was that they were easy to query, so it took no effort to find out if you were on the list, and if you were, there was usually a living, breathing human being somewhere that you could contact to get the matter resolved. It was a pain in the butt, but it could be handled.
During this time, the responsibility for ensuring that a message was received shifted from the recipient to the sender. Beleaguered systems administrators soon found themselves in a situation where they were supposed to be responsible for both ensuring that no spam reached their customers and that all of their customers' email reached the intended recipients.
Gmail came along and decided that, because they were operating "at scale", they didn't need to play in the same ecosystem. Over the years, ensuring that a message lands in a Gmail user's inbox has turned in to an infuriating game of trial-and-error. Gmail can do this because they now manage between 40% and 60% of the internet's email traffic.
AT&T/SBCGlobal/Yahoo/whoever they are now seem to have recently penalized all of Linode's and DigitalOcean's IP space. I deployed several mail exchanges and didn't have any luck reaching any addresses managed by AT&T's network. And, again, there's nobody I can kibbutz with to resolve it.
AOL has been such a hot tire fire that I ended up blackholing any outbound traffic to them. I know that sounds drastic, but anytime a single AOL user clicked the "spam" button on an email from one of my customers -- who weren't spamming, newslettering, or anything else remotely skeezy -- it would generate an automated complaint from AOL to Linode, and Linode would threaten to suspend my account. I'd have to dig the relevant traffic out of the logs and respond back to Linode with a polite "AOL's full of shit, please stop listening to them". I explained the situation to the few people that were impacted by it and everybody got on with their lives.
Microsoft's outlookprotection.com filter has been a gigantic pain recently too. It's annoyingly capricious and, again, the tools just aren't there to resolve it.
Email delivery has been a bit tricky for several years, but the last year especially it has become impossible for small services. You have SPF, DKIM, DMARC, great, but it turns out that Gmail also dings you if you don't have ipv6 records arranged right or if you aren't transporting mail over ssl or or or or.
Email was designed as a cooperative system but the BigCos have carved it up and are working hard to ensure that if a message doesn't come from one of their networks, then it's immediately suspicious.
Sendgrid isn't much better in this regard. My network received a flood of spam from them, with legitimate traffic mixed in. I couldn't block them without complaints and I couldn't not block them without complaints. I wondered if I was the only, so I signed up for their service and routed some mail through them for a few days to see what it was like as one of their subscribers. Turns out that Comcast and half a dozen other service providers hate them just as much and deliverability was around 79%.
The things you and others are experiencing, and that I experienced, are going to keep getting worse. The bigger networks are going to keep squeezing customers out of the smaller ones, breaking email bit by bit in the process. I hope Fastmail is able to grow quickly enough to keep sitting at the table.
One of the most important things I learned about SPF records was that the entire record had to be under 256 bytes. We were sending out emails from multiple servers on one domain, and were having so many deliverability issues because the SPF record had too many entries in it and would be seen as invalid by some spam firewalls. Once I cleaned up the record and split some email out to subdomains our delivery rates went way up. But of course as others mentioned SPF is just the beginning. DKIM, DMARC, reverse records, etc are all very important as well.
> One of the most important things I learned about SPF records was that the entire record had to be under 256 bytes.
It sounds like this is a case of buggy implementations. You can have TXT and even SPF records longer than 255 bytes. The fundamental issue is that TXT records are composed of segments of up to 255 bytes each, where each segment has an 8-bit length header. As Resource Records (RR) individually, as well as DNS packets as a whole, have a maximum length of 65535 (because of 16-bit length headers), DNS supports TXT RRs of approx. 256 segments, 255 bytes each.
It's not too surprising implementations might be buggy in this respect. It's a rather esoteric aspect of the TXT RR specification. (If only because few people, including developers of DNS-related tools, bother actually reading the specification!) And unfortunately neither the popular rfc4408-tests.yml nor rfc7208-tests.yml test suites seem to have a test case for long TXT records. Even if a test had been included it likely wouldn't have caught many violators as it's actually a matter of the low-level DNS library providing the proper interface(s), many DNS libraries are deficient in how well they expose DNS record parsing or packet injection interfaces, and most SPF libraries would have likely integrated the test suite at a higher level that didn't involve actually making queries using a low-level DNS library, which can be difficult to do.
That said, RFC 4408 says that,
> SPF implementations MUST limit the number of mechanisms and modifiers that do DNS lookups to at most 10 per SPF check, including any lookups caused by the use of the "include" mechanism or the "redirect" modifier.
IME experience this limit is rather low and many, many SPF policies require more than 10 queries. (And if you need SPF policies larger than 255 bytes, you're probably skirting this or other similar limits.) When I was writing and deploying my own implementation I very quickly realized I needed to increase the minimum maximums, but choosing that value is a total crap shoot.
SPF records large than 256 bytes are pretty common. See e. g. aspmx.sailthru.com (1st what found right now). You just have to split text into multiple strings, but don't forget to add a space after or before a derective, because strings are joined without an extra spaces: https://tools.ietf.org/html/rfc7208#section-3.3
May be there are buggy implementations which don't understand this, but doubt they are common.
You can fix this by putting a single SPF entry for each envelope/returnpath domain. SPF scales smoothly that way without any need to overload a single top level record with entries. This will pass DMARC as well.
"We set a specific header in the email called Return-Path with all emails sent through our email service. If you take a peek at a raw email sent over OhMySMTP you’ll find that the return path ends in @mailer.ohmysmtp.com, and if you look this up in the DNS system, you’ll find an SPF record that points to our server IP address."
While this will mean that and mail sent will pass SPF, it doesn't play well with DMARC. The SPF pass domain won't align with the From header domain, so the SPF authentication will be ignored.
"But SPF alone doesn’t completely solve spam, we also need DKIM, and to a lesser extent DMARC. More on those later!"
Realistically, SPF doesn't do anything to solve spam, nor does DKIM or DMARC. All of these technologies address spoofing and give a greater confidence of the sender responsible for the email which can then be used in reputation engines.
Passing SPF/DKIM/DMARC say nothing about the message contents.
SPF, DKIM, DMARC are must have for today mail and yet many domain owners fail to make even the first and easiest step - publish a valid SPF record. Some mistakes I commonly see (don't do this):
* no space between directives: "v=spf1 ip4:192.0.2.0/24 include:example.org-all"
* space inside a directive: "v=spf1 ip4:192. 0.2.0/24 -all"
* bad mechanism: "v=spf1 ipv4:192.0.2.1 -all"
* no mechanism: "v=spf1 192.0.2.1 example.com"
* = instead of : "v=spf1 include=example.com"
* Unicode, mostly dashes (but I've seen zero-width spaces too): "v=spf1 —all"
* Two SPF records for the same domain: "v=spf1 mx:example.com -all", "v=spd1 include:example.net ?all"
Don't know where it coming from, but IMHO we see a generation of developers/admins raised by Google search: you can type a search query with mistakes and still get what you want. Or may be web browsers to blame - you can screw up HTML/CSS in dozens ways and they'll render content anyway. And they start to think that all IT systems forgive small mistakes and typos. They just don't get that sometimes you have to read RFC (or at least simplified how-to) and follow it exactly.
What you describe, adding the DNS TXT record for DKIM, is just the icing on the cake. It's way more work and complex at postfix (+rspamd to sign or whatever) level. Suggesting that you can just,
>Hostgator -> cPanel -> Email Authentication
...is ignoring 99% of the complexity and long term maintainance that DKIM implies.
And why bother? It's not like the mega-corp walled gardens care if you have the perfect mailserver implementation. They'll mark you as spam or blacklist you anyway, because they can, because it's easier, because it's a job someone's doing for profit and you don't effect that.
No. An SPF record is enough for determining if mail is legitimately from it's real mailserver. DKIM is cargo cult nonsense for most cases. A sacrifice to the google/microsoft/apple gods to hopefully earn their favor, but one that is almost always ignored anway.
I've always wondered - what could have been done to the original specification for email that would have avoided this hellscape of spam and antispam measures?
Or does spam appear in any popular messaging system anyways because it is such a strong selective pressure?
People who send spam never think it is spam. They often think their UBE is somehow different from all the other stuff. They claim that the recipients want this bit of email, even if those same recipients complain about getting spam.
Apparently, having multiple A Records can cause problems with spf records. The founder of hanami.run, an email forwarding service, showed me how to fix mine.
[+] [-] _nickwhite|5 years ago|reply
I see e-mail as another Internet service that is slowly centralizing, and if you operate outside of the big tech companies- for example, if you don't use Google/Yahoo/M365 e-mail, you're probably going to be in for a headache, even if you run perfect e-mail infrastructure.
[+] [-] throwawayboise|5 years ago|reply
[+] [-] bobflorian|5 years ago|reply
[+] [-] jjav|5 years ago|reply
If all we had was a few giant corporations with proprietary apps and proprietary protocols, that's not the Internet.
As I post every time on these email discussions, the difficulty of running your own email infrastructure is vastly overstated on HN. The more people do it, the better off the Internet world is for all of us.
[+] [-] briHass|5 years ago|reply
Even using SendGrid, it will be a week or two and a few hundred emails to 'warm up' one of their mailserver's IP addresses and get reliable delivery. Like you, I found VZ to be the worst at trusting new servers.
The days of firing off an email from any ol' SMTP server are completely gone.
[+] [-] human|5 years ago|reply
[+] [-] sebmellen|5 years ago|reply
[+] [-] acidburnNSA|5 years ago|reply
[+] [-] endgame|5 years ago|reply
[+] [-] fogihujy|5 years ago|reply
As far as I can tell, it simply is not possible to buy a new domain, pay for a email service and have mails delivered straight into people's inboxes. After all, if you would be able to do that then so could the spammers.
Ramp up volumes slowly. Make sure the recipients actually want the mails you send, and for all that is holy, educate your users that the "Junk" and "Trashcan" buttons are two very different things. People who repeatedly mark your mails as spam need to be blacklisted, no matter if they're paying customers. Period.
Oh, and handle abuse complaints right away. Don't be afraid to tell people they're not supposed to mark your mails as spam and to explain the consequences.
[+] [-] karmicthreat|5 years ago|reply
Even with my private IPs I've had random anti-spam services just decide to list all Sendgrid IPs.
[+] [-] thaumaturgy|5 years ago|reply
The first RBL showed up around 1998, operated by Paul Vixie of DNS fame. I was using a small ISP in the East Bay at the time and they hated the MAPS RBL, especially the idea that they had to deal with some third party to resolve mail transport issues. But, from 1998 until the early 2000s, RBLs proliferated and lots of mail services subscribed to one or more. The one thing they had going for them was that they were easy to query, so it took no effort to find out if you were on the list, and if you were, there was usually a living, breathing human being somewhere that you could contact to get the matter resolved. It was a pain in the butt, but it could be handled.
During this time, the responsibility for ensuring that a message was received shifted from the recipient to the sender. Beleaguered systems administrators soon found themselves in a situation where they were supposed to be responsible for both ensuring that no spam reached their customers and that all of their customers' email reached the intended recipients.
Gmail came along and decided that, because they were operating "at scale", they didn't need to play in the same ecosystem. Over the years, ensuring that a message lands in a Gmail user's inbox has turned in to an infuriating game of trial-and-error. Gmail can do this because they now manage between 40% and 60% of the internet's email traffic.
AT&T/SBCGlobal/Yahoo/whoever they are now seem to have recently penalized all of Linode's and DigitalOcean's IP space. I deployed several mail exchanges and didn't have any luck reaching any addresses managed by AT&T's network. And, again, there's nobody I can kibbutz with to resolve it.
AOL has been such a hot tire fire that I ended up blackholing any outbound traffic to them. I know that sounds drastic, but anytime a single AOL user clicked the "spam" button on an email from one of my customers -- who weren't spamming, newslettering, or anything else remotely skeezy -- it would generate an automated complaint from AOL to Linode, and Linode would threaten to suspend my account. I'd have to dig the relevant traffic out of the logs and respond back to Linode with a polite "AOL's full of shit, please stop listening to them". I explained the situation to the few people that were impacted by it and everybody got on with their lives.
Microsoft's outlookprotection.com filter has been a gigantic pain recently too. It's annoyingly capricious and, again, the tools just aren't there to resolve it.
Email delivery has been a bit tricky for several years, but the last year especially it has become impossible for small services. You have SPF, DKIM, DMARC, great, but it turns out that Gmail also dings you if you don't have ipv6 records arranged right or if you aren't transporting mail over ssl or or or or.
Email was designed as a cooperative system but the BigCos have carved it up and are working hard to ensure that if a message doesn't come from one of their networks, then it's immediately suspicious.
Sendgrid isn't much better in this regard. My network received a flood of spam from them, with legitimate traffic mixed in. I couldn't block them without complaints and I couldn't not block them without complaints. I wondered if I was the only, so I signed up for their service and routed some mail through them for a few days to see what it was like as one of their subscribers. Turns out that Comcast and half a dozen other service providers hate them just as much and deliverability was around 79%.
The things you and others are experiencing, and that I experienced, are going to keep getting worse. The bigger networks are going to keep squeezing customers out of the smaller ones, breaking email bit by bit in the process. I hope Fastmail is able to grow quickly enough to keep sitting at the table.
[+] [-] dillutedfixer|5 years ago|reply
[+] [-] wahern|5 years ago|reply
It sounds like this is a case of buggy implementations. You can have TXT and even SPF records longer than 255 bytes. The fundamental issue is that TXT records are composed of segments of up to 255 bytes each, where each segment has an 8-bit length header. As Resource Records (RR) individually, as well as DNS packets as a whole, have a maximum length of 65535 (because of 16-bit length headers), DNS supports TXT RRs of approx. 256 segments, 255 bytes each.
Both RFC 4408 (https://tools.ietf.org/html/rfc4408#section-3.1.3) and it's successor, RFC 7208 https://tools.ietf.org/html/rfc7208#section-3.3, require SPF implementations to concatenate the segments and process them as a single string. This means you should be able to publish a single SPF record of approx. 65025 bytes (256 x 255).
It's not too surprising implementations might be buggy in this respect. It's a rather esoteric aspect of the TXT RR specification. (If only because few people, including developers of DNS-related tools, bother actually reading the specification!) And unfortunately neither the popular rfc4408-tests.yml nor rfc7208-tests.yml test suites seem to have a test case for long TXT records. Even if a test had been included it likely wouldn't have caught many violators as it's actually a matter of the low-level DNS library providing the proper interface(s), many DNS libraries are deficient in how well they expose DNS record parsing or packet injection interfaces, and most SPF libraries would have likely integrated the test suite at a higher level that didn't involve actually making queries using a low-level DNS library, which can be difficult to do.
That said, RFC 4408 says that,
> SPF implementations MUST limit the number of mechanisms and modifiers that do DNS lookups to at most 10 per SPF check, including any lookups caused by the use of the "include" mechanism or the "redirect" modifier.
IME experience this limit is rather low and many, many SPF policies require more than 10 queries. (And if you need SPF policies larger than 255 bytes, you're probably skirting this or other similar limits.) When I was writing and deploying my own implementation I very quickly realized I needed to increase the minimum maximums, but choosing that value is a total crap shoot.
[+] [-] citrin_ru|5 years ago|reply
[+] [-] annoyingnoob|5 years ago|reply
[+] [-] brightball|5 years ago|reply
Very common issue.
[+] [-] marcbradshaw|5 years ago|reply
While this will mean that and mail sent will pass SPF, it doesn't play well with DMARC. The SPF pass domain won't align with the From header domain, so the SPF authentication will be ignored.
"But SPF alone doesn’t completely solve spam, we also need DKIM, and to a lesser extent DMARC. More on those later!"
Realistically, SPF doesn't do anything to solve spam, nor does DKIM or DMARC. All of these technologies address spoofing and give a greater confidence of the sender responsible for the email which can then be used in reputation engines.
Passing SPF/DKIM/DMARC say nothing about the message contents.
[+] [-] citrin_ru|5 years ago|reply
[+] [-] XCSme|5 years ago|reply
I recently wrote a short article on those, a tldr for setting up basic email authentication and anti-spam: https://www.usertrack.net/blog/stop-others-use-your-domain-e...
[+] [-] superkuh|5 years ago|reply
>Hostgator -> cPanel -> Email Authentication
...is ignoring 99% of the complexity and long term maintainance that DKIM implies.
And why bother? It's not like the mega-corp walled gardens care if you have the perfect mailserver implementation. They'll mark you as spam or blacklist you anyway, because they can, because it's easier, because it's a job someone's doing for profit and you don't effect that.
No. An SPF record is enough for determining if mail is legitimately from it's real mailserver. DKIM is cargo cult nonsense for most cases. A sacrifice to the google/microsoft/apple gods to hopefully earn their favor, but one that is almost always ignored anway.
[+] [-] unknown|5 years ago|reply
[deleted]
[+] [-] superasn|5 years ago|reply
[+] [-] smiley1437|5 years ago|reply
Or does spam appear in any popular messaging system anyways because it is such a strong selective pressure?
[+] [-] DanBC|5 years ago|reply
[+] [-] m-p-3|5 years ago|reply
https://serverfault.com/a/293123
[+] [-] unknown|5 years ago|reply
[deleted]
[+] [-] unknown|5 years ago|reply
[deleted]
[+] [-] codazoda|5 years ago|reply
[+] [-] qu1mby|5 years ago|reply