Running an email server became a real pain about 4 years ago. Until then, we could run our own exchange server, control everything and it was awesome. Then came the email security products from the large companies.
It became a self fulfilling prophecy. Our email would trigger a rule in big co's third party security app, which would then report to a centralized rule breaker clearing house automatically. Clients couldn't receive our emails, we would dive in, submit appeal to clearing house, get clear and a week later do it all again.
Everything was configured properly on our end. We passed all the online validation engines. IPs were our own personally owned block and pristine.
It became too much.
Switched to office365 and all problems magically disappeared. Sent emails to same big cos with third party email security and haven't had a single issue.
At my last job at an email security provider, this was a huge pain in the ass. We would provision new mail servers, and because the new servers' IP reputation to Office365, Google, etc. was poor initially, someone (until it became a shell script) had to spend the better part of a week taking the new server out of rotation, putting it back in, and so forth, so not too many of our client's emails got stuck in the new server's reputation blackhole.
I was there for 10 years, and this was only a problem in the final ~2 years of my employment, which roughly lines up with your 4 years remark.
Yup. As more and more domains centralize email in the handful of mega-corp hosted solutions the hosts have less and less reason to care about accepting mail from outside the walled gardens.
It's not about setting up perfect signed email and all the new techs. It's not about having the same clean IP for a decade. It's just the same old network effect combined with profit motive. The obvious change in the last few years (I've run a mailserver for 9) is managerial not technical.
I know I've said this before here but.. it goes the other way too.
People make a big deal of SPF/DKIM etc. I think you should, for the sake of best practice. But you can set yourself up with gsuite or Office 365 and pay no attention to these things, and you can fully expect mail to get through to everyone. You can even setup a broken SPF record that disallows office 365 email and for the most part, everyone will still get your email.
Yep, you can do everything outlined in the article and have an IP that has not send spam in 10+ years and gmail will still spambox every single one of your mails.
However if a domain signs up for Google Apps, magically you get whitelisted, even if you are handling your own email, funny thing that.
I guess Google's customers are just more trustworthy...
A middle-ground that can be worth considering is mailgun.org or something similar - you can host the rest of the stack yourself and just use them for the last hop of outbound SMTP from your own MTA.
You don't have to throw out the baby with the bathwater (:
I disagree. The last couple places I've worked at have all had their mail hosted internally, meaning on a dedicated IP from one of the big ISP's on a business plan.
No real issues. Sure, you make it on a spam blacklist once in a while, but at least you know when it's going to be fixed, instead of dealing with zero response (or denial) from a big hosted mail provider.
Besides, not all of "our email" comes out of our office, our CRM and ERP products send mail on our behalf, and they have problems delivering mail more than we do.
Running an email server has been a pain forEVER. Running one's own personal mail server is a terrible idea unless you are a professional sysadmin with time to kill.
a) a docker container (or other standardized, easy to deploy technology) with a "mail server in a box implementing best practices"
b) a test for the stuff that Docker can't cover like DNS config, ideally with scripts to set it up on the most popular providers and clear copy&paste instructions for the rest.
People shouldn't need to fully understand DKIM, SPF, etc. - they should just be able to click a button, copy&paste generated DNS records, and be up and running.
Otherwise, commercial providers are just the only option that makes operational sense for most cases, _even if you know how to do it right_, because doing everything right manually step by step is still to much work to be worth it in many cases.
Agreed. A few years back when moving my personal domain from a failing provider I considered running my own email server on Digital Ocean; they had some great tutorials on setting up various parts and I followed a HN recommended tutorial. I was already happy setting up LAMP with a nginx reverse proxy, how hard could it be - I got it setup, and realised two things: Keeping everything running without getting compromised was going to take way too long compared to the benefit. Two, Google and MS wouldn't accept my email - not even to myself when I whitelisted it - it's a crap shoot.
Email has serious economies of scale and keeping such a critical system operational, on a small scale, was going to be some deal of stress.
So I just went with my hosting providers Plesk-fronted setup, which matches your para.3 description.
* Use an outbound suppression list to ensure that unsubscribe requests or unwanted mail complaints have lasting effect.
* Article touches on DMARC reporting, for which I suggest Postmark's free monitoring tool at https://dmarc.postmarkapp.com/
* From time to time run your domain name(s) and IP address(es) through a panel of reputation services. Check out any red flags (some of them matter, some don't).
And for content:
* Re-evaluate email templates. Many out there are hot garbage, practically inviting users to hit the junk button, which just hurts sender reputation. Ensure there's always a plaintext part. Don't use tracking/opening pixels. Write validating HTML. Minimise size. Don't bury the unsubscribe link, put it at the top, and make sure it clearly works without hostile barriers like "are you sure" pages or sign-in.
* In the very first paragraph, and without scrolling on a mobile, we must explain, honestly and concisely, why we are contacting someone, and why they might want to read the rest, because we have no entitlement to someone's attention.
* Run all mailers through a spam scoring tool as part of CI/CD, because a high score is a bug.
The volume of spam is mind boggling. When we first implemented DMARC / DKIM etc our legit emails were 3% or so of all outbound mail per the reporting we got back! We have a somewhat trusted domain.
Some mail delivery systems start to notice that your domain name is being used as part of a lot of spam / bogus emails, so you can have perfect IP history / no spam and STILL start triggering some random filters (no big players but smaller protection product filters).
So the game must be absolutely never ending for everyone and the inbox a valuable target - especially now that unsolicited phone calls really do seem to get ignored these days - I feel like spammers killed the golden goose on phone calls and the telcos let them.
I will say DKIM / DMARC is working well, except google (which we now use for outbound) gives us transient SPF errors even though we are 100% using their IPs. Not sure why that is (ie, SPF failure on an IP that should clear)
Tangentially related but I'm having some issues related to email beeing refused by Gmail.
In my organization we have ~10 linux VM and they are all configured to send email to an exim MTA. For example monitoring stuff. The system mail name is set to vmName.srv.domain.name (domain.name is replaced by our actual domain and this resolve to a local address) so it's quite common to receive mail from [email protected]. This cause no issue when it's delivered to the local Cyrus server but when it's transfered to a Gmail address for some reason Gmail reject the email because the headers are bad.
I am considering rewriting the address to [email protected] in exim when receiving email from local VMs and I am wondering if this could be a bad idea.
I'm going to redo this mail server from scratch soon because it has been left untouched for something like 6 years. Some software was compiled on it with shared libraries which then were updated but the software was not recompiled. It's a dumpster fire and I'm actually surprised most of our emails seem to go through.
I would also add to the article that's it's a good idea to have some rate limit on outbound email, outbound email spam checks and to check that the user sending email has the right to use that address.
Quick question (as someone who doesn't manage email): why not using an external mail provider (like mailjet) ? Privacy concerns and/or internal mails that shouldn't get leaked ?
Microsoft (Live/Hotmail/Outlook/...) is the single worst offender against small mail servers. I have run a mail server for the last ten years spam-free and vulnerability-free. It has never been on any spam blacklists, delivery to GMail is just fine. I have implemented everything from SPF to DKIM to correct rDNS.
Yet, my IP address predictably ends up on Microsoft's "blacklist" every two months or so because of "spam behaviour or user complaints". I am rather sure that neither is true. Every time I have to fill in their appeal form, get unlocked immediately but have to do the same dance again two months later.
They have every right to reject a mail server but at least they shouldn't lie about the reason. My server's IP address gets blacklisted again and again even during months I don't send any mail to Microsoft addresses.
With all the fixes and standards just to send a cute cat GIFs to your friends on Gmail is such a pain in the butt for most self hosters. If you're doing it, great! Keep it up! All the better for you. But for me, using a e-mail hosting provider with a good reputation is preferable, they've already done all the hard work (my preference is FastMail, but it's not everyone's cup of tea). I've been in the IT field for almost 20 years, I've done the self hosting e-mail thing. In my early teens and 20s, I didn't have anyone to please or take care of, so keeping up was easy with new e-mail standards as they rolled out, 15-20 years later, I'm divorced, work full time and have two kids, I simply don't have the time to self-host anymore. I'd rather give somebody a few $ a month to take care of that for me.
I've been so irritated with self hosting I've gone the other way: I'm in the process of setting up an email hosting co-op, with the aim of paying for managed service from a small hosting provider for most things but paying no more than I do now. That way it won't only fall to me to deal with deliverability issues etc.
> I'd rather give somebody a few $ a month to take care of that for me
I'll do it though i'm not FastMail!
I'm working on a SES backed email client so SES takes care of deliverability (assuming the user does the basic DKIM, MX, TXT set ups) so it will work out much cheaper to send and receive mails than a FastMail/GSuite type solution.
Plus you'll get unlimited disposable emails for free, a shared inbox (if you want to share an inbox with your family to help stay on top of every one's schedules, for exam), and while the client I have built is pretty rudimentary, i'll be adding bells and whistles to make it better.
I made Forward Email after running into many issues with self-hosted solutions (and realized thousands of others have the same issue due to the high-demand after launch back in 2017). It's a great alternative to having to set up your own small mail server.
Will the fully-fledged email service also be open source?
(This is the first time I've become aware of Forward Email despite having run my own forwarding email server for years -- you should do more/better marketing/outreach!)
I'd still have to have a mail server somewhere that receives the messages your service forwards to me though? Your service just provides one level of indirection, correct? Edit: and what about outbound email?
Having the top reply to an article about hosting your own e-mail be somebody shilling their service based around the idea of how you should not in fact, run your own mail service is a bit gross.
Mailinabox handles all of that automatically and nicely.
You just need to do some manual DNS work and check your IP against blacklists on http://multirbl.valli.org/ - and mostly done.
I think part of the point of the post and the PDF to which it links is that Gmail and the other big operators don't use the blacklists, so for deliverability to the majority of recipients it makes no difference whether you are or are not on those lists.
Thank you!
I recently managed to get a static IP and was looking forward to set my own mail server to be independent of the big providers.
Now I have been searching for something exactly like this, a recent, concise overview of the hoops I have to jump through to be allowed to send to the big guys.
Point 14 is a very good one. If any account gets compromised, you might as well request a new IP as your previous IP will be in the dog house for a long time.
Almost more important than monitoring failed logons, monitor successful logons, for instance by country. If it is a small mail server, chances are your users are in a handful of countries. If you suddenly see a successful logon from vietnam or brazil, you probably want to be notified ASAP.
Addendum: if you host a mailing list server such as Mailman, you'll need to do one of (1) don't change the content or DKIM-signed headers at all (e.g. by adding Subject tags or unsubscribe instructions); (2) rewrite the From: header so that you don't look like you are impersonating the individual senders; (3) use ARC [https://en.wikipedia.org/wiki/Authenticated_Received_Chain] signatures. I don't know if Gmail and the other big providers actually honor ARC headers yet, but if they ever do, that would seem to be the cleanest solution if you need to add unsubscribe instructions or similar banners.
I set up a personal mail server (SMTP+IMAP, no webmail) a couple months ago for personal/family use —- no sense in paying about $60 per user per year for GSuite or 365.
(Microsoft actually offer a custom domain email solution for families that doesn’t charge per user provided you host your domain with GoDaddy, but it has some limitations, eg with respect to aliases, so that wasn’t quite right for me.)
I did so with some trepidation, having been told by multiple people that I shouldn’t. Maybe I got lucky and had an IP with good reputation, but I was able to send mail to Gmail, Outlook, Yahoo and specific email lists after a few rounds of configuration.
The only problem I had was with Apple’s iCloud email, but engaging with ProofPoint soon fixed that.
Overall it wasn’t too bad an experience. I’d class myself as beginner level in that I’ve done this for myself ages ago (but never in a professional capacity).
Just leaving this anecdote out here as a small counterpoint to all the horror stories out there. If you’ve got an IP address with a poor reputation I can imagine it’d be a much more difficult exercise.
Not sure if I want to carry on host other people's email anymore, it's become a huge pain. It's bad enough that Google and MS frequently refuse to deliver mail; without any apparent reason or any humans to discuss the problem with. But now that zimbra is dead (no new open source versions), email hosting is pretty much back to where it was in the 00s.
I went through the journey of configuring Postfix for a small side project I was working on.
The idea was to accept emails sent to a unique, per-user address and forward all attachments to the user’s cloud storage account.
I looked into using mail services like SES and Mailgun, but given the average expected email size and the fact that I would primarily receiving mail, the additional cost didn’t make sense. Along the way, I ended up learning quite a bit about Postfix filters, virtual domains, SPF, and DKIM.
I also learned how quickly spammers can detect open relays. Apparently, while copy-pasting a config from somewhere, I had a slightly incorrect 172.x subnet listed as part of “mynetworks”. As a result, a spammer with a machine belonging to this subnet was able to use my server as an open relay. Long story short, it took me quite a while to root cause the issue and get my domain off the blacklists!
Make sure you add yourself to dnswl.org if you're running a mailsever. Lots of people who use Spamassassian will check it (by default, it's part of the base install) and if you put yourself in there you will score yourself a few negative figures to help get your mail delivered.
Of course DMARC, DKIM and SPF are key as well.
I've run my own mailserver for the last 20 years and properly signing messages and being sure not to accept/forward spam (i use rspamd) has helped me be able to have a stable personal server all those years.
The only places I have trouble with are Microsoft, because it seems if you don't send a mail every 24 hours your reputation gets "reset" and it can be hard to deliver into them.
I have been running my own UNIX mailserver since 1997. I have consistently and loudly evangelized this practice, especially here on HN, and pushed back against the notion that it was too difficult or too time-consuming to implement and maintain one's own mailserver.
But ...
Having just recently made the major transition from relying solely on clean IP space and proper DNS records (which, in the last 18 months, has become increasingly untenable[1]) I have changed my mind.
I will continue to run my own mailserver, for activist, ideological reasons, but I must now agree that it is too difficult and complicated - and fragile.
In no particular order, some of the things that I find overly complex and disturbing are:
- The DKIM implementations are very, very over-engineered. I understand, in principle, why we want DKIM to be a pluggable component that can be used as a general building block in various implementations, blah blah blah, but there is no reason that DKIM can't just be a feature, with a corresponding blob of config lines, in sendmail or OpenSMTPd or whatever. Having to pkg-install, and track, and maintain, whatever DKIM implementation you choose, along with all of its various dependencies, etc., is a big truckload of complexity and fragility that I just can't see ever appreciating. And the irony ? A popular DKIM implementation is to use the bundled DKIM functionality built into rspamd (!) ... so it ended up being just a feature anyway.
- The whole SSL component ... I appreciate this, I understand this, and for many use-cases I just don't care. I don't currently need encrypted email and if someone makes a decision to disallow sending to addresses/domains that don't support it, fine. Of course, if all of gmail decides not to, now I have another giant truckload of complexity and fragility that I need to implement and maintain. My mailserver should be an extremely tight, stripped down host with as few packages and daemons possible[2]. Instead, I am now installing a big list of packages just so that letsencrypt can fire up a temporary web server to clicky click make my certificates[3].
- Now my mailserver is running a database (Redis) which is required to run rspamd, which is required to implement DKIM, without which I cannot send mail to yahoo.com addresses. True story.
So much complexity and fragility ...
[1] About 18 months ago yahoo.com stopped, with no errors or bounces, delivering mail from an 11 year clean IP with proper DNS/MX entries. I implemented DKIM and the problem is solved.
[2] My mailserver is a FreeBSD jail and previously ran only sendmail - and nothing else. Now I am running OpenSMTPd, rspamd, redis ... and once in a while I am running a webserver to communicate with letsencrypt.
[3] Speaking of letsencrypt ... requesting, generating and implementing SSL certs (even for use on the web) is simply creating, and trading, blocks of plain old ASCII text. I haven't looked into this, but is there an "expert mode" somewhere in letsencrypt where I don't have to run webserver instances and ... I can just run openssl command lines and cut and paste blobs of ASCII ?
I think of it as a process of punctuated equilibrium.
Every several years (or ten), it requires a flurry of activity to update to new preferred daemons, or new hardware.
Between these brief periods, everything is stable and solid. I don't have to think about it at all. In 25 years, I've rebuilt things 5 or 6 times. After the first few times, it has been more of a chore than an adventure, but it's still worth it to me, for now.
One development that might change my mind about that: Ubiquitous and non-vendor-dependent MFA. If email was not the security mechanism of last resort (or the skeleton key), then I might consider outsourcing it.
...
Re: Rspamd: It works well, but I did not enjoy the configuration effort. And yes, I never wanted Redis on my mailserver, but there it is -- along with an astonishing amount of RAM usage, which offends me in theory even if it has no practical impact. On the positive side, after the initial few hours of effort to get things set up, I have spent zero hours maintaining any of it, so I can't reasonably complain.
Re: letsencrypt: I use dehydrated for certificate renewal, and I run a temporary webserver as you describe. Not a full dedicated webserver package, just python (or ruby) which is already installed. It's launched from the same script that automates the license renewal. Not elegant, but adequate. You could probably get away with netcat.
Of course you can validate by DNS instead of HTTP. But that's not always convenient either.
..and even if you follow all the best practices you'll just randomly end up blacklisted for what I can only imagine is misguided users clicking on spam.
If that happens on the wrong email provider you're out of luck. One of the mail servers I have a hand in has been blocked by one of those difficult providers for the past 2 months and we sent them 4 mails/week before the block on average :)
> Now my mailserver is running a database (Redis) which is required to run rspamd, which is required to implement DKIM, without which I cannot send mail to yahoo.com addresses. True story.
I have DKIM set up just fine without running Redis or rspamd so this part is purely down to your choices.
> Speaking of letsencrypt
Let's Encrypt is a website/service. There are many clients and you can write your own. At some point Let's encrypt will need to verify that you control the domain so you will need to be able to either host something via HTTP or have sufficient dynamic control over the DNS server. Neither DNS nor HTTP needs to be hosted on the same host as your mail server though.
Can you elaborate on your problem with Yahoo? I ran into that a while back, where a yahoo email address couldn't send to me... strange since DKIM validates for the domain.
There are all valid points and I question my decision to run my mailserver, but in the spirit of the Internet I continue to do so.
> [3] Speaking of letsencrypt ... requesting, generating and implementing SSL certs (even for use on the web) is simply creating, and trading, blocks of plain old ASCII text. I haven't looked into this, but is there an "expert mode" somewhere in letsencrypt where I don't have to run webserver instances and ... I can just run openssl command lines and cut and paste blobs of ASCII ?
Let's Encrypt provides an ACME service. So, you will want to (run something that can) speak ACME, a somewhat RESTful HTTP API documented in RFC 8555 - https://tools.ietf.org/html/rfc8555. You can use a web service to do this part, but then you're relying on some random web service and you have to manually do a bunch of steps to renew at least every 90 days which I'm going to assume you agree sucks.
There are lots of people who have built such tools that can look after this for you automatically, perhaps you would like https://acme.sh/ which as its name might suggest is a shell script.
Let's Encrypt offers three of the (increasingly misnamed) Ten Blessed Methods, you can prove control over a DNS name by doing any of these three things:
1. Running an HTTP server on port 80 which can answer certain well-known requests in a specific way acknowledging your ACME keys. This server needn't have any privileges, acme.sh can help you spin one up that will basically say "Yes, rsync is allowed this certificate" for any request. Except instead of "rsync" it's a key only you have, so your server answering this way never helps any bad guys so long as you keep your ACME keys secure. [You may not be aware you have "ACME keys" the software you've used probably just minted some and squirreled them away in a dot directory]. With that server in place, your requests for a certificate for that name will always work, because the key matches, Let's Encrypt will call your server, make up a random nonce and ask if that's authorised and for who, your HTTP server will always answer "rsync is authorised" so it works, tidy.
2. ALPN on port 443. A TLS server answering port 443 can accept a special ACME-only ALPN protocol choice from the Let's Encrypt servers and use that to prove control. You almost certainly don't want this unless you implement web servers (which run on port 443 with TLS already)
3. DNS queries for a magic name that can't be a hostname. The queries ask for _acme-challenge which has an underscore in it. Underscores are allowed in DNS but can't be hostnames, which is fine because this isn't a hostname. Your ACME client will need to write to DNS, it can either have a way to write to your actual DNS, or you can use a CNAME to redirect the names you need to another DNS server and give the ACME client control over that. [Edited to fix DNS name mistake]
You can insulate the ACME client from the server completely with either the trick I described in (1) or using DNS like (3) and using a Certificate Signing Request, CSR, which is one of the many ASCII text blobs you can get from OpenSSL. If you're comfortable re-using a private key for longer periods than the 90-days assumed in Let's Encrypt (a 2048-bit RSA key seems safe to me for the usual lifetime of a mailserver) you can even re-use the same CSR again and again to request each new certificate without sending any data from the mail server to the system calling Let's Encrypt.
You will need to arrange to get each new certificate installed (the private keys don't change in this scenario). But certificates are public documents, if it came to it the mail server can (this is very silly but it would actually work) just wait until 24h after the certificate should have been issued and then go get it from https://crt.sh/ -- In reality you can probably find some sensible way to get this public information back to the mail server each time the certificate is renewed.
Gmail still sends good emails to spambox even today... For example, a realestate agent sent documents with really personal info in them and it still got qualified as spam
Very interesting article, but hosting my own email is a fantasy for me that I will probably never spend the time setting up and maintaining. ProtonMail does everything I need in an email service, with FastMail and Hey (which I only tried for a week) being excellent runners-up.
Sorry for going off topic but everyone should host their own web site(s) and stop putting their content on other people's platforms.
[+] [-] pgrote|5 years ago|reply
It became a self fulfilling prophecy. Our email would trigger a rule in big co's third party security app, which would then report to a centralized rule breaker clearing house automatically. Clients couldn't receive our emails, we would dive in, submit appeal to clearing house, get clear and a week later do it all again.
Everything was configured properly on our end. We passed all the online validation engines. IPs were our own personally owned block and pristine.
It became too much.
Switched to office365 and all problems magically disappeared. Sent emails to same big cos with third party email security and haven't had a single issue.
[+] [-] ltriant|5 years ago|reply
I was there for 10 years, and this was only a problem in the final ~2 years of my employment, which roughly lines up with your 4 years remark.
[+] [-] superkuh|5 years ago|reply
It's not about setting up perfect signed email and all the new techs. It's not about having the same clean IP for a decade. It's just the same old network effect combined with profit motive. The obvious change in the last few years (I've run a mailserver for 9) is managerial not technical.
[+] [-] technion|5 years ago|reply
People make a big deal of SPF/DKIM etc. I think you should, for the sake of best practice. But you can set yourself up with gsuite or Office 365 and pay no attention to these things, and you can fully expect mail to get through to everyone. You can even setup a broken SPF record that disallows office 365 email and for the most part, everyone will still get your email.
[+] [-] throwaway2048|5 years ago|reply
However if a domain signs up for Google Apps, magically you get whitelisted, even if you are handling your own email, funny thing that.
I guess Google's customers are just more trustworthy...
[+] [-] colordrops|5 years ago|reply
[+] [-] Legogris|5 years ago|reply
You don't have to throw out the baby with the bathwater (:
[+] [-] bluedino|5 years ago|reply
No real issues. Sure, you make it on a spam blacklist once in a while, but at least you know when it's going to be fixed, instead of dealing with zero response (or denial) from a big hosted mail provider.
Besides, not all of "our email" comes out of our office, our CRM and ERP products send mail on our behalf, and they have problems delivering mail more than we do.
[+] [-] ubermonkey|5 years ago|reply
[+] [-] tgsovlerkhgsel|5 years ago|reply
a) a docker container (or other standardized, easy to deploy technology) with a "mail server in a box implementing best practices"
b) a test for the stuff that Docker can't cover like DNS config, ideally with scripts to set it up on the most popular providers and clear copy&paste instructions for the rest.
People shouldn't need to fully understand DKIM, SPF, etc. - they should just be able to click a button, copy&paste generated DNS records, and be up and running.
Otherwise, commercial providers are just the only option that makes operational sense for most cases, _even if you know how to do it right_, because doing everything right manually step by step is still to much work to be worth it in many cases.
None of this seems _hard_, it's just tedious.
[+] [-] mihaifm|5 years ago|reply
It’s pretty good, I use it for my own server. It still requires some config, but better than running all the services from scratch.
[+] [-] pbhjpbhj|5 years ago|reply
Email has serious economies of scale and keeping such a critical system operational, on a small scale, was going to be some deal of stress.
So I just went with my hosting providers Plesk-fronted setup, which matches your para.3 description.
[+] [-] inopinatus|5 years ago|reply
* Setup ARC (RFC8617) for all mailing lists and forwarding.
* Sign up for the Google & Microsoft postmaster tools. They're not great, but it's better than nothing.
* Implement RFC8058 / RFC2369 unsubscribe mechanisms.
* Use an outbound suppression list to ensure that unsubscribe requests or unwanted mail complaints have lasting effect.
* Article touches on DMARC reporting, for which I suggest Postmark's free monitoring tool at https://dmarc.postmarkapp.com/
* From time to time run your domain name(s) and IP address(es) through a panel of reputation services. Check out any red flags (some of them matter, some don't).
And for content:
* Re-evaluate email templates. Many out there are hot garbage, practically inviting users to hit the junk button, which just hurts sender reputation. Ensure there's always a plaintext part. Don't use tracking/opening pixels. Write validating HTML. Minimise size. Don't bury the unsubscribe link, put it at the top, and make sure it clearly works without hostile barriers like "are you sure" pages or sign-in.
* In the very first paragraph, and without scrolling on a mobile, we must explain, honestly and concisely, why we are contacting someone, and why they might want to read the rest, because we have no entitlement to someone's attention.
* Run all mailers through a spam scoring tool as part of CI/CD, because a high score is a bug.
In addition, Google's sender guidelines are well worth reading, they're more than just self-serving gatekeeping: https://support.google.com/mail/answer/81126?hl=en
[+] [-] thoraway1010|5 years ago|reply
Some mail delivery systems start to notice that your domain name is being used as part of a lot of spam / bogus emails, so you can have perfect IP history / no spam and STILL start triggering some random filters (no big players but smaller protection product filters).
So the game must be absolutely never ending for everyone and the inbox a valuable target - especially now that unsolicited phone calls really do seem to get ignored these days - I feel like spammers killed the golden goose on phone calls and the telcos let them.
I will say DKIM / DMARC is working well, except google (which we now use for outbound) gives us transient SPF errors even though we are 100% using their IPs. Not sure why that is (ie, SPF failure on an IP that should clear)
[+] [-] icefo|5 years ago|reply
In my organization we have ~10 linux VM and they are all configured to send email to an exim MTA. For example monitoring stuff. The system mail name is set to vmName.srv.domain.name (domain.name is replaced by our actual domain and this resolve to a local address) so it's quite common to receive mail from [email protected]. This cause no issue when it's delivered to the local Cyrus server but when it's transfered to a Gmail address for some reason Gmail reject the email because the headers are bad.
I am considering rewriting the address to [email protected] in exim when receiving email from local VMs and I am wondering if this could be a bad idea.
I'm going to redo this mail server from scratch soon because it has been left untouched for something like 6 years. Some software was compiled on it with shared libraries which then were updated but the software was not recompiled. It's a dumpster fire and I'm actually surprised most of our emails seem to go through.
I would also add to the article that's it's a good idea to have some rate limit on outbound email, outbound email spam checks and to check that the user sending email has the right to use that address.
[+] [-] johnchristopher|5 years ago|reply
[+] [-] user5994461|5 years ago|reply
@vmName.srv.domain.name is not configured so it goes to spam.
[+] [-] dillonmckay|5 years ago|reply
[+] [-] TonyTrapp|5 years ago|reply
Yet, my IP address predictably ends up on Microsoft's "blacklist" every two months or so because of "spam behaviour or user complaints". I am rather sure that neither is true. Every time I have to fill in their appeal form, get unlocked immediately but have to do the same dance again two months later.
They have every right to reject a mail server but at least they shouldn't lie about the reason. My server's IP address gets blacklisted again and again even during months I don't send any mail to Microsoft addresses.
[+] [-] davemtl|5 years ago|reply
[+] [-] hkt|5 years ago|reply
[+] [-] saimiam|5 years ago|reply
I'll do it though i'm not FastMail!
I'm working on a SES backed email client so SES takes care of deliverability (assuming the user does the basic DKIM, MX, TXT set ups) so it will work out much cheaper to send and receive mails than a FastMail/GSuite type solution.
Plus you'll get unlimited disposable emails for free, a shared inbox (if you want to share an inbox with your family to help stay on top of every one's schedules, for exam), and while the client I have built is pretty rudimentary, i'll be adding bells and whistles to make it better.
Github @ https://github.com/saiorama/ses-email-client Landing page to submit a request @ https://shared-inbox.landen.co/
Note to others - I'm open to collaborating to take back email from the big guys. Hit me up via Github.
[+] [-] niftylettuce|5 years ago|reply
https://forwardemail.net
P.S. I'm coming out with a fully-fledged email service built on top of my R&D with Forward Email.
[+] [-] dochtman|5 years ago|reply
(This is the first time I've become aware of Forward Email despite having run my own forwarding email server for years -- you should do more/better marketing/outreach!)
[+] [-] tux1968|5 years ago|reply
[+] [-] CraftThatBlock|5 years ago|reply
[+] [-] cyberpunk|5 years ago|reply
[+] [-] throwaway2048|5 years ago|reply
[+] [-] juskrey|5 years ago|reply
[+] [-] jeffbee|5 years ago|reply
[+] [-] evilsetg|5 years ago|reply
[+] [-] saimiam|5 years ago|reply
I'm also trying to build my own email client for SES @ https://github.com/saiorama/ses-email-client. Happy to collaborate in case you want to work on this issue.
[+] [-] cm2187|5 years ago|reply
Almost more important than monitoring failed logons, monitor successful logons, for instance by country. If it is a small mail server, chances are your users are in a handful of countries. If you suddenly see a successful logon from vietnam or brazil, you probably want to be notified ASAP.
[+] [-] jks|5 years ago|reply
[+] [-] signal11|5 years ago|reply
(Microsoft actually offer a custom domain email solution for families that doesn’t charge per user provided you host your domain with GoDaddy, but it has some limitations, eg with respect to aliases, so that wasn’t quite right for me.)
I did so with some trepidation, having been told by multiple people that I shouldn’t. Maybe I got lucky and had an IP with good reputation, but I was able to send mail to Gmail, Outlook, Yahoo and specific email lists after a few rounds of configuration.
The only problem I had was with Apple’s iCloud email, but engaging with ProofPoint soon fixed that.
Overall it wasn’t too bad an experience. I’d class myself as beginner level in that I’ve done this for myself ages ago (but never in a professional capacity).
Just leaving this anecdote out here as a small counterpoint to all the horror stories out there. If you’ve got an IP address with a poor reputation I can imagine it’d be a much more difficult exercise.
[+] [-] bluehatbrit|5 years ago|reply
[+] [-] elric|5 years ago|reply
[+] [-] WarOnPrivacy|5 years ago|reply
This year though, most of my dodgy SMTP traffic is coming 3 new problem children - Gmail(spam), Amazon(spam, other) and Microsoft Azure(other).
"Other" here is anything besides spam, like dictionary attacks, transactionless connections or service probes.
[+] [-] Cyph0n|5 years ago|reply
The idea was to accept emails sent to a unique, per-user address and forward all attachments to the user’s cloud storage account.
I looked into using mail services like SES and Mailgun, but given the average expected email size and the fact that I would primarily receiving mail, the additional cost didn’t make sense. Along the way, I ended up learning quite a bit about Postfix filters, virtual domains, SPF, and DKIM.
I also learned how quickly spammers can detect open relays. Apparently, while copy-pasting a config from somewhere, I had a slightly incorrect 172.x subnet listed as part of “mynetworks”. As a result, a spammer with a machine belonging to this subnet was able to use my server as an open relay. Long story short, it took me quite a while to root cause the issue and get my domain off the blacklists!
[+] [-] muppetman|5 years ago|reply
[+] [-] rsync|5 years ago|reply
I have been running my own UNIX mailserver since 1997. I have consistently and loudly evangelized this practice, especially here on HN, and pushed back against the notion that it was too difficult or too time-consuming to implement and maintain one's own mailserver.
But ...
Having just recently made the major transition from relying solely on clean IP space and proper DNS records (which, in the last 18 months, has become increasingly untenable[1]) I have changed my mind.
I will continue to run my own mailserver, for activist, ideological reasons, but I must now agree that it is too difficult and complicated - and fragile.
In no particular order, some of the things that I find overly complex and disturbing are:
- The DKIM implementations are very, very over-engineered. I understand, in principle, why we want DKIM to be a pluggable component that can be used as a general building block in various implementations, blah blah blah, but there is no reason that DKIM can't just be a feature, with a corresponding blob of config lines, in sendmail or OpenSMTPd or whatever. Having to pkg-install, and track, and maintain, whatever DKIM implementation you choose, along with all of its various dependencies, etc., is a big truckload of complexity and fragility that I just can't see ever appreciating. And the irony ? A popular DKIM implementation is to use the bundled DKIM functionality built into rspamd (!) ... so it ended up being just a feature anyway.
- The whole SSL component ... I appreciate this, I understand this, and for many use-cases I just don't care. I don't currently need encrypted email and if someone makes a decision to disallow sending to addresses/domains that don't support it, fine. Of course, if all of gmail decides not to, now I have another giant truckload of complexity and fragility that I need to implement and maintain. My mailserver should be an extremely tight, stripped down host with as few packages and daemons possible[2]. Instead, I am now installing a big list of packages just so that letsencrypt can fire up a temporary web server to clicky click make my certificates[3].
- Now my mailserver is running a database (Redis) which is required to run rspamd, which is required to implement DKIM, without which I cannot send mail to yahoo.com addresses. True story.
So much complexity and fragility ...
[1] About 18 months ago yahoo.com stopped, with no errors or bounces, delivering mail from an 11 year clean IP with proper DNS/MX entries. I implemented DKIM and the problem is solved.
[2] My mailserver is a FreeBSD jail and previously ran only sendmail - and nothing else. Now I am running OpenSMTPd, rspamd, redis ... and once in a while I am running a webserver to communicate with letsencrypt.
[3] Speaking of letsencrypt ... requesting, generating and implementing SSL certs (even for use on the web) is simply creating, and trading, blocks of plain old ASCII text. I haven't looked into this, but is there an "expert mode" somewhere in letsencrypt where I don't have to run webserver instances and ... I can just run openssl command lines and cut and paste blobs of ASCII ?
[+] [-] quesera|5 years ago|reply
I think of it as a process of punctuated equilibrium.
Every several years (or ten), it requires a flurry of activity to update to new preferred daemons, or new hardware.
Between these brief periods, everything is stable and solid. I don't have to think about it at all. In 25 years, I've rebuilt things 5 or 6 times. After the first few times, it has been more of a chore than an adventure, but it's still worth it to me, for now.
One development that might change my mind about that: Ubiquitous and non-vendor-dependent MFA. If email was not the security mechanism of last resort (or the skeleton key), then I might consider outsourcing it.
...
Re: Rspamd: It works well, but I did not enjoy the configuration effort. And yes, I never wanted Redis on my mailserver, but there it is -- along with an astonishing amount of RAM usage, which offends me in theory even if it has no practical impact. On the positive side, after the initial few hours of effort to get things set up, I have spent zero hours maintaining any of it, so I can't reasonably complain.
Re: letsencrypt: I use dehydrated for certificate renewal, and I run a temporary webserver as you describe. Not a full dedicated webserver package, just python (or ruby) which is already installed. It's launched from the same script that automates the license renewal. Not elegant, but adequate. You could probably get away with netcat.
Of course you can validate by DNS instead of HTTP. But that's not always convenient either.
[+] [-] _jal|5 years ago|reply
Regarding letsencrypt, you can use DNS challenges. See https://letsencrypt.org/docs/challenge-types/ .
[+] [-] iMerNibor|5 years ago|reply
If that happens on the wrong email provider you're out of luck. One of the mail servers I have a hand in has been blocked by one of those difficult providers for the past 2 months and we sent them 4 mails/week before the block on average :)
[+] [-] account42|5 years ago|reply
I have DKIM set up just fine without running Redis or rspamd so this part is purely down to your choices.
> Speaking of letsencrypt
Let's Encrypt is a website/service. There are many clients and you can write your own. At some point Let's encrypt will need to verify that you control the domain so you will need to be able to either host something via HTTP or have sufficient dynamic control over the DNS server. Neither DNS nor HTTP needs to be hosted on the same host as your mail server though.
[+] [-] throwaway2048|5 years ago|reply
http://dkimproxy.sourceforge.net/
[+] [-] stratosmacker|5 years ago|reply
There are all valid points and I question my decision to run my mailserver, but in the spirit of the Internet I continue to do so.
[+] [-] tialaramex|5 years ago|reply
Let's Encrypt provides an ACME service. So, you will want to (run something that can) speak ACME, a somewhat RESTful HTTP API documented in RFC 8555 - https://tools.ietf.org/html/rfc8555. You can use a web service to do this part, but then you're relying on some random web service and you have to manually do a bunch of steps to renew at least every 90 days which I'm going to assume you agree sucks.
There are lots of people who have built such tools that can look after this for you automatically, perhaps you would like https://acme.sh/ which as its name might suggest is a shell script.
Let's Encrypt offers three of the (increasingly misnamed) Ten Blessed Methods, you can prove control over a DNS name by doing any of these three things:
1. Running an HTTP server on port 80 which can answer certain well-known requests in a specific way acknowledging your ACME keys. This server needn't have any privileges, acme.sh can help you spin one up that will basically say "Yes, rsync is allowed this certificate" for any request. Except instead of "rsync" it's a key only you have, so your server answering this way never helps any bad guys so long as you keep your ACME keys secure. [You may not be aware you have "ACME keys" the software you've used probably just minted some and squirreled them away in a dot directory]. With that server in place, your requests for a certificate for that name will always work, because the key matches, Let's Encrypt will call your server, make up a random nonce and ask if that's authorised and for who, your HTTP server will always answer "rsync is authorised" so it works, tidy.
2. ALPN on port 443. A TLS server answering port 443 can accept a special ACME-only ALPN protocol choice from the Let's Encrypt servers and use that to prove control. You almost certainly don't want this unless you implement web servers (which run on port 443 with TLS already)
3. DNS queries for a magic name that can't be a hostname. The queries ask for _acme-challenge which has an underscore in it. Underscores are allowed in DNS but can't be hostnames, which is fine because this isn't a hostname. Your ACME client will need to write to DNS, it can either have a way to write to your actual DNS, or you can use a CNAME to redirect the names you need to another DNS server and give the ACME client control over that. [Edited to fix DNS name mistake]
You can insulate the ACME client from the server completely with either the trick I described in (1) or using DNS like (3) and using a Certificate Signing Request, CSR, which is one of the many ASCII text blobs you can get from OpenSSL. If you're comfortable re-using a private key for longer periods than the 90-days assumed in Let's Encrypt (a 2048-bit RSA key seems safe to me for the usual lifetime of a mailserver) you can even re-use the same CSR again and again to request each new certificate without sending any data from the mail server to the system calling Let's Encrypt.
You will need to arrange to get each new certificate installed (the private keys don't change in this scenario). But certificates are public documents, if it came to it the mail server can (this is very silly but it would actually work) just wait until 24h after the certificate should have been issued and then go get it from https://crt.sh/ -- In reality you can probably find some sensible way to get this public information back to the mail server each time the certificate is renewed.
[+] [-] coronadisaster|5 years ago|reply
[+] [-] mark_l_watson|5 years ago|reply
Sorry for going off topic but everyone should host their own web site(s) and stop putting their content on other people's platforms.