Forwarding email with outlook.com/Office365/Microsoft Exchange also breaks DMARC. Exchange Server sometimes modifies the headers of emails when forwarding them, invalidating the DKIM signature, so then a DMARC policy rejects the forwarded email. Apparently Microsoft have a fix in the pipeline for this, but it's been taking ages.
The primary problem is that DMARC is fundamentally flawed, and was not enacted using a standards process that respected all of the stakeholders. As a result, it fundamentally becomes a matter of power politics.
If there are a bunch of people who need to participate in a particular mailing list --- say, IETF mailing list or the Linux Kernel development lists --- more than they need to stick with a particular mail provider, it becomes possible to say to them, "you want to participate in our community"? Change mail providers.
In the cases where a mailing list community badly needs the Yahoo users, Yahoo can dictate to the mailing list --- change your mailing list software and inflict pain all on your mailing list users, or you don't get access to our e-mail user community. (And rewriting the from field has all sorts of bad effects, including corrupting contact databases that auto populate based on names and addresses in the from field, making the mail summary index useless, breaking reply-to sender, etc. So there is real pain involved here.)
Part of the issue was that DMARC was originally intended for domains that only sent official announcements (e.g. for credit card companies or banks) and where employee e-mails came from a different domain. For that original use case, DMARC worked perfectly well. Apparently Yahoo then decided it had a terrible SPAM problem, and decided DMARC was a blunt instrument to use, and inflicted on consumer e-mails. Other companies didn't think ahead, and used the same domain for official e-mails as their employee e-mails, and decided that protecting their users was more important than inconveniencing their employees.
This is why it's ultimately all about power politics. If you want to participate in Linux Kernel development, you need an e-mail address that won't arbitrarily cause your e-mails to be dropped (for example, so Linus Torvalds doesn't get your pull requests). Despite this, growth in Linux Kernel Development continues to be growing, which means that when choosing between the inconvenience of changing or using an alternate e-mail provider, versus participating in Linux development, developers choose the former. But that only work because Linux has the economic power and clout to get away with it.
Heck, over ten years ago, refusing to buckle in to crap e-mail systems forced IBM to allow its Linux Technology Center folks install a standards-complaint IMAP server instead of using that abomination caused Lotus Notes which the rest of the company was forced to use. (And this was probably a better demonstration of the power of Linux, given how much IBM was stubbornly attached to Blotus Goats.) But make no mistake. This is Trump style, power politics. It doesn't hurt those with a lot of economic power, but if you're some tiny, podunk church mailing list, or some other group lacking in economic power, you're screwed.
I stopped reading when it referred to SPF as "Sender Protection Framework." As usual, the discussion on HN is far better than R'ing TFA.
We recently had a collision between the practices of one of our vendors, the silly way that .edu's tend to forward mail, and Gmail's DMARC policy. The vendor is doing everything "right" in terms of their own (completely transactional) mailings on our behalf but some still want to blame them for poorly forwarded messages sent to .edu addresses that subsequently get swallowed up by Gmail and are never seen by the authors or referees.
I've seen the same issue with an edu address using office 365. The user set his mail for our org to forward to his .edu address. The .edu address was set to forward to his personal Gmail. The Gmail at the end of the chain was rejecting the mail because the edu didn't rewrite the from address.
Authenticated Received Chain (ARC) is being developed to work simultaneously with DMARC and help provide relief to mail agents that break DKIM by allowing ARC aware receivers to sign authentication results that allow downstream processors to review what the initial results were and make their decisions.
> For many years, there was no real way to verify that you really got the email the person that the From header states.
DMARC doesn't really assert your identity---it asserts the identity of the server.
Remember that you can also use PGP to assert _yourself_, and this works well with mailing lists (your mail client will hopefully distinguish between the signed message and the mailing list footer, which is unsigned). It also persists---if a mailing list is stripping DMARC headers, then that doesn't help you any.
I work on Gaggle Mail which is a group mailing list provider that avoids all this by only using a From address that we control. I do think stricter addressing policies will become the norm in years to come which is why we took this approach.
As someone who thought it'd be a good idea to run their own mail server, I found that without SPF/DKIM/DMARC your mail gets identified as spam a lot. Having DMARC just means you get told about it.
SPF has another problem. I sent an email to someone recently @theirdomain.com. I subsequently saw by chance that the email was rejected because they were hosting @theirdomain.com with a random ISP but they had configured the mail to be forwarded to a mailbox in @gmail.com.
Gmail sees the email coming from @theirdomain.com's servers, rather than my server. Gmail checks the SPF record which doesn't match, and it rejects it.
I understand that this style of forwarding is anyway bad because gmail see's all email the user receives @theirdomain.com as coming from those servers, not their true origin. If @theirdomain.com receives (and forwards) any spam, it looks like a spammer to gmail.
The solution to this is SRS[0], but yes, not every mail-forwarder implements this (correctly), and it seems to break DKIM. The good news is that an SPF soft-fail and a DKIM pass usually means a DMARC pass. More information here[1].
If the mailing list changes the From address, then the message won't show the correct author in the message list in my email client. Replying may work (because of Reply-To:), but not having the correct "From" in the client is really a deal breaker.
Besides, DKIM already solved this problem by not caring where the mail comes from, just that the message was signed. It seems the solution to me is to stop using SPF if you want to use mailing lists.
Ran into this problem setting up a new mailing list system. I'd always wondered by mailing list mail sent from yahoo went directly to spam, while mail from gmail worked just fine.
Mailing list mail will always fail DMARC if you keep the original sender's address, but when setting up DMARC for a domain, you specify what should happen to mail from your domain if it fails the DMARC check.
For gmail and other sane providers, their DMARC failure is set to ignore, which means DMARC failing for those addresses gets calculated into spam probability, but isn't an auto-fail. For yahoo and AOL, they have it set to reject, which means mailing list mail from those domains automatically goes to spam for gmail.
Groupserver mailing list software resolves this by checking the DMARC settings for the sender. If DMARC is set to ignore, the sender address is kept, because it won't actually affect delivery. If it's set to reject, it mangles the sender address so that the mail at least gets delivered.
I've had some contact with a big client e-mail filter, (30k+ users) and we chose to expose a soft SPF in the external DNS and a hard SPF in an internal DNS used as a cache by the mailfilter.
This was a workaround because the big org had countless 3rd party suppliers of services who many times wanted to mail as @big.corp. And this was fine when mailing to big corp as we could whitelist their senders in the SPF service config, but when mailing to the rest of the internet like gmail, yahoo and hotmail we needed a soft SPF fail instead of having to add those countless servers in our DNS record.
Unfortunately, having the list software create a new message breaks threading. It is common practice to Cc people (subscribed to the list or not) that may need to interact with the thread, but they get the message directly from the sender while list recipients get a different message (different From, different Message-ID). Similarly, if the original sender replies to their own outgoing message, it will not thread for list recipients.
Threading can be preserved when creating a new message as long as the mailing software re-uses the same: 'message-id', 'references', 'in-reply-to' and 'thread-index' headers.
Once, I needed to email something to djb. I tried, and his email system sent back a challenge which was dumped into my spam box. I never did manage to get an email to him.
This is a terrible article that's almost completely wrong as it doesn't even mention the distinction between envelope (SMTP) from and header (RFC822) from.
The envelope from is what is transmitted in SMTP commands and specifies where bounces due to delivery delays or failures are supposed to be sent, the header from is what is displayed to the recipient as the sender, but is completely ignored by SMTP.
SPF only deals with the envelope from, DKIM only deals with the header from (and other parts of the email headers/content).
Really, there is nothing there that necessarily prevents mailing lists from working just fine:
A mailing list can (and should) replace the envelope from with its own address, so that bounces from subscribers aren't sent to the author of the message, but to the mailing list software, which then can do bounce management, such as automatically unsubscribing addresses that consistently bounce because they don't exist anymore. As the mailing list software should use its own domain for the bounce addresses, the mailing list operator can set up SPF to authorize the mailing list server as an outbound server for that domain just fine, and that has no effect whatsoever on the header from that's shown to the recipient, and that replies would go to by default.
As for DKIM, you simply should not modify the message, and it will deliver just fine, whether through a mailing list or directly. Modifying the message mostly really shouldn't be necessary. Reply-to mangling is a bad idea anyhow, and the mailing list should be recognized by the client software either because it's in the destination headers, or by using mailing-list headers added by the mailing list software, instead of mangling the body or the subject.
> As the mailing list software should use its own domain for the bounce addresses, the mailing list operator can set up SPF to authorize the mailing list server as an outbound server for that domain just fine
This will make SPF pass but it has no effect on the DMARC result. For DMARC to use the SPF result, the envelope FROM needs to be aligned with the header From. Pretty much every mailing list already does what you've described but it doesn't help with DMARC.
> As for DKIM, you simply should not modify the message
Many mailing lists modify every message to add a footer with unsubscribe information. Others only modify some messages when necessary such as to convert HTML messages to plain-text, to strip attachments, to wrap lines to 72 characters etc.
Actually, DMARC refines SPF to also apply to the header from. If you have only an SPF record on your domain then the old rule applies (only envelop from). If you also have the _dmarc.<domain> then SPF also applies to header from.
[+] [-] mileswu|9 years ago|reply
More info at: https://blogs.msdn.microsoft.com/tzink/2016/05/19/why-does-m...
[+] [-] tytso|9 years ago|reply
If there are a bunch of people who need to participate in a particular mailing list --- say, IETF mailing list or the Linux Kernel development lists --- more than they need to stick with a particular mail provider, it becomes possible to say to them, "you want to participate in our community"? Change mail providers.
In the cases where a mailing list community badly needs the Yahoo users, Yahoo can dictate to the mailing list --- change your mailing list software and inflict pain all on your mailing list users, or you don't get access to our e-mail user community. (And rewriting the from field has all sorts of bad effects, including corrupting contact databases that auto populate based on names and addresses in the from field, making the mail summary index useless, breaking reply-to sender, etc. So there is real pain involved here.)
Part of the issue was that DMARC was originally intended for domains that only sent official announcements (e.g. for credit card companies or banks) and where employee e-mails came from a different domain. For that original use case, DMARC worked perfectly well. Apparently Yahoo then decided it had a terrible SPAM problem, and decided DMARC was a blunt instrument to use, and inflicted on consumer e-mails. Other companies didn't think ahead, and used the same domain for official e-mails as their employee e-mails, and decided that protecting their users was more important than inconveniencing their employees.
This is why it's ultimately all about power politics. If you want to participate in Linux Kernel development, you need an e-mail address that won't arbitrarily cause your e-mails to be dropped (for example, so Linus Torvalds doesn't get your pull requests). Despite this, growth in Linux Kernel Development continues to be growing, which means that when choosing between the inconvenience of changing or using an alternate e-mail provider, versus participating in Linux development, developers choose the former. But that only work because Linux has the economic power and clout to get away with it.
Heck, over ten years ago, refusing to buckle in to crap e-mail systems forced IBM to allow its Linux Technology Center folks install a standards-complaint IMAP server instead of using that abomination caused Lotus Notes which the rest of the company was forced to use. (And this was probably a better demonstration of the power of Linux, given how much IBM was stubbornly attached to Blotus Goats.) But make no mistake. This is Trump style, power politics. It doesn't hurt those with a lot of economic power, but if you're some tiny, podunk church mailing list, or some other group lacking in economic power, you're screwed.
DMARC: making email "great" again.
[+] [-] mpa000|9 years ago|reply
We recently had a collision between the practices of one of our vendors, the silly way that .edu's tend to forward mail, and Gmail's DMARC policy. The vendor is doing everything "right" in terms of their own (completely transactional) mailings on our behalf but some still want to blame them for poorly forwarded messages sent to .edu addresses that subsequently get swallowed up by Gmail and are never seen by the authors or referees.
[+] [-] discreditable|9 years ago|reply
[+] [-] roketridah|9 years ago|reply
Details at http://arc-spec.org/
[+] [-] mikegerwitz|9 years ago|reply
DMARC doesn't really assert your identity---it asserts the identity of the server.
Remember that you can also use PGP to assert _yourself_, and this works well with mailing lists (your mail client will hopefully distinguish between the signed message and the mailing list footer, which is unsigned). It also persists---if a mailing list is stripping DMARC headers, then that doesn't help you any.
[+] [-] jedbrown|9 years ago|reply
[+] [-] gcp|9 years ago|reply
Edit: given the Ts'o response above I guess they enforce the sender to not have DMARC.
"There are people with google.com addresses that need to use non-Google addresses in order to participate on the Linux Kernel Mailing List."
See also: https://www.ietf.org/mail-archive/web/dmarc/current/msg03236...
[+] [-] shutton|9 years ago|reply
More details here: https://gaggle.email/how-emails-are-addressed-and-sent
[+] [-] proaralyst|9 years ago|reply
[+] [-] hannob|9 years ago|reply
In my experience there are a lot of urban legends around this topic.
[+] [-] peterwaller|9 years ago|reply
Gmail sees the email coming from @theirdomain.com's servers, rather than my server. Gmail checks the SPF record which doesn't match, and it rejects it.
I understand that this style of forwarding is anyway bad because gmail see's all email the user receives @theirdomain.com as coming from those servers, not their true origin. If @theirdomain.com receives (and forwards) any spam, it looks like a spammer to gmail.
[+] [-] mwpmaybe|9 years ago|reply
0. http://www.openspf.org/SRS
1. https://blog.fastmail.com/2016/12/24/spf-dkim-dmarc/
[+] [-] hackuser|9 years ago|reply
[+] [-] __david__|9 years ago|reply
Besides, DKIM already solved this problem by not caring where the mail comes from, just that the message was signed. It seems the solution to me is to stop using SPF if you want to use mailing lists.
[+] [-] garaetjjte|9 years ago|reply
[+] [-] noja|9 years ago|reply
[+] [-] gcp|9 years ago|reply
[+] [-] joecot|9 years ago|reply
Mailing list mail will always fail DMARC if you keep the original sender's address, but when setting up DMARC for a domain, you specify what should happen to mail from your domain if it fails the DMARC check.
For gmail and other sane providers, their DMARC failure is set to ignore, which means DMARC failing for those addresses gets calculated into spam probability, but isn't an auto-fail. For yahoo and AOL, they have it set to reject, which means mailing list mail from those domains automatically goes to spam for gmail.
Groupserver mailing list software resolves this by checking the DMARC settings for the sender. If DMARC is set to ignore, the sender address is kept, because it won't actually affect delivery. If it's set to reject, it mangles the sender address so that the mail at least gets delivered.
[+] [-] INTPenis|9 years ago|reply
This was a workaround because the big org had countless 3rd party suppliers of services who many times wanted to mail as @big.corp. And this was fine when mailing to big corp as we could whitelist their senders in the SPF service config, but when mailing to the rest of the internet like gmail, yahoo and hotmail we needed a soft SPF fail instead of having to add those countless servers in our DNS record.
[+] [-] jedbrown|9 years ago|reply
[+] [-] shutton|9 years ago|reply
[+] [-] dekhn|9 years ago|reply
[+] [-] thresh|9 years ago|reply
Source: I run two big mailman installations.
[+] [-] throwanem|9 years ago|reply
Source: I administered a variety of Mailman-driven lists for years. Never again.
[+] [-] unknown|9 years ago|reply
[deleted]
[+] [-] zAy0LfpBZLC8mAC|9 years ago|reply
The envelope from is what is transmitted in SMTP commands and specifies where bounces due to delivery delays or failures are supposed to be sent, the header from is what is displayed to the recipient as the sender, but is completely ignored by SMTP.
SPF only deals with the envelope from, DKIM only deals with the header from (and other parts of the email headers/content).
Really, there is nothing there that necessarily prevents mailing lists from working just fine:
A mailing list can (and should) replace the envelope from with its own address, so that bounces from subscribers aren't sent to the author of the message, but to the mailing list software, which then can do bounce management, such as automatically unsubscribing addresses that consistently bounce because they don't exist anymore. As the mailing list software should use its own domain for the bounce addresses, the mailing list operator can set up SPF to authorize the mailing list server as an outbound server for that domain just fine, and that has no effect whatsoever on the header from that's shown to the recipient, and that replies would go to by default.
As for DKIM, you simply should not modify the message, and it will deliver just fine, whether through a mailing list or directly. Modifying the message mostly really shouldn't be necessary. Reply-to mangling is a bad idea anyhow, and the mailing list should be recognized by the client software either because it's in the destination headers, or by using mailing-list headers added by the mailing list software, instead of mangling the body or the subject.
[+] [-] callahad|9 years ago|reply
[+] [-] temprature|9 years ago|reply
This will make SPF pass but it has no effect on the DMARC result. For DMARC to use the SPF result, the envelope FROM needs to be aligned with the header From. Pretty much every mailing list already does what you've described but it doesn't help with DMARC.
> As for DKIM, you simply should not modify the message
Many mailing lists modify every message to add a footer with unsubscribe information. Others only modify some messages when necessary such as to convert HTML messages to plain-text, to strip attachments, to wrap lines to 72 characters etc.
[+] [-] phicoh|9 years ago|reply
[+] [-] gcp|9 years ago|reply
unsubscribe