"The fundamental issue is that SendGrid’s business model depends on making it easy for legitimate businesses to send email at scale."
I disagree with this conclusion, if not only because other email service providers don't have this issue.
It wouldn't surprise me if something was broken with SendGrid's internal infrastructure. I used to be a SendGrid customer until my deliverability started being affected by this issue. SendGrid took weeks to reply to my customer service messages about resolving this, even though I was a paying customer and was renting private IP addresses from them to send mail.
I finally gave up and closed my SendGrid account in July 2021. Despite this, they continued to send me monthly invoices until May 2022. Multiple SendGrid representatives promised that they had resolved the issue, but it wasn't until one CSR added me to SendGrid's global suppression list that they finally stopped.
>closed my SendGrid account ....continued to send me monthly invoices
I used to run IT for a medium company. The amount of times I saw this with various SaaS companies was troubling. We had hundreds of services some as small as a single manager that demanded X and company wide tools. It was frequently a several months long hassle to get them to stop billing us when we cut ties with them. I wish I kept personal records now it was a minority but definitely in the 15%'ish range.
If the attackers in this case are cleverly exploiting anything, I would bet on aggressive grey patterns like that more than I would US culture wars. Noticing that a company has policies that let you hide in plain sight means that you're paying close attention. Knowing what issues are hot button culture flamewars means you can access literally any American news outlet.
For popular senders: sort-of: in your incoming mail server, substring-match the display name of the sender against popular brands, and ensure the actual domain matches.
This works remarkably well for proper brands (FedEx et al), but breaks down when the brand name regularly occurs in "normal" names, the sending brand sends mail from all over the place, or "innocuous" impersonation takes place all the time.
Like, somehow, From: "VODAFONE" <shipping-update@dpd.co.uk> is a 100% legit sender (assuming SPF and DKIM verification pass), despite both Vodafone and DPD being pretty common impersonation targets. You'd think they'd know better, but alas.
Making a custom rule for a specific sender feels like fighting a fire with a glass of water.
It's better to focus on more systematic solutions. There exist a lot of them, SPF, DKIM, Recipient mail filtering (Your mail provider).
The screenshotted emails don't even do anything tricky like spoofing the sender address, it looks like "Sent from no-reply@theraoffice.com". If it spoofed the domain it would have been caught by SPF/DKIM.
Most of the time the user doesn't need to do much, you can just be weary of sender domains, and report the email as phishing and help blacklist that specific IP address/domain. Similar to how in medicine sometimes the physician tells you to drink water and rest, no medicine needed, just let the immune system do its thing.
It seems like Twilio has a conflict of interest that prevents them from offering WebAuthn, as that would be a tacit admission that their SMS and Authy products are not actually that secure.
rich irony that twilio numbers don't qualify to receive SMS codes when senders check if it's a virtual number (the regulated aka important ones do check)
Having a friendly name listed in the From field is part of the problem. SPF, DKIM, and DMARC make it possible to control who can send as your domain, if the receiver cares to check. If you have strict SPF and DMARC rules, most receivers will drop or not accept emails that fail the rules. But you can't control using your brand from unaffiliated domains.
Would you even open an email from noreply@drummond.com if that's what showed up in the message list?
On mobile it's worse. Gmail (Android) doesn't even show the From address at all when you open an email. For some emails, I can tap the sender icon and see the address, for others I have to find the hit reply (but if DMARC et al doesn't validate a Reply-To address) or go find a computer and see the message there.
SendGrid phishing emails are some of the best phishing emails. I get emails that there's elevated error rates on an API (`/v1/send`). Looks very legit, good design, reasonable call to action, some urgency which makes me want to click. They know from MX records I send email with Sendgrid, so it's well targeted. Easy catch when I see the domain, but other than that it's the best I've seen in years.
We've been getting similar phishing emails claiming to be from SendGrid, except they're along the lines of "we're adding a rainbow banner to the footer of all emails to show LGBT support, click here to opt out".
It's especially funny because SendGrid isn't even one of our vendors.
First thought... Why would ICE need donations? I then realized how unrecognizable scams have become to me now. Older people are going to be in a worse position.
relatedly, my wife received polititexts destined to her conservative father. The latest was actually genius IMO, in that it stated "Dear STEVEN, due to inactivity, your registration will be changed to DEMOCRAT in 20 minutes unless you navigate to this link." It, I assume, redirected to some support page to donate to the US conservative party or its affiliates. The social engineering is getting more effective
I don't know if the fact that it fully slipped into the absurd or the fact that it probably still worked on people is sadder.
I do love the idea of voter registration oscillating back and fourth at 20 minutes intervals forever. Would make voting in the primaries way more exciting as the voter base kept flipping.
Inevitably some people are going to be away from their phones when they receive that, so I wonder what they think when they continue getting needy messages from Republicans after that!
I can't think of one email I received from sendgrid I would consider legitimate. Anytime I receive an email distributed by sendgrid I have found it actually had no value to me. Sometimes it's from a business I have dealt with but I never wanted or was interested in the content.
Do you specifically go out of your way to check who sent every transactional email you receive and take notes on which email sending service your order confirmation was sent by? That would be a very weird thing to do and would be the only way to know that.
I’m more troubled by the fact these emails are hitting my sendgrid only email address.
Is this related to the breach that SendGrid said didn’t happen? I set my account up in 2021 for reasons I don’t recall and it’s since been deleted/deactivated by them.
> The political sophistication on display here (BLM, LGBTQ+ rights, ICE, even the Spanish language switch playing on immigration anxieties) suggests someone with a deep understanding of American cultural fault lines.
I received one, though it was for adding a footer honoring MLK. I kinda thought it was odd, but did't think much of it, since I'm apparently not in the group that would be offended in any way. I wonder if the variation they use is random, or in any way location-based to maximize response (I'm in Texas).
I've also received a bunch of API failure phishing emails, as well as some implying we needed to change our auth to Sinch.
It would be good to hold carriers accountable for fishing and spam. Sendgrid , Twilio and other saas messaging carriers need to do a better job with integrity. I don’t expect them to carry the whole burden, but some negative incentive to promote investment . It could be as simple as enforcing sender pays metering . We all know spam is 60+ % of traffic, so sender pays would drive down spam very quickly
SendGrid and their competitors are already the very definition of “sender pays” for email. “Sender pays” is how they make money. This isn’t a problem of monetary incentives.
The problem is that companies get their SendGrid credentials compromised via password re-use or phishing.
I wonder why Gmail and other email providers don't just run an LLM/ML pipeline to detect phishing emails. It seems that matching an email's content with the sender's domain (and possibly analyzing the content behind links) would be enough to show, with high certainty, a warning like "Beware: this looks like a phishing email." Is it too expensive? Too many false positives?
I think you're about 20 years behind the times if you think they don't.
There are a whole lot of problems with it when you start pressing the finer details like you list. For example, just look at the legit emails banks send out. They will tell you not to click links claiming to be your bank, then include links (claiming to be your bank) for more information.
Simply put the rules block too much corporate email because people that write corporate email do lots of dumb things with the email system.
The most essential check is SPF and DKIM which authenticate if the message has come from an authorized server. The problem is that most mail services are too lenient with mismatched sender identification. On one hand, people would be quite vocal about their mail provider sending way too much legitimate (but slightly misconfigured) mail to the spam folder. However it allows situations like to happen where the FROM header, the "From:" address, and the return path are all different.
Most mail systems have several stages of filters, and the first ones (checking authentication) are quite basic. After that, attachments, links, and contents are checked for known malware. Machine learning might kick in after this, if certain criteria are met. Mail security is very complicated and works well except for the times it falls flat on its face like this.
I get a flood of these every single day. Because we use SendGrid as a critical part of our product, I have to look for any emails from them pretty closely. It’s gotten impossible to do with all of these phishing attempts. I gotta hand it to them, though, the attempts are excellent.
The OP didn’t explain or showed the unsubscribe button compromise trick. Anyone here can shed some light on it?
I always had the habit of clicking on the unsubscribe button whenever I see an unwanted email. And I’d like to know what would happen if I click on malicious unsubscribe link.
Is this a new trend in phishing emails? They appear to be using legitimate domains to bypass spam detection. Usually the domains are associated with legitimate companies who are completely oblivious. I always wondered how this works. Is it a broken contact form somewhere?
One way is to look for companies that have SPF records (or whatever the system is these days) that contain ranges/names of large providers like sendgrid. Then they test sending mails with those large providers names under said system until they get ones that go out, and launch a campaign.
the article talked about how the sendgrid accounts are real, and presume compromised.
I suspect that once the sendgrid account is compromised, they then send out these phishing emails, hoping to compromise _other_ sendgrid accounts to look for password overlap and/or keep the flow going.
Not just SendGrid, I have received very sophisticated phishing emails “from” MailGun as well. I think the advantages of getting into your email channel justify a lot of investment by the bad guys.
Interesting that politics is a vector for contagion.
When you think about politics is very contagious, politicians infect activists, who infect regular folk that advocate for stuff they don't benefit from, when elections come near, it's flu season.
Double parasite burgers where a new parasite leeches of an existing vector are common in biology as well. Like malaria and mosquitoes.
Philosophically fun, sure, but the article also points out that another vector was "Your language settings have been changed to Spanish", so I don't know if it's as profound as you're making it out to be. Anything that makes us panic can be a vector.
> We know that state actors have invested heavily in understanding and exploiting these divisions. Russian active measures campaigns have been documented doing exactly this kind of work: identifying wedge issues and creating content designed to inflame both sides. North Korea has demonstrated similar sophistication in their social engineering operations by targeting academics and foreign policy experts
What about "read Twitter in between bouts of using one susceptible user's API key to spam other users for their API keys" _really_ requires the sophistication of a state-level actor? Statements like this aren't journalism, they're exactly the same kind of manipulation being used by the phishers.
Before anyone launches themselves into the sky: the title is clickbait. This is about phishing attempts that use ICE to persuade you to click. Sendgrid the company is not emailing about supporting ICE. But technically Sendgrid the infrastructure is.
Author here. I quickly thought of the title for the article and shipped it. I agree it's clickbait-y and apologize to SendGrid (and any confused readers) but yes, as you say it's _technically_ correct in a very narrow sense – SendGrid's infrastructure and users are sending these emails, it's just that they're fraudulently associated with SendGrid the company.
In any case, I revised the title to "SendGrid isn’t emailing you about ICE or BLM. It’s a phishing attack."
Maybe someone can edit the title of the submission on HN accordingly?
I think HN should embrace AI to the point of having an alternative AI-generated title next to the original title, to reduce clickbait and reduce the global rage index.
Maybe one day our knee jerk reactionary outrage will be quelled not by any enlightenment but because we are forced to grow weary of falling prey to phishing attacks.
I'd feel pretty stupid getting worked up about something only to realize that getting worked up about it was used against me.
I'm writing this because for a moment I did get worked up and then had the slow realization it was a phishing attack, slightly before the article got to the point.
Anyways, I think the clickbait is kindof appropriate here because it rather poignantly captures what is going on.
The title is genius; it uses the same psychological trick as the phishers are, to point out to us how vulnerable we are. Obviously, for you to know the title is clickbait, you'd've had to click through and read it, which is the exact social engineering vulnerability the author is trying to demonstrate being exploited.
I thank the author for getting me this way, as I would have likely fallen for the unsubscribe trick.
right, so on the topic of "phishing emails designed to elicit enough emotion that you forget to consider the button might be a phish", the headline itself of this blog post is doing the exact same thing, really. The headline should be, "Phishing scams launched through SendGrid exploit deep political sentiments to achieve success" or something like that.
but that would be clear and very boring. nobody would read your blog then. A headline that very obviously implies Sendgrid the company supports ICE, and so much so that they are emailing all their customers about it, clicks galore. Well done.
martey|1 month ago
I disagree with this conclusion, if not only because other email service providers don't have this issue.
It wouldn't surprise me if something was broken with SendGrid's internal infrastructure. I used to be a SendGrid customer until my deliverability started being affected by this issue. SendGrid took weeks to reply to my customer service messages about resolving this, even though I was a paying customer and was renting private IP addresses from them to send mail.
I finally gave up and closed my SendGrid account in July 2021. Despite this, they continued to send me monthly invoices until May 2022. Multiple SendGrid representatives promised that they had resolved the issue, but it wasn't until one CSR added me to SendGrid's global suppression list that they finally stopped.
citizenpaul|1 month ago
I used to run IT for a medium company. The amount of times I saw this with various SaaS companies was troubling. We had hundreds of services some as small as a single manager that demanded X and company wide tools. It was frequently a several months long hassle to get them to stop billing us when we cut ties with them. I wish I kept personal records now it was a minority but definitely in the 15%'ish range.
pksebben|1 month ago
ZeroConcerns|1 month ago
For popular senders: sort-of: in your incoming mail server, substring-match the display name of the sender against popular brands, and ensure the actual domain matches.
This works remarkably well for proper brands (FedEx et al), but breaks down when the brand name regularly occurs in "normal" names, the sending brand sends mail from all over the place, or "innocuous" impersonation takes place all the time.
Like, somehow, From: "VODAFONE" <shipping-update@dpd.co.uk> is a 100% legit sender (assuming SPF and DKIM verification pass), despite both Vodafone and DPD being pretty common impersonation targets. You'd think they'd know better, but alas.
So, yeah, room for improvement and such...
layer8|1 month ago
And/or, long-press or right-click on any link to inspect the linked domain.
rphillips|1 month ago
1. Add expressions to: If ALL of the following match the message.
2. Expression 1: Type: Advanced content match Location: Full headers Match type: Matches regex (?im)^from:\sSendGrid(?:\s+\w+)\s*<[^>\r\n]+>+$
3. Expression 2: Type: Advanced content match Location: Sender header Match type: Not matches regex (?i)^[A-Za-z0-9._%+-]+@(sendgrid\.com|twilio\.com)$
Set the rule to reject or quarantine. Users will not see the messages unless the attackers change the From header.
TZubiri|1 month ago
It's better to focus on more systematic solutions. There exist a lot of them, SPF, DKIM, Recipient mail filtering (Your mail provider).
The screenshotted emails don't even do anything tricky like spoofing the sender address, it looks like "Sent from no-reply@theraoffice.com". If it spoofed the domain it would have been caught by SPF/DKIM.
Most of the time the user doesn't need to do much, you can just be weary of sender domains, and report the email as phishing and help blacklist that specific IP address/domain. Similar to how in medicine sometimes the physician tells you to drink water and rest, no medicine needed, just let the immune system do its thing.
agwa|1 month ago
It seems like Twilio has a conflict of interest that prevents them from offering WebAuthn, as that would be a tacit admission that their SMS and Authy products are not actually that secure.
Terretta|1 month ago
toast0|1 month ago
Would you even open an email from noreply@drummond.com if that's what showed up in the message list?
On mobile it's worse. Gmail (Android) doesn't even show the From address at all when you open an email. For some emails, I can tap the sender icon and see the address, for others I have to find the hit reply (but if DMARC et al doesn't validate a Reply-To address) or go find a computer and see the message there.
scosman|1 month ago
creeble|1 month ago
I only used a SendGrid account briefly, as a potential backup to my current outgoing transaction mail provider. Sent exactly 5 test emails I think.
The ICE one this morning gave me pause, but only about 2s before I deleted it and moved on with my busy day of reading HN posts.
parliament32|1 month ago
It's especially funny because SendGrid isn't even one of our vendors.
itintheory|1 month ago
zzzeek|1 month ago
forthwall|1 month ago
nashashmi|1 month ago
david_shaw|1 month ago
https://www.pay.gov/public/form/start/23779454
sig-term|1 month ago
SketchySeaBeast|1 month ago
I do love the idea of voter registration oscillating back and fourth at 20 minutes intervals forever. Would make voting in the primaries way more exciting as the voter base kept flipping.
mcintyre1994|1 month ago
nathanyz|1 month ago
detourdog|1 month ago
plorkyeran|1 month ago
SoftTalker|1 month ago
gadgetoid|1 month ago
Is this related to the breach that SendGrid said didn’t happen? I set my account up in 2021 for reasons I don’t recall and it’s since been deleted/deactivated by them.
xg15|1 month ago
Or an AI.
bdcravens|1 month ago
I've also received a bunch of API failure phishing emails, as well as some implying we needed to change our auth to Sinch.
schnable|1 month ago
tonymet|1 month ago
tatersolid|1 month ago
The problem is that companies get their SendGrid credentials compromised via password re-use or phishing.
kgeist|1 month ago
pixl97|1 month ago
I think you're about 20 years behind the times if you think they don't.
There are a whole lot of problems with it when you start pressing the finer details like you list. For example, just look at the legit emails banks send out. They will tell you not to click links claiming to be your bank, then include links (claiming to be your bank) for more information.
Simply put the rules block too much corporate email because people that write corporate email do lots of dumb things with the email system.
yabones|1 month ago
The most essential check is SPF and DKIM which authenticate if the message has come from an authorized server. The problem is that most mail services are too lenient with mismatched sender identification. On one hand, people would be quite vocal about their mail provider sending way too much legitimate (but slightly misconfigured) mail to the spam folder. However it allows situations like to happen where the FROM header, the "From:" address, and the return path are all different.
Most mail systems have several stages of filters, and the first ones (checking authentication) are quite basic. After that, attachments, links, and contents are checked for known malware. Machine learning might kick in after this, if certain criteria are met. Mail security is very complicated and works well except for the times it falls flat on its face like this.
https://en.wikipedia.org/wiki/Sender_Policy_Framework https://en.wikipedia.org/wiki/DomainKeys_Identified_Mail
zahlman|1 month ago
flatcakes|1 month ago
jscheel|1 month ago
alila|1 month ago
I always had the habit of clicking on the unsubscribe button whenever I see an unwanted email. And I’d like to know what would happen if I click on malicious unsubscribe link.
sakopov|1 month ago
pixl97|1 month ago
lbotos|1 month ago
I suspect that once the sendgrid account is compromised, they then send out these phishing emails, hoping to compromise _other_ sendgrid accounts to look for password overlap and/or keep the flow going.
unknown|1 month ago
[deleted]
educasean|1 month ago
Is this a UX issue? Should email clients highlight and emphasize the sender domain more than their display name?
schnable|1 month ago
yes
snowwrestler|1 month ago
TZubiri|1 month ago
When you think about politics is very contagious, politicians infect activists, who infect regular folk that advocate for stuff they don't benefit from, when elections come near, it's flu season.
Double parasite burgers where a new parasite leeches of an existing vector are common in biology as well. Like malaria and mosquitoes.
SketchySeaBeast|1 month ago
unknown|1 month ago
[deleted]
eps|1 month ago
It might be 50 days by an (admittedly very cool) bus, but it's only 84 days in foot!
* Consult your Google Maps and a sense of humor if it sounds to good to be true!
amw|1 month ago
> We know that state actors have invested heavily in understanding and exploiting these divisions. Russian active measures campaigns have been documented doing exactly this kind of work: identifying wedge issues and creating content designed to inflame both sides. North Korea has demonstrated similar sophistication in their social engineering operations by targeting academics and foreign policy experts
What about "read Twitter in between bouts of using one susceptible user's API key to spam other users for their API keys" _really_ requires the sophistication of a state-level actor? Statements like this aren't journalism, they're exactly the same kind of manipulation being used by the phishers.
lbrito|1 month ago
ekjhgkejhgk|1 month ago
idiotsecant|1 month ago
unknown|1 month ago
[deleted]
losthobbies|1 month ago
afavour|1 month ago
mecredis|1 month ago
In any case, I revised the title to "SendGrid isn’t emailing you about ICE or BLM. It’s a phishing attack."
Maybe someone can edit the title of the submission on HN accordingly?
sys32768|1 month ago
santadays|1 month ago
I'd feel pretty stupid getting worked up about something only to realize that getting worked up about it was used against me.
I'm writing this because for a moment I did get worked up and then had the slow realization it was a phishing attack, slightly before the article got to the point.
Anyways, I think the clickbait is kindof appropriate here because it rather poignantly captures what is going on.
indigodaddy|1 month ago
buellerbueller|1 month ago
I thank the author for getting me this way, as I would have likely fallen for the unsubscribe trick.
isk517|1 month ago
cheald|1 month ago
darth_avocado|1 month ago
ExpertAdvisor01|1 month ago
zzzeek|1 month ago
but that would be clear and very boring. nobody would read your blog then. A headline that very obviously implies Sendgrid the company supports ICE, and so much so that they are emailing all their customers about it, clicks galore. Well done.
unknown|1 month ago
[deleted]
ProofHouse|1 month ago
[deleted]
cindyllm|1 month ago
[deleted]