The issue reported here is linked to App Engine and Gmail tightening up their spam filters. The root cause was an increase in organizations sharding out their spam systems to utilize App Engine’s free tier in such a way that is (a) in direct violation of our ToS and (b) making all of our lives suck a bit more (raise your hand if you want spam). It’s unfortunate that while App Engine is trying to provide a free tier that enables developers to easily use our platform, others see it as an opportunity for exploitation. Even more unfortunate is that it has a negative effect on legitimate users. It’s a fine balance that has been highlighted by several users within this thread.
Spam filtering is not a perfect science, and we’re constantly tweaking things -- with our customers in mind. This issue should be limited to new applications where the trust signal might be a bit lower. Thus existing apps / customers shouldn’t be experiencing issues (which was also highlighted by a few within this thread). If this isn’t the case email me: [email protected]. For those asking, “hey, why am I being penalized for being a new customer?” See my previous comment about spam filtering not being a perfect science. Then email me.
Just a quick thank you for the free tier that App Engine offers. I use it to provide an educational app for my wife, who's a teacher, that she uses as part of her classes. It's nice to be able to provide that to her and her class, without hitting the bank (the school does that enough...). So, thank you.
Long time App Engine user here. If you're sending email as part of your app, I'd recommend using Mailgun (or similar services). You can set up DKIM, inbound routes, etc, and generally have more control over how your mail is processed. I never got the impression App Engine really wanted to be in the email delivery business, so if that's important to you, there are tools that are focused on that (and free up to a monthly quota). My $0.02.
> This issue should be limited to new applications where the trust signal might be a bit lower. Thus existing apps / customers shouldn’t be experiencing issues.
1. My app is not new. It's been running without issue for 2 or 3 years.
2. My app is not on the free tier.
3. On average, my app sends out under 8 emails a day, and it's done the same for over 2 years.
How your algorithm considers my app a spam risk sure beats me.
I'm missing 11 days worth of quote requests from customers. Are these recoverable? (is there a hidden outgoing-spam bin?)
Possible reason that email service providers and other email sending services might see an increase in new signups and free tier use recently: http://blog.mandrill.com/important-changes-to-mandrill.html Mandrill in essence raised their price, eliminated free tier, and required prepurchased blocks of sending capacity IIRC
The numbers Mandrill released about that business suggest that they had a large number of low volume senders, who may now be looking for a new home.
Chris, you say Gmail tightened up their spam filters... Did this take place in the last ~6 weeks or so?
I've noticed the spam filter on Gmail for Google Apps has gotten significantly less accurate recently, resulting in far more false-positives than usual. Any ideas if this is a known issue? I can only presume it's more accurate overall, but it was definitely a noticeable change for our organisation.
Google Cloud Support here. To correct the misleading title, this is not a generalized outage of any kind, but an issue with some applications sending mail that trigger spam/malware/etc filters. Since spammers generally do not comply with RFC 3514, filtering is unfortunately not an exact science, so false positives happen and we're working with the customers in question on a case-by-case basis.
Also, App Engine is integrated with both Sendgrid and Mailgun, and we strongly recommend using these if you're planning on sending larger quantities of mail:
5 weeks ago your infrastructure stopped sending out mail. Our apps didn't change, your infrastructure did.
Whether it's your outgoing spam filter that's overzealous, or if a datacenter is on fire is irrelevant. Your infrastructure stopped doing what your docs said it would.
Paying customers like myself have apps running on your infrastructure. Your infrastructure stopped delivering the exact same emails that it had been delivering for years. It did this without notification and without triggering any errors.
Approximately 35 quote requests that should have been delivered have disappeared into thin air. You've cost me money, and I'm disappointed to find out that you'd been notified of this issue weeks before it began affecting me.
I wonder if people at Google know how awful it is to report an Appengine bug. You have to battle for days to get your bug acknowledged.
I've noticed that they've added more support people since the early days but it's still a pain as they seem to have an incentive to answer the tickets as fast as possible without doing any real investigation.
The linked issue is a good example of this behavior.
If you're running production services, you should probably sign up for a paid support package, which have guaranteed response times down to 15 minutes: https://cloud.google.com/support/
Reading through the thread, it looks like in the last few days it's been narrowed down to "messages which contain the URL of the App Engine instance silently disappear with no bounces or errors."
Maybe I'm old-fashioned but if I was affected by this issue (and it had persisted for so long) I would set up an SMTP server and an appropriate MX record, then demonstrate using mail logs or port snooping that this was definitely not a receive-side issue.
Of course the support organisation here (Google) should be well equipped to set up this kind of testing themselves and work with a customer to root cause the issue. In my experience though these issues are rarely taken seriously until there is no-one else left for support to blame.
I had a Google engineer tell me once that the App Engine email api was never meant to send a lot of mail. That is why the quota is so low. I would recommend using a third party service to do e-mail.
I'm pretty surprised by the long wait times between "updates" from support... that's pretty rough. I understand it can be hard to get a good signal to noise ratio with support, but when so many other people are reporting issues, perhaps it's time to take it a bit more seriously?
This thread is a great example of why support is so hard.
not really. I wish I could get away with providing such crap support as the big players do.
If my customers don't get a reply within a few hours, then all hell breaks loose. Even if, there request comes in at 3am / 0300 on a Sunday morning my time.
To be fair, when I point this out to them they do apologise, didn't realise time difference, are in a different part of the world where Sunday is not a day off etc. I then ask what they expect from some billion dollar companies support wise and they say they are happy with a reply within a week. Asking them why a one man shop doing quite a bit less than billions of dollars is expected to provide so much more.
Anyway, rant over. Support isn't hard. You hire enough competent people as your customers require. The end. Happy customers
Must be an isolated issue. I run two App Engine apps and send 30K emails a day and I haven't noticed any change in my email open/click stats, which implies that emails are being delivered as usual.
The problem is that pesky humans can't decide what they want. Not sending mail is a bug but not receiving mail (i.e. spam filtering) is a feature? I can appreciate how this is confusing.
Yeah, but pricing is ten times Appengine's (actually more - recently we don't seem to be being charged for email at all). Amazon SES is the only service with a comparable cost that I've found. I'd love to hear of other alternatives at the 1c/100 level.
At least a few reported that it was bouncing at the destination once they set up an appropriate handler. They are only "not being sent" when there is a url in the message and that is a big red flag on the receiving side (not much text but a url). It doesn't look like an App Engine problem. It looks like messages being identified as spam problem.
I ran into a similar problem a while back (using Heroku) --- after a brief foray into the world of spam filtering, it turned out to be caused by our domain being identified by SpamAssassin (IIRC) as a spam signal. We changed the wording of our emails and moved to a different subdomain and suddenly all the email got through. It was.. interesting to debug.
This is what happens when you buy a service from an organization oriented toward ad-supported services. The mindset is that it doesn't matter if the user isn't being served.
[+] [-] cramsdale|10 years ago|reply
The issue reported here is linked to App Engine and Gmail tightening up their spam filters. The root cause was an increase in organizations sharding out their spam systems to utilize App Engine’s free tier in such a way that is (a) in direct violation of our ToS and (b) making all of our lives suck a bit more (raise your hand if you want spam). It’s unfortunate that while App Engine is trying to provide a free tier that enables developers to easily use our platform, others see it as an opportunity for exploitation. Even more unfortunate is that it has a negative effect on legitimate users. It’s a fine balance that has been highlighted by several users within this thread.
Spam filtering is not a perfect science, and we’re constantly tweaking things -- with our customers in mind. This issue should be limited to new applications where the trust signal might be a bit lower. Thus existing apps / customers shouldn’t be experiencing issues (which was also highlighted by a few within this thread). If this isn’t the case email me: [email protected]. For those asking, “hey, why am I being penalized for being a new customer?” See my previous comment about spam filtering not being a perfect science. Then email me.
We’re here and we want to help.
-- Chris (Lead PM for App Engine)
[+] [-] fpgaminer|10 years ago|reply
[+] [-] brianmcconnell|10 years ago|reply
[+] [-] markdown|10 years ago|reply
> This issue should be limited to new applications where the trust signal might be a bit lower. Thus existing apps / customers shouldn’t be experiencing issues.
1. My app is not new. It's been running without issue for 2 or 3 years.
2. My app is not on the free tier.
3. On average, my app sends out under 8 emails a day, and it's done the same for over 2 years.
How your algorithm considers my app a spam risk sure beats me.
I'm missing 11 days worth of quote requests from customers. Are these recoverable? (is there a hidden outgoing-spam bin?)
My app started sending mail again today after I changed the src of an image in it from https://example.appspot.com/images/logo.png to http://www.example.com/images/logo.png
How well do you think email clients are going to like emails with images embedded in them without HTTPS?
[+] [-] jcrites|10 years ago|reply
The numbers Mandrill released about that business suggest that they had a large number of low volume senders, who may now be looking for a new home.
[+] [-] je42|10 years ago|reply
Or are you afraid that "rogue" applications will use it to produce messages that are SPAM but yet not trigger the SPAM filter ?
[+] [-] JackWebbHeller|10 years ago|reply
I've noticed the spam filter on Gmail for Google Apps has gotten significantly less accurate recently, resulting in far more false-positives than usual. Any ideas if this is a known issue? I can only presume it's more accurate overall, but it was definitely a noticeable change for our organisation.
[+] [-] jpatokal|10 years ago|reply
Also, App Engine is integrated with both Sendgrid and Mailgun, and we strongly recommend using these if you're planning on sending larger quantities of mail:
https://cloud.google.com/appengine/docs/python/mail/sendgrid
https://cloud.google.com/appengine/docs/python/mail/mailgun
[+] [-] markdown|10 years ago|reply
Whether it's your outgoing spam filter that's overzealous, or if a datacenter is on fire is irrelevant. Your infrastructure stopped doing what your docs said it would.
Paying customers like myself have apps running on your infrastructure. Your infrastructure stopped delivering the exact same emails that it had been delivering for years. It did this without notification and without triggering any errors.
Approximately 35 quote requests that should have been delivered have disappeared into thin air. You've cost me money, and I'm disappointed to find out that you'd been notified of this issue weeks before it began affecting me.
[+] [-] martindelemotte|10 years ago|reply
I've noticed that they've added more support people since the early days but it's still a pain as they seem to have an incentive to answer the tickets as fast as possible without doing any real investigation.
The linked issue is a good example of this behavior.
[+] [-] jpatokal|10 years ago|reply
If you're running production services, you should probably sign up for a paid support package, which have guaranteed response times down to 15 minutes: https://cloud.google.com/support/
Disclaimer: I work in Google Cloud Support.
[+] [-] stcredzero|10 years ago|reply
No. (Or they don't care.)
[+] [-] spicyj|10 years ago|reply
[+] [-] kordless|10 years ago|reply
[+] [-] fencepost|10 years ago|reply
[+] [-] achant|10 years ago|reply
[+] [-] markdown|10 years ago|reply
It was sending an HTML email with the appspot domain used for the logo: https://example.appspot.com/images/logo.png
Changing that to http://www.example.com/images/logo.png fixed the problem, but now I have to host the image elsewhere on a HTTPS enabled domain.
A separate question is what has google done with all the messages that users of this app sent? Are they gone for good, or stuck in a queue somewhere?
[+] [-] tominous|10 years ago|reply
Of course the support organisation here (Google) should be well equipped to set up this kind of testing themselves and work with a customer to root cause the issue. In my experience though these issues are rarely taken seriously until there is no-one else left for support to blame.
[+] [-] cyberpanther|10 years ago|reply
[+] [-] ajackfox|10 years ago|reply
This thread is a great example of why support is so hard.
[+] [-] user_0001|10 years ago|reply
If my customers don't get a reply within a few hours, then all hell breaks loose. Even if, there request comes in at 3am / 0300 on a Sunday morning my time.
To be fair, when I point this out to them they do apologise, didn't realise time difference, are in a different part of the world where Sunday is not a day off etc. I then ask what they expect from some billion dollar companies support wise and they say they are happy with a reply within a week. Asking them why a one man shop doing quite a bit less than billions of dollars is expected to provide so much more.
Anyway, rant over. Support isn't hard. You hire enough competent people as your customers require. The end. Happy customers
[+] [-] waleedka|10 years ago|reply
[+] [-] Maxious|10 years ago|reply
Email is hard.
[+] [-] pmlnr|10 years ago|reply
[+] [-] chris_wot|10 years ago|reply
[+] [-] awinter-py|10 years ago|reply
[+] [-] mikecb|10 years ago|reply
[+] [-] foxylad|10 years ago|reply
[+] [-] SuperKlaus|10 years ago|reply
[+] [-] daveguy|10 years ago|reply
[+] [-] markdown|10 years ago|reply
Nothing whatsoever changed in my code. I haven't touched it in a year, and all of sudden email stopped on the 1st of April.
There's nothing in the changelogs to indicate there was a change that should cause this behaviour. It's absolutely an App Engine problem.
https://cloud.google.com/appengine/docs/python/release-notes
[+] [-] jlees|10 years ago|reply
[+] [-] hartator|10 years ago|reply
[+] [-] mattbillenstein|10 years ago|reply
[+] [-] markdown|10 years ago|reply
[+] [-] unknown|10 years ago|reply
[deleted]
[+] [-] samskeller|10 years ago|reply
[+] [-] tedmiston|10 years ago|reply
Or not.
[+] [-] Animats|10 years ago|reply