top | item 5414866

Sendgrid is down

103 points| sunils34 | 13 years ago |support.sendgrid.com | reply

97 comments

order
[+] grey-area|13 years ago|reply
I'm sad to see that HN is being used as a platform for this sort of demagogy, and sad that stories about this tawdry little tale are indiscriminately voted up above other stories which are far more interesting.

DDOS of a mailing service that lots of websites rely on because a completely unrelated company decided to fire someone is not an occasion for lol and schadenfreude as some posters here would have it. As a method of justice it has more in common with a lynch mob than a court of law - this isn't going to get the guy's job back, and it certainly isn't going to teach anyone a lesson, apart from that the internet is fickle, and monumentally stupid. But I very much doubt the people behind this attack are interested in justice or truly care about the man who lost his job, they're just doing it for the lolz and are punishing the internet at large over a silly little dispute at a tech conference.

Congratulations to the mob, I guess; it has shown its power, if not any sense of discrimination or proportionality.

[+] kybernetyk|13 years ago|reply
Now you can argue as much as you want - but the (sad) truth is that mob mentality exists and we will have to deal with it for at least the next few centuries.

This should be a lesson for companies that hire highly confrontational people as their official community representatives.

Yes, in a perfect world there shouldn't be DDOS' and other attacks because of a tweet about an immature joke but it's not a perfect world and we should be very wary what personalities we hire to represent us.

As much as I don't like it - the "right" thing for sengrid would be to replace their 'developer evangelist' with someone who's less confrontational. Yes, it sucks. But as businesses we have to deal with reality.

[+] obstacle1|13 years ago|reply
>As a method of justice it has more in common with a lynch mob than a court of law

You mean like when someone gets personally offended by a comment and instead of resolving it one-on-one turns to her Twitter account to shame and blame the "violators"?

Yeah. Except the person who instigated this set up the lynching in her capacity as a professional, i.e. on her "evangelist" twitter account. The kids DDOSing aren't representing companies or doing business

[+] Lewton|13 years ago|reply
Sometimes something happens where posts on hacker news are part of the debacle[1] and it results in an influx of new users. I don't want to sound like an old coot, but I'm a bit concerned about the type of user this thing has been attracting

[1] in this case, the guy who was called out for the jokes, posted in a comment on here that he was fired

[+] untog|13 years ago|reply
OK, is there any actual evidence that this is a DDOS of SendGrid? Because their status page indicates nothing of the sort, and all I'm seeing is HN jumping to conclusions.
[+] adambenayoun|13 years ago|reply
I couldn't agree with you more on the subject of what happened with PyCon. I don't think it should be mixed with what is happening now.

On the flip side, I'm happy that this story has been up-voted and that it attracted some attention because I was able to know that Sendgrid is down (never received any mail notification and I don't follow them on Twitter).

[+] fmavituna|13 years ago|reply
> ... this isn't going to get the guy's job back, and it certainly isn't going to teach anyone a lesson

As much as I don't support the methods, seems like this actually sends a message (right or wrong). So it perfectly works from attacker's perspective.

I'm pretty sure by now every critical person in Sendgrid know the details of what happened, why happened and people involved. Not because of the initial twitter discussions but because this actually costs them money directly.

[+] aculver|13 years ago|reply
This was a Heroku win for us. I just visited their add-ons page, identified a comparable provider, and restored mail in our application with the following:

    heroku addons:add mailgun:basic
    heroku addons:docs mailgun:basic
    vim config/initializers/mail.rb
    git add config/initializers/mail.rb
    git commit -m "Switching to back-up mail provider."
    git push heroku master
Since the account creation is all automatic and billing is all through Heroku, I never even had to visit Mailgun's website.
[+] adambenayoun|13 years ago|reply
How do you handle SPF/DKIM and domain keys records? Do you send out a lot of emails? Have you had a mailgun account ready?
[+] slig|13 years ago|reply
Even though I'm a SG customer, I can't help but feel a little schadenfreude.

edit: when I made this comment I thought this was a random service failure that would last couple of minutes. More than 1 hour later, I don't think it's that funny anymore as I'm being affected as well.

[+] rdl|13 years ago|reply
Schadenfreude is, however, not an RFC compliant message delivery protocol.
[+] efa|13 years ago|reply
SG customer as well. What's all the negativity towards them? Is there something I should know about them? I wasn't aware of any incidents.
[+] robflynn|13 years ago|reply
Same here. I just flipped the switch to send things from an internal server, bypassing SG for now.
[+] rmrfrmrf|13 years ago|reply
Really? All it says to me is that people in the tech field are so against a woman standing up for herself that they're actually willing to bring down the company she works for. It's disgusting.

Will you also feel schadenfreude when they hack her bank account?

[+] adambenayoun|13 years ago|reply
Sendgrid customer here.

Since many services rely on SMTP providers like Sendgrid, they should have a way to notify customers when their server go down and transactional notifications may be disrupted.

I shouldn't be notified by someone who know we're using Sendgrid and happen to read HN.

[+] rdl|13 years ago|reply
Yeah -- outage notifications to a service's users should go out, rather than just being pulled.

Also seems like best practice would be to have SPF/etc. entries in place for multiple ESPs, even if you routinely just use one, and be able to switch, for just this reason.

[+] mryan|13 years ago|reply
They've been tweeting about it for an hour or so: https://twitter.com/SendGrid

I would expect them to send emails to customers in the event of serious issues though, expecting your customers to learn about problems through Twitter isn't that much better than learning about them from HN.

[+] whaevr|13 years ago|reply
Aaand the PR slide for SendGrid continues. Perfectly valid point though.
[+] jlebrech|13 years ago|reply
how does one send outage notices if the mail is down?
[+] lifeisstillgood|13 years ago|reply
OK, I'll bite - is this a coincidence - or are they being DDOS'd over the PyCon incident?
[+] joshrotenberg|13 years ago|reply
The dreaded FDDOS: Forking Dongle Denial of Service.
[+] uptown|13 years ago|reply
I'm surprised PlayHaven hasn't shared the blame for their role in this whole thing. Adria, and by-extension, SendGrid shoulder some of the responsibility for what happened, but PlayHaven chose not to back their own employee.
[+] whaevr|13 years ago|reply
It seems that Adria's personal blog is having similar problems..

http://butyoureagirl.com/

[+] canttestthis|13 years ago|reply
Yes she mentioned that her website was getting DDOSed and that she was going to put it behind Cloudflare. I'm not sure its a DDOS though, it could just be the amount of traffic she's getting from HN/Reddit/Twitter/etc.

https://news.ycombinator.com/item?id=5400134

EDIT: It seems that her personal website (adriarichards.com) is down too.

[+] jneal|13 years ago|reply
As a SendGrid customer, I'm certainly glad I visit HN frequently and was able to find out about this. I'm surprised we weren't notified by them directly, though.
[+] markeganfuller|13 years ago|reply
We've never been directly notified by them despite their SMTP breaking a number of times.
[+] entropyneur|13 years ago|reply
First we replaced SMTP that was distributed and designed to be resilient to outages (i.e. not lose messages) with proprietary HTTP APIs and now we complain that they don't notify us via email when they are down. Nice.

I am guilty of using SES myself, but it's sad to see email becoming increasingly centralized.

[+] symfoniq|13 years ago|reply
We use SendGrid for transactional email, but handle (solicited) bulk email in-house. The centralization of email is at least partly due to the fact that ensuring email gets delivered is difficult and time-consuming.

While I'm sure a lot of SendGrid's customers don't want (or don't know how) to configure a mail server, there are other customers who know that delivering email isn't as simple as installing Postfix. The rise of centralized email services is the inevitable byproduct.

[+] benatkin|13 years ago|reply
Actually SendGrid recommends using SMTP. And their API endpoint is very SMTP-like.

As services go, its lock-in seems very minimal.

[+] plusbryan|13 years ago|reply
FWIW, we use Amazon SES and enjoy it immensely.
[+] andyl|13 years ago|reply
Great for Adria: more blog hits. Not so great for SendGrid: a PR disaster.
[+] unhe|13 years ago|reply
all thnx to adria and her cyber bullying
[+] Kudos|13 years ago|reply
All thanks to a dickhead with a botnet