Dropbox Cofounder & CTO Arash Ferdowsi responds to yesterday's bug

[rdl]

I think Dropbox would benefit from bringing in a competent PR person to advise them on making this kind of announcement and communicating with customers. (which is saying a lot, since I generally disdain PR people). Commenting on a problem you've caused people without actually apologizing is kind of a dick move. Even the US military does a better job when it accidentally kills people!

The statements should come from the top, but should probably only be written by founders if they're actually good at this kind of thing.

There are a lot of companies who are relatively good at crisis communications with affected parties. Network carriers are often good (at least when talking to other networking professionals), although they often try to restrict the spread of information via NDA.

There are some reasons to limit disclosure immediately after an incident (if you think the vulnerability still exists and might put you at risk, especially for a security threat and not a regular outage). You absolutely want to include "why this won't happen again" in your message (which Arash kind of did this time), but you also want to accept blame at an emotional level -- you can do this in ways which make you look perfectionist and hyper-professional, vs. weak.

I don't actually care that much about Dropbox security myself, but if anyone in the cloud computing space fucks up badly enough and frequently enough to make users distrustful of cloud resources, it makes life harder for everyone else in the space. That is a problem for me, and for other startup founders.

[gjm11]

> Even the US military does a better job when it accidentally kills people!

What does the US military do when it perpetrates a boneheaded security screwup that might or might not have compromised a small amount of data belonging to a few people?

(No question, Dropbox's response to this screwup is totally inadequate, but comparing it to how another organization responds when it has killed people is a bit silly.)

[ra]

if anyone in the cloud computing space fucks up badly enough and frequently enough to make users distrustful of cloud resources, it makes life harder for everyone else in the space

That's exactly how I felt after the AWS outage. There was a slew of (incorrect) media reports about why cloud computing was inherently bad for your business.

[tshtf]

If you’re concerned about any activity that has occurred in your account, you can contact us at [email protected].

I'd like the full access logs, including timestamps and IP addresses of every time my account was accessed in this timeframe. I've written [email protected] about this, and am waiting to hear back.

[rakkhi]

Asked the same. Quick response to be fair:

"I am unable to provide log data at this time.

We're working around the clock to gather additional data. We will notify affected users if we detect any unusual logins or activity in their account. We are reviewing our logs that record password authentication events in accounts.

We have not been able to detect any relevant account activity for your account during the time period in question, so we believe that your account was unaffected by the bug."

[phillco]

In fact, they should add this permanently to the UI, like Gmail.

[tptacek]

They should be going out of their way to retain these.

[allending]

I don't understand why they are not assuming everyone is going to be concerned about activity in their account instead of saying "if you are". I would personally expect every customer to be provided with the relevant information proactively. Should it really be up to the user to go through the geebees of files in their account?

[dave1619]

I've also emailed them and will be waiting to hear from them. This was a big breach of security.

[mef]

Somewhat disconcerting that I am a Dropbox customer, yet neither the breach nor the explanation has been communicated to me by email; I've had to find out about it by reading HN.

[veidr]

I completely agree. Sure, this way >99% of their users will never hear about this episode. But it makes dropbox look so much worse innthe eyes those of us who do.

[tomjen3]

I don't think that was a wrong decision actually -- most people who use dropbox are like my mom, not at all computer savy.

There is no way dropbox would be able to explain to them what happened without scaring them silly.

And if nobody logged in, there can't possibly be any danger to their account.

[dave1619]

An email should go out to every Dropbox customer with an option to log in and see all log info during the breached timeframe.

[iaskwhy]

That leads to an interesting question: should you be informed of situations like these even if you not affected by it?

[ctide]

I don't understand why Drew doesn't either make these posts or find a PR person to filter Arash's comments through. Arash never comes off well in these blog posts or in his comments here surrounding these issues.

His comments always come off feeling like 'sorry bros, it's really not that bad, and we fixed it so no problemo' unlike Drew's comments which generally have a much more personal feel to them.

[phillco]

Agreed. This would be like a major bank saying "for a 4-hour window yesterday, anyone could withdraw money from anyone's account. But don't worry, it's not a problem, because only a few people actually did."

[raganwald]

To each their own, but I am far more interested in what happened, what they are doing about it, whether it can happen again, and why I heard aboutnit on HN instead of directly from Dropbox.

I personally am not interested in an apology or a gaveling tone. These people are not my personal friends, I don't have an emotional investment in whether they pretend to care about my feelings. I have an objective intrest in how they choose to act and the information they give me so that I can make my own informed choices.

[orofino]

Just emailed them the following:

Dropbox Team,

For the duration of the event described in the post on your blog on Sunday I'd like the time of login and IP address of any authenticated sessions during the window. I'd also like to understand why a post from one of your customers linked to via hacker news was the only notification I received as one of your paying customers. If you know which user accounts we logged into during the event it seems rather straight forward that you would notify those impacted. It seems clear that had a 3rd party not brought this to light you'd have felt it unnecessary to notify your customers.

I look forward to your prompt response.

[epistasis]

I haven't kept anything that absolutely had to be secret on Dropbox, but I have kept stuff that I'd slightly prefer not to be public.

I'm going to change my behavior, the only things that go on Dropbox now will be completely public. They've completely lost my trust.

[bennysaurus]

Security leaks happen. The lesson here is treat anything that you put on a machine you don't completely control as being at-risk, even if you pay for top notch security and the vendor guarantees it. The guarantee (and any compensation) isn't a lot of comfort if the information hits the wild. This includes everything from web servers on Amazon to your Gmail account.

In the case of Dropbox, I would suggest PGP or Truecrypt anything sensitive and keep the keys locally or in another location that is completely unrelated to the box.

[Timothee]

Not that it would really change anything, but there isn't much wording around apologies or being sorry about letting that happened.

More details would have been nice too. Allowing anybody to log into anybody's account is a big deal, even if in the end a small percentage of people were likely affected. It's not like I couldn't access my account for a few hours or that the sync got messed up somehow.

Also, it'd be nice to know how the bug was discovered on Dropbox's side: did they realize it themselves or was it from nice people who found the problem?

[kanak]

I believe Chris Soghoian was the person who discovered the bug (there was an HN submission about this earlier).

http://twitter.com/#!/csoghoian

[tptacek]

It took almost 4 hours to discover that anyone could log in without a password? There has to be more to it than that, right?

[georgemcbay]

The blog post mentions that they ended all logged in sessions, but did they also rollback any new sync settings set from the start of the incident to when they did their cleanup? eg:

User Joe's account is logged into by attacker Tom. Tom sets his computer as one of Joe's "My Computers". Does what they did to clean up the problem invalidate this or does Joe have to log into his account, look at his list of "My Computers" and remove the ones he doesn't recognize manually to stop Tom's system from automatically syncing all his stuff until he does finally notice (which is likely never for most users, I'd assume)?

[jerrya]

IIRC from the last dropbox security fiasco, they also have to revoke the magic cookies that permit access without a password.

[jdludlow]

Good point.

Probably worth checking your own sharing and connected device settings.

== Connected Computers and Devices https://www.dropbox.com/account#manage

== Sharing https://www.dropbox.com/share

[CGamesPlay]

I'm not affiliated with DropBox or familiar with the way their systems work internally, but it seems pretty likely to me that the "magic files that let you log in without a password" are what DropBox calls "sessions", since this is the only form of access control you have after a password. So yes, they did roll these back.

[smoody]

Doesn't this completely blow their "only a few employees have access to your data" stance out of the water? Doesn't mean that, for example, every one of their programmers has access to user data? If a presumably simple bug can give the whole world access to your data, then it can't be that difficult for any dropbox employee to access it (again, from the inside).

[robryan]

Doesn't matter how much encryption you have on data if your decrypting it for the wrong people.

[mestudent]

If you weren't around for the last discussion and haven't figured it out by the web client: dropbox encrypts and decrypts your data on their servers which inherently provides anyone with access to them and their encryption decryption methods access to your files.

The problem as discussed many times is that of security and usability, dropbox takes the usability route and for some users this is great for others not so much.

[jerrya]

Compare and contrast how Dropbox has communicated this breach with the storm that rained down on LastPass when they forced password changes because of an activity they couldn't explain.

This might explain why Dropbox's poor communication of this security breach is the #winning move.

[g0atbutt]

There has been a lot of comments bashing Arash's response. His post certainly wasn't personal (and should have had a more apologetic tone), but beyond that what would you've liked to see him communicate? He seemed to do a good job explaining what happened and outlined a plan moving forward.

I'm not trying to defend him but I am genuinely curious what other people think his response was missing.

[random42]

1. The tone MUST have been apologetic.

2. The error is presented like casual news. Sort of downplaying the incident and not acknowledging that they screwed up. If the accounts were freely accessible for 4 odd hours, it is pretty serious.

3. No actual details of bug introduced were provided.

4. The communication was done on blog, and emails were not send. Again, it is an attempt to downplay the incident.

[dacort]

Client-side encryption feature request: https://www.dropbox.com/votebox/21/client-side-encryption

Ya know, just in case you want to upvote it.

[famousactress]

I'm not sure I totally understand. Is he saying that the users who were already logged in during the code update were the ones who had access to any account? Or is he wrangling words and letting us know that < 1% of Dropbox users had active web sessions at the time?

I understand the challenge that the Dropbox team finds themselves in here, but I'd very much like more details as to exactly what happened and what was possible for 4+ hours. I'm less interested in they're eventual report of the bad stuff they think did or didn't happen during that time.

[acangiano]

The lack of "sorry", "apologies", "rest assured" and similar keywords is astounding. Where is the heartfelt apology for the screw up?

[ary]

As it could so easily be seen as bashing/trolling I need to preface this question by saying that I'm asking in complete seriousness.

How does one accidentally set the failure mode of any piece of authentication code to escalate privileges instead of denying them? Even when I was first learning to code web apps I never found myself accidentally accepting any password.

It's shocking (to me, at least) because it seems like such a beginner mistake.

[dunham]

On possibility: A developer short circuits permissions checking for local testing and accidentally checks in the change. But automated testing should catch these kinds of mistakes.

[mason55]

Lots of ways, for example: Delegate authentication to another server, have your authentication server go down, mess up a try/catch clause in your app that doesn't handle a bad connection correctly

It's very doubtful something like Dropbox is as simple as

  if(user.hashedPassword == sha1(salt + enteredPassword)) {
    letUserIn();
  } else {
    error();
  }

Who knows though, it could have been something even as simple as

  if(user.hashedPassword == sha1(salt + enteredPassword)) ; {
    letUserIn();
  }

Now everyone gets in!

[random42]

Sample code:

Production

----------

@authenticate

def foobar():

    ....

Development

-----------

#@authenticate ( Because developers are lazy, while testing)

def foobar():

    ....

And accidentally pushing the code, with authentication decorator (python) commented.

[JohnsonB]

They made an apparently appalling security error; in the name of transparency they should be more specific about what code was the problem and how this bug worked so people have a better idea of really how bad they screwed up. Right now it could be something that could have blindsided anyone or something incredibly obvious that they should have seen or had some process to detect.

[mark_h]

Since a few people have commented on finding out via HN rather than Dropbox directly (as did I), I thought I'd post that I received an email from them just then.

In my case though it included "we noticed some potentially suspicious activity during the period", which for me was linking the desktop app. I don't know if everyone will be receiving a note, or just those who fit the potentially-suspicious profile. A quicker notification would have been nice, but I appreciate that they're also obviously looking for any aberrant behaviour.

[getsat]

Sounds like they need to add a spec where a valid user tries to login with an invalid password. That's quite embarrassing.