As always, these disclosures are an opportunity for all of us to learn something without having to pay the substantial reputational price for doing so. I'd encourage everyone on HN to read the How We Ended Up Here parts carefully and take simple remediation steps in your own projects/companies.
The unauthorized access was confirmed to have taken place ... via a debug version of our website that was visible to the public.
This is a ridiculously common way for attackers to get into your network. You both want to have an accurate inventory of the apps/boxes your organization has (highly non-trivial at scale) and you don't want dev/staging/etc to be publicly accessible. The simplest way is to firewall off traffic to them.
If you need them to be accessible for e.g. callbacks from public services, default-deny everything and control access at the network or frontend web server level before things get to your applications (which are likely going to be your vulnerable underbelly).
In most cases, the dev servers will not themselves have really interesting things on them. They will instead be used by attackers to pivot to other, more interesting servers on your network. Patreon's understanding is that the dev servers lacked SSH keys to the production servers -- that's a good thing, but after the bad guy has a dev server with local network access to interesting machines, they're going to do things like e.g. probe for exposed admin consoles to network appliances, outdated RedMine instances, internal tools, etc etc, and they will eventually own every machine that talks to any machine that your dev server can talk to. You will eventually lose the entire deployment if the bad guy wants you to.
The development server included a snapshot of our production database, which included encrypted data.
Never, ever do this. (It's been a firing offense at most shops I've worked at in my career, and that's for good reasons.)
>>The development server included a snapshot of our production database, which included encrypted data.
>Never, ever do this. (It's been a firing offense at most shops I've worked at in my career, and that's for good reasons.)
Would you use a migration script that "scrambled" certain columns on the way back from Production to Development? That's the only way I've seen it done properly at $EMPLOYER.
I've worked for organizations where you could be disciplined for not stripping out certain data from a database before using it for testing - primarily SSN and email addresses, but it's absolutely necessary at times where using real data will allow you to identify the cause of an issue whereas troubleshooting blindly on test data could take 10 times as long.
I agree. When i read this email this morning, I came away feeling more secure with the company, rather than less. I hope that more companies do it this well, and realize that admitting the breach well can strengthen the trust.
> unauthorized access to registered names, email addresses, posts, and some shipping addresses. Additionally, some billing addresses
Considering patreon has some users who have been, and still are, targets of harassment, this could be much worse than having their credit card or password leaked.
> Although accessed, all passwords, social security numbers and tax form information remain safely encrypted with a 2048-bit RSA key.
> ...
> We protect our users’ passwords with a hashing scheme called ‘bcrypt’ and randomly salt each individual password. Bcrypt is non-reversible, so passwords cannot be “decrypted.” We do not store plaintext passwords anywhere.
[+] [-] patio11|10 years ago|reply
The unauthorized access was confirmed to have taken place ... via a debug version of our website that was visible to the public.
This is a ridiculously common way for attackers to get into your network. You both want to have an accurate inventory of the apps/boxes your organization has (highly non-trivial at scale) and you don't want dev/staging/etc to be publicly accessible. The simplest way is to firewall off traffic to them.
If you need them to be accessible for e.g. callbacks from public services, default-deny everything and control access at the network or frontend web server level before things get to your applications (which are likely going to be your vulnerable underbelly).
In most cases, the dev servers will not themselves have really interesting things on them. They will instead be used by attackers to pivot to other, more interesting servers on your network. Patreon's understanding is that the dev servers lacked SSH keys to the production servers -- that's a good thing, but after the bad guy has a dev server with local network access to interesting machines, they're going to do things like e.g. probe for exposed admin consoles to network appliances, outdated RedMine instances, internal tools, etc etc, and they will eventually own every machine that talks to any machine that your dev server can talk to. You will eventually lose the entire deployment if the bad guy wants you to.
The development server included a snapshot of our production database, which included encrypted data.
Never, ever do this. (It's been a firing offense at most shops I've worked at in my career, and that's for good reasons.)
[+] [-] voltagex_|10 years ago|reply
>Never, ever do this. (It's been a firing offense at most shops I've worked at in my career, and that's for good reasons.)
Would you use a migration script that "scrambled" certain columns on the way back from Production to Development? That's the only way I've seen it done properly at $EMPLOYER.
[+] [-] tallanvor|10 years ago|reply
[+] [-] techdragon|10 years ago|reply
Congratulations Patreon, you just executed a textbook "good" public disclosure of a security breach!
[+] [-] wanderfowl|10 years ago|reply
[+] [-] werid|10 years ago|reply
Considering patreon has some users who have been, and still are, targets of harassment, this could be much worse than having their credit card or password leaked.
[+] [-] babaghanoush|10 years ago|reply
[+] [-] rdl|10 years ago|reply
[+] [-] 21echoes|10 years ago|reply
> ...
> We protect our users’ passwords with a hashing scheme called ‘bcrypt’ and randomly salt each individual password. Bcrypt is non-reversible, so passwords cannot be “decrypted.” We do not store plaintext passwords anywhere.
Updated post now says the above.
[+] [-] unknown|10 years ago|reply
[deleted]
[+] [-] Animats|10 years ago|reply
[+] [-] eugenekolo2|10 years ago|reply
[+] [-] unknown|10 years ago|reply
[deleted]