Shopify employees accessed customer databases without authorization

[synunlimited]

Got this email from Fangamer about Shopify earlier today. ---- Dear Fangamer customer,

Shopify, the company whose software runs the Fangamer store (and more than a million others online), has informed us that an internal security event it has been investigating since late last year included Fangamer customer data. Information regarding customer financial accounts and payment cards was not affected, but we are writing to make you aware of the situation.

According to Shopify, certain members of its support team used their Shopify credentials to obtain archived customer data from several hundred stores without authorization. The team members accessed data associated with order fulfillment — names, addresses, email addresses, cart contents, and phone numbers — but did not access or acquire any financial-account or payment-card information.

We are extremely frustrated and sorry to be sending you this email; Fangamer's internal development team takes data security extremely seriously. Data not in Fangamer's Shopify store — including Kickstarter backer information, account information and passwords, and email addresses used to sign up for our newsletter — was not accessed, and the store continues to operate as normal. Fangamer Japan, which operates as a separate store, was also not affected.

Shopify has terminated the employees who did this and eliminated the vulnerabilities that made it possible. Shopify has also reported that it will be providing any other relevant information to us as its investigation continues, and we'll pass along any new material details. If you have any questions, though, please contact us at [e-mail].

Thank you, Fangamer

[wyxuan]

The only icing in the cake is that at least Shopify has been both transparent and quick - it's only taken a couple months and they've managed to get bottom of the case. Couple months might seem long but from what I've seen it takes about a year of lag time from the start of the breach to when the company finds out/acknowledges.

In any case I'm wondering - how did Shopify discover this intrusion? Do they check logs regularly? Did they receive a tip off?

[twunde]

This is a common requirement for security-conscious organizations, especially those with HIPAA or PCI requirements. For shopify, this likely was originally created as a customer requirement, so that clients could monitor their staff. The typical setup is to generate internal user logs and feed it into a SIEM of some type, potentially with custom rules to do some checking. Alternatively, this may very well have been caught by a type of DLP (data loss protection) or network monitoring product

[dgudkov]

It seems like employees are becoming the weakest link in cloud security. If Google will be breached one day, most probably it will happen not because of a technical vulnerability, but due to employee sabotage.

I'm pretty sure that at exactly this moment somewhere someone criminal is already analyzing organization structures, employee profiles, internal security policies and tools of the cloud giants.

[Dracophoenix]

That already happened a decade ago: https://www.wired.com/2010/09/google-spy/

[sep_field]

Speaking as an ex-Facebook engineer it would be incredibly easy to get nefarious people employed there in an engineering role. Once inside, they have access to -all- user data; there is no actual access control (there are some basic access checks built into the Facebook application to keep you from accessing "private" data fields by accident, but all you have to do is edit that code and remove the access check and recompile Facebook on your laptop and you can access anything in the production database -- including peoples private Messenger chats).

Facebook warn you when you are hired not to actually do this, because they have auditing systems to watch for it and you will be fired (supposedly) but for people employed by some other agency specifically for the purpose of getting high-value private data out of Facebook, being fired by Facebook for doing so is part of the expected outcome and no big deal.

A well funded agency could easily keep getting people hired at Facebook to get whatever data they want, as often as they want. Facebook is constantly trying to keep their hiring pipelines full and despite the image Facebook likes to portray, it isn't "only the best talent" that gets a job there. There are some very smart, capable people at Facebook, but there are a ton of very mediocre engineers that lucked out in the hiring process, as well. It's really just a numbers game to get in.

I'm sure much the same is true at Google/Twitter/etc.

[manbackharry]

Didn't receive an email, but are they just now referring to the incident that took place September 23 2020?

https://www.cbc.ca/news/business/shopify-data-breach-1.57351...

[tantalor]

> not the result of some sort of technical vulnerability

So what then? Did they deceive or coerce somebody?

[imhoguy]

Two Tell HNs in two days about Shopify with some weak cases. I smell some stock shorting strategy here.

[motohagiography]

We have recourse against platform employees who snoop user data for personal reasons, and even share it with their friends or political organizations? Literally thought that was a perk of their jobs.

Someone should tell reddit/google/facebook/amazon as that will blow things up pretty badly.

Wait until they are subject to normal privacy regulations that require the companies to list the names of people who have accessed their user data.

[sitkack]

Something like this should probably legislated https://cloud.google.com/access-transparency

But it doesn't stop the government from bulk copying "business data" via powers granted by the Patriot Act.

[twunde]

From what I've heard from ex-FB employees, they are told repeatedly during onboarding that snooping for personal reasons is a first-offensive firable offense, to the point that engineers often won't try to look up information even when debugging prod issues.

[thebrain]

I got the same email from Fangamer, I'm surprised I haven't gotten similar emails from other Shopify stores I've used.

[spoonjim]

Shopify may have an audit trail of exactly which stores had data compromised, and perhaps even a trail of which specific customers.

[jasfi]

How do you protect against this sort of thing as a SaaS developer?

[jpalomaki]

Prevent direct database access or at least allow it only from jump servers which don’t allow file transfers.

For troubleshooting purposes create debugging tools. Log and check their usage. When things mature, you can even require multiple admins to work together for certain actions.

Minimize human access to production envs. Automate deployments. When access is needed, use jump servers and block file transfers (or force them to go through channel that is audited).

Do review logs and alerts on regular basis. Put effort to minimize false alerts and excessive logging. Quite when reviewing logs you just notice things that “don’t look right”.

Nothing is 100% secure, but also people with bad intensions don’t always have unlimited skills/energy/time.

[Intermernet]

A step in the right direction is to use encrypted backend data / databases. This is still fraught with problems, but it provides another layer of protection and can demonstrate the difference between "The data was just sitting there" and "We had to manually exfiltrate the encryption keys to read the data".

It's not perfect, but it adds another layer to prove malicious intent.

[mytailorisrich]

Employees should have access to customers' data on a need to know basis. Most employees do not need access so should not have access.

Then, there should be an audit trail of all accesses and this should be known to employees. First that dissuades employees from acting improperly, second that allows the company to verify that they indeed do not act improperly and to track down culprits if something happens.

[notadev]

I sometimes go out of my way to hide my identity from sites/services I sign up with. Easiest way to get doxxed is for someone to ask one of their polticially-aligned buddies working at a site to pull up your info.

[thinkingkong]

I mean... its only news because it got out. If you seriously believe companies arent accessing your data its borderline delusional.

[natmaka]

Any system admin accessing to a server is able to take a copy of any data stored (or even transiently present on its network interface) on it.

If he is a spy/robber, if he is corruptible or threatened... a third party will obtain this copy. For the main culprit this doesn't induce any risk (where is the evidence?). This is absolutely not as with your bank, for example, which cannot really steal money without you taking notice.

How serious people are willing to store confidential data on any rented or hosted server is completely beyond me. Then some of their competitors' proposals are "just a little bit" better than theirs', or seem to have a pretty good grasp on some R&D or customer database.

Many here work on some cloud thing, most are honest and some will be upset by my comment. This is not about you but about rotten fruits in the basket.

[tantalor]

Um what? The problem was the lack of internal authorization to do so. Do you not see how that's a huge liability? It's basically an "inside job". If one employee can do it, then anybody with similar credentials can.

The "oh shit" scenario is when the stolen data is used against to commit crimes against customers, e.g., identity theft, stalking, you name it.

[krthkv]

The "employee access to customer data isn't protected" sits as unsolved an opportunity canvas/brief in almost every SaaS company. You can get to a fair amount of controls with little to no code and only with process changes (aka SoC and ISO certifications), which is also what SaaS security teams spend quite a bit of time on. There are a fair amount of problems to be solved here.

[bg24]

Agree. And as much as policies are in place, it is not unusual to see csv exports downloaded to local laptops for analysis as part of work.

[xtiansimon]

How was the event discovered?