It’s bad that Backblaze did not do their due diligence while integrating with Facebook pixel, but the bigger problem is the tendency of third party integrations having default settings that are overly aggressive when it comes to access of user/site data.
This should be a warning to every developer, when you integrate with any third party - don’t just copy the default snippet of code they recommend, read the docs thoroughly and then test/monitor what is being sent back to the third party. And if you are writing a third party widget, be considerate and make the defaults the least aggressive possible when it comes to accessing user/site/server data.
At this point, every developer should (and needs) to be aware of how invasive any tracking code they blindly copy/paste into a website. I think it's required from people who operate normal websites, blogs, etc... but missing it is somewhat forgivable, especially from people just starting.
But the level of irresponsibility with customers' most private data from a company who's MAIN JOB is to protect it is absolutely shocking. Yes, it's the freaking pixel that does the tracking, BUT IT'S THEIR RESPONSIBILITY TO KNOW WHAT THE HECK IS HAPPENING ON THEIR WEBSITE. Don't they have any sort of vulnerability assessment or security code review? Their reply tweet is almost as infuriating as how something like that could even happen to begin with. Like yeah, sorry, ain't our fault! It's astounding.
I considered using backblaze a couple of times and now I'm very happy that I eventually didn't.
As much as I loathe and despise Facebook, this one is on Backblaze: they integrated with a well-known evil (seriously this surprises absolutely nobody) and should have been more careful with their settings. Or, you know, not have done it at all.
> but the bigger problem is the tendency of third party integrations having default settings
It's certainly a problem, but the biggest problem is simply that Backblaze doesn't need to integrate Facebook pixel to their web interface, especially when users are logged in.
There is absolutely zero benefit to this for their users, but I also fail to see what it could bring to them as a company??
They have now created a major PR problem for themselves, and what did they get in exchange?
(Ok, as the saying goes, there's no such thing as bad publicity. But still.)
>And if you are writing a third party widget, be considerate and make the defaults the least aggressive possible when it comes to accessing user/site/server data.
Unless your business model is literally collecting all personal and private information, then don't do it.
>>It’s bad that Backblaze did not do their due diligence while integrating with Facebook pixel
How much "due diligence" should be required to know that adding anything from facebook INSIDE your backup / file storage product is a bad idea... hell adding anything from facebook at any product should be considered a bad idea...
> and then test/monitor what is being sent back to the third party.
Trust, but verify. Especially FB.
I see a lot of comments about a paid service using tracking pixels. I understand the backlash but this is also how companies that are taking your hard earned money improve and iterate on their products. I know people don’t like tracking but usage analytics are invaluable to improving the product. There’s only so much customer engagement you can do, and often that leaves really unexpected outcomes in the dark (if you asked most customers, they’d ask for a faster horse).
The issue here is that Backblaze injected untrusted code into their core product which by its very nature is extremely sensitive. I’m not really sure what they were thinking. Rolling your own tracking is a pain, but some security contexts demand it.
"but the bigger problem is the tendency of third party integrations having default settings that are overly aggressive when it comes to access of user/site data.
"
Our permission schemes are way too broad in general. I have done some tests with Zapier and IFTTT. For every integration you consent with them seeing ALL your data, being able to modify and so on. You also don't get a log how often the permission was used.
We're looking to invert this equation at Transcend. All client side network emissions can be regulated in accordance with tracking consent following a block/quarantine-by-default model.
This means that any network emissions you add to your site would have to be categorized by URL or domain instead of relying on shaky special-cased integrations that can fail as soon as an API changes.
PSA for any devs out there implementing FB Pixels:
– Facebook's pixel will, by default, attach click listeners to the page and send back associated metadata. This simplifies implementation needs, but can create unintentional information leaks in privileged contexts. To disable this behavior, there's a flag[1] you can use. After which, you can manually trigger FB Pixel hits and control both when they're fired and what information is included in them.
– There's a feature for the FB Pixel called Advanced Matching[2] that allows you to send hashed PII as parameters with your FB events. "Automatic Advanced Matching" can be enabled at any time via a toggle in the FB interface. I believe that setting autoConfig to false as mentioned above will similarly prevent Automatic Advanced Matching from working (since it disables the auto-creation of all those listeners to begin with). When manually triggering pixel calls like above, you can use this functionality via "Manual Advanced Matching"[3].
As a general rule, I'd strongly encourage anyone implementing a Facebook pixel to also include the autoConfig = false flag. This makes it work like most other pixels, where the base tag just instantiates an object. After which, hits only occur when explicitly defined in the site code, and include specifically what details you include in it. That way you're fully aware of the scope of data disclosure happening and any need from marketing to include sensitive (or potentially sensitive) information in these calls has to be explicitly requested (and theoretically vetted) as part of the standard dev process.
The thing that concerns me about the FB Pixel (and GTM) is that the host is completely free to do anything and everything to the page. Even if they don't do anything "evil" today, tomorrow is a different story completely. This scares the pants off of me and makes me want to rip out any "tracking" that I've ever installed on any site anywhere. Actually, that's probably not a bad idea.
Are there no browser level protections for this type of thing? I thought CORS was supposed to prevent these activities from happening.
As a protection for the users, addons like Facebook Container for Firefox [0] can isolate all Facebook tracking and prevent the scripts from running on pages that are not facebook.com.
And if that doesn't tick your creepy boxes lets try the financials. If a user hits your tracking pixel they (and those like them) will more likely see ads similar to yours, meaning potential customers will be more expensive to obtain now.
One easy way to avoid this kind of mistake in your own product: make a clear distinction between your publicly-facing web site ("corpweb") and your web app for logged-in users. Preferably, they should be served from separate infrastructure.
Corpweb should be as static as possible, except for whatever third-party JS the marketing professionals think is necessary. It's their job, they know what's best.
Your app should have zero third-party JS except for technical analytics (New Relic, Datadog, whatever).
(This distinction can be fuzzier for free services, and for consumer stuff with non-sensitive data; Backblaze is neither of those.)
Pay careful attention to the response from Backblaze:
> "Hi Brett! The pixels we use are primarily for audience building when we advertise on other platforms like Facebook for example." [...]
The carefully calculated cutesy "Hi Brett!" with the exclamation point is the same reason big tech companies use infantile graphics [0]: by seeming playful, they create the illusion they are a Safe Friend you can Trust.
(For the curious: I couldn't be bothered to install https://projectnaptha.com/ (probably would've worked), I just resized a terminal to the same column width and blindly retyped the text into a shell printf statement.)
And also when you upload something to blackblaze, you are already handing out your files to a third party. There shouldn't be anything sensitive that you didn't encrypt client side.
There are programs that automatically encrypt files before storing them. A particularly enticing one is Cryptomator, which acts as a ‘disk’ keeping files on various providers and thus syncs between devices—however its Android app isn't open-source so I never actually tried it.
It's still revealing two levels of filename extesions - type of encryption and file format. Full filename should be simply random alphanumeric ASCII string without any filename extensions, although this requires to manage and store a map of keys and filenames.
They could maybe salvage goodwill with genuine corporate soul-searching that ends up asserting/reasserting values -- and leads them to focus on providing trustworthy service to their users, and conspicuously away from some "tech" industry norms of selling out one's users.
As a provider of a paid service, it seems like they're in a better position to take the high road than a lot of tech companies are, but they have to decide that's who they are, and be clear they mean it.
Yev from Backblaze here -> we’ve looked into and verified the issue and have pushed out a fix. We will continue to investigate and will provide updates as we have them.
* Announces this breach to the relevant data protection authorities. They have 48 hours from learning about it to giving an initial report to the UK's ICO.
* Makes a blog post apologizing, explaining how it happened, and what they've done to prevent it happening again.
I doubt it’s intentional, but companies should seriously consider if the benefits of integrating things like Facebook Analytics tools outweighs the negatives. It seems like considering their audience they would not use Facebook of all things.
I get why they might be doing it ( audience building ), but they should have restricted it only to their HomePage or some info pages. ( check azernik's response on how to do this correctly ! )
It's easy to attack Backblaze.
Before attacking them for this, please make sure the company that you are working for or building doesn't do the same thing. ( I know for a fact that a lot of startups make heavy use of Audience building).
> Before attacking them for this, please make sure the company that you are working for or building doesn't do the same thing.
I don't get why these two are related at all. One can do both the second and the first. By their own admission in other HN threads, Backblaze earns several million dollars a year and is proud not to have VC backing. So it doesn't seem like anybody is attacking an underdog who's struggling to change the status quo and needs to be held to lower standards.
Have been a Backblaze customer for many years, mostly because of their state of HD here on HN. Lost all confidence in Backlaze.
Alternatives? Had been using rsyncnet in startups for many years, but was more expensive (now using Backblaze for storing GBs of raw images from DSLRs)
Wow they pretty much flushed all user trust and good will down the toilet with this. I had considered using them in the past but I sure won't be considering them now. Even if this were an accident how are there not procedures in place that would have required approval and vetting?
Does anyone have any idea how long this has been going on?
And: it only happens when rendering the filenames to a browser window right? So only when I browse folder y in bucket x are my filenames for that folder shared with FB?
Backblaze, I really enjoyed being a paying customer. Until now. Bunch of dorks.
Edit: goes to show you have to encrypt EVERYTHING at rest, even file names...
Great. I finally shook off my lethargy and did some research and thought they were perfect feature and price-wise. Setup a backup script using rclone and did a backup yesterday - worked great too - now this.
Had a look at rsync.net - too pricey for my puny sub-500GB data.
This should be trivially and universally fixable everywhere:
Never include the 3rd party marketing scripts on any pages where a user is authenticated.
But of course that would deprive many companies of mountains of valuable data so it ain't ever happening, right?
---
Also, am I missing something obvious here? If Backblaze -- or anyone else really -- wants analytics, what do they need the Facebook pixel for? There are so many good analytics services out there.
[+] [-] benja123|5 years ago|reply
This should be a warning to every developer, when you integrate with any third party - don’t just copy the default snippet of code they recommend, read the docs thoroughly and then test/monitor what is being sent back to the third party. And if you are writing a third party widget, be considerate and make the defaults the least aggressive possible when it comes to accessing user/site/server data.
[+] [-] beshrkayali|5 years ago|reply
But the level of irresponsibility with customers' most private data from a company who's MAIN JOB is to protect it is absolutely shocking. Yes, it's the freaking pixel that does the tracking, BUT IT'S THEIR RESPONSIBILITY TO KNOW WHAT THE HECK IS HAPPENING ON THEIR WEBSITE. Don't they have any sort of vulnerability assessment or security code review? Their reply tweet is almost as infuriating as how something like that could even happen to begin with. Like yeah, sorry, ain't our fault! It's astounding.
I considered using backblaze a couple of times and now I'm very happy that I eventually didn't.
[+] [-] magicalhippo|5 years ago|reply
Here's the thing though, being a third party, due diligence means you constantly have to check what they're doing.
Because otherwise how do you know when they suddenly change their script to do something entirely different?
This is why having any 3rd party scripts on a dashboard service like this is in my view entirely inexcusable.
When I go to visit my cloud-based dashboard for Acronis' backup service[1] there's one single domain involved, and that's how it should be.
[1]: https://www.acronis.com/en-us/products/true-image/
[+] [-] gruturo|5 years ago|reply
[+] [-] bambax|5 years ago|reply
It's certainly a problem, but the biggest problem is simply that Backblaze doesn't need to integrate Facebook pixel to their web interface, especially when users are logged in.
There is absolutely zero benefit to this for their users, but I also fail to see what it could bring to them as a company??
They have now created a major PR problem for themselves, and what did they get in exchange?
(Ok, as the saying goes, there's no such thing as bad publicity. But still.)
[+] [-] tomaskafka|5 years ago|reply
If you want to retarget me on FB, fine, do it on a marketing pages of a website, but don't let the trackers anywhere close to my private data.
[+] [-] agilob|5 years ago|reply
Unless your business model is literally collecting all personal and private information, then don't do it.
[+] [-] syshum|5 years ago|reply
How much "due diligence" should be required to know that adding anything from facebook INSIDE your backup / file storage product is a bad idea... hell adding anything from facebook at any product should be considered a bad idea...
[+] [-] qeternity|5 years ago|reply
Trust, but verify. Especially FB.
I see a lot of comments about a paid service using tracking pixels. I understand the backlash but this is also how companies that are taking your hard earned money improve and iterate on their products. I know people don’t like tracking but usage analytics are invaluable to improving the product. There’s only so much customer engagement you can do, and often that leaves really unexpected outcomes in the dark (if you asked most customers, they’d ask for a faster horse).
The issue here is that Backblaze injected untrusted code into their core product which by its very nature is extremely sensitive. I’m not really sure what they were thinking. Rolling your own tracking is a pain, but some security contexts demand it.
[+] [-] atYevP|5 years ago|reply
[+] [-] spaetzleesser|5 years ago|reply
Our permission schemes are way too broad in general. I have done some tests with Zapier and IFTTT. For every integration you consent with them seeing ALL your data, being able to modify and so on. You also don't get a log how often the permission was used.
[+] [-] Sephr|5 years ago|reply
This means that any network emissions you add to your site would have to be categorized by URL or domain instead of relying on shaky special-cased integrations that can fail as soon as an API changes.
[+] [-] xorcist|5 years ago|reply
[+] [-] cosmie|5 years ago|reply
– Facebook's pixel will, by default, attach click listeners to the page and send back associated metadata. This simplifies implementation needs, but can create unintentional information leaks in privileged contexts. To disable this behavior, there's a flag[1] you can use. After which, you can manually trigger FB Pixel hits and control both when they're fired and what information is included in them.
– There's a feature for the FB Pixel called Advanced Matching[2] that allows you to send hashed PII as parameters with your FB events. "Automatic Advanced Matching" can be enabled at any time via a toggle in the FB interface. I believe that setting autoConfig to false as mentioned above will similarly prevent Automatic Advanced Matching from working (since it disables the auto-creation of all those listeners to begin with). When manually triggering pixel calls like above, you can use this functionality via "Manual Advanced Matching"[3].
As a general rule, I'd strongly encourage anyone implementing a Facebook pixel to also include the autoConfig = false flag. This makes it work like most other pixels, where the base tag just instantiates an object. After which, hits only occur when explicitly defined in the site code, and include specifically what details you include in it. That way you're fully aware of the scope of data disclosure happening and any need from marketing to include sensitive (or potentially sensitive) information in these calls has to be explicitly requested (and theoretically vetted) as part of the standard dev process.
[1] https://developers.facebook.com/docs/facebook-pixel/advanced...
[2] https://www.facebook.com/business/help/611774685654668
[3] https://developers.facebook.com/docs/facebook-pixel/advanced...
[+] [-] miked85|5 years ago|reply
[+] [-] Corrado|5 years ago|reply
Are there no browser level protections for this type of thing? I thought CORS was supposed to prevent these activities from happening.
[+] [-] Lukas_Skywalker|5 years ago|reply
[0] https://addons.mozilla.org/en-US/firefox/addon/facebook-cont...
[+] [-] corobo|5 years ago|reply
Don't give data to Facebook lmao.
[+] [-] hertzrat|5 years ago|reply
[+] [-] azernik|5 years ago|reply
Corpweb should be as static as possible, except for whatever third-party JS the marketing professionals think is necessary. It's their job, they know what's best.
Your app should have zero third-party JS except for technical analytics (New Relic, Datadog, whatever).
(This distinction can be fuzzier for free services, and for consumer stuff with non-sensitive data; Backblaze is neither of those.)
[+] [-] Zhenya|5 years ago|reply
I'm guessing if you're logged into facebook, now FB can correlate:
1) you use a backup service
2) all the metadata from the file names
Whoops.
This is why my network runs pi-hole / diversion with all tracking blocked network wide.
[+] [-] berniemadoff69|5 years ago|reply
> "Hi Brett! The pixels we use are primarily for audience building when we advertise on other platforms like Facebook for example." [...]
The carefully calculated cutesy "Hi Brett!" with the exclamation point is the same reason big tech companies use infantile graphics [0]: by seeming playful, they create the illusion they are a Safe Friend you can Trust.
[0] http://jollo.org/LNT/public/nursery.html
[+] [-] exikyut|5 years ago|reply
It's just GPG-encrypted backups.
Look. Here's what the innerText says, with the `\n`s interpreted:
Once again, noise. Perfect.(For the curious: I couldn't be bothered to install https://projectnaptha.com/ (probably would've worked), I just resized a terminal to the same column width and blindly retyped the text into a shell printf statement.)
[+] [-] cm2187|5 years ago|reply
[+] [-] aasasd|5 years ago|reply
[+] [-] durnygbur|5 years ago|reply
[+] [-] guywhocodes|5 years ago|reply
[+] [-] celsoazevedo|5 years ago|reply
https://twitter.com/backblaze/status/1373751015594356739
Not sure if they intended to send file names and sizes to FB, but in any case this doesn't look good. I'm currently looking for alternatives.
[+] [-] neilv|5 years ago|reply
As a provider of a paid service, it seems like they're in a better position to take the high road than a lot of tech companies are, but they have to decide that's who they are, and be clear they mean it.
[+] [-] kardos|5 years ago|reply
[+] [-] atYevP|5 years ago|reply
[+] [-] CA0DA|5 years ago|reply
[+] [-] londons_explore|5 years ago|reply
* Announces this breach to the relevant data protection authorities. They have 48 hours from learning about it to giving an initial report to the UK's ICO.
* Makes a blog post apologizing, explaining how it happened, and what they've done to prevent it happening again.
[+] [-] alphabettsy|5 years ago|reply
[+] [-] ing33k|5 years ago|reply
It's easy to attack Backblaze.
Before attacking them for this, please make sure the company that you are working for or building doesn't do the same thing. ( I know for a fact that a lot of startups make heavy use of Audience building).
Note : I have used their service in the past and moved on to OVH due to a different issue in the past . https://www.backblaze.com/blog/b2-503-500-server-error/
[+] [-] newscracker|5 years ago|reply
> Before attacking them for this, please make sure the company that you are working for or building doesn't do the same thing.
I don't get why these two are related at all. One can do both the second and the first. By their own admission in other HN threads, Backblaze earns several million dollars a year and is proud not to have VC backing. So it doesn't seem like anybody is attacking an underdog who's struggling to change the status quo and needs to be held to lower standards.
[+] [-] KingOfCoders|5 years ago|reply
[+] [-] KingOfCoders|5 years ago|reply
Have been a Backblaze customer for many years, mostly because of their state of HD here on HN. Lost all confidence in Backlaze.
Alternatives? Had been using rsyncnet in startups for many years, but was more expensive (now using Backblaze for storing GBs of raw images from DSLRs)
[+] [-] AdmiralAsshat|5 years ago|reply
Even if you're paying for the product, you're probably still the product.
[+] [-] risyachka|5 years ago|reply
In this particular one you are not.
[+] [-] bogomipz|5 years ago|reply
[+] [-] isoprophlex|5 years ago|reply
And: it only happens when rendering the filenames to a browser window right? So only when I browse folder y in bucket x are my filenames for that folder shared with FB?
Backblaze, I really enjoyed being a paying customer. Until now. Bunch of dorks.
Edit: goes to show you have to encrypt EVERYTHING at rest, even file names...
[+] [-] cameronperot|5 years ago|reply
[1] https://github.com/restic/restic
[2] https://help.backblaze.com/hc/en-us/articles/115002880514-Ho...
[+] [-] noisy_boy|5 years ago|reply
Had a look at rsync.net - too pricey for my puny sub-500GB data.
[+] [-] pdimitar|5 years ago|reply
Never include the 3rd party marketing scripts on any pages where a user is authenticated.
But of course that would deprive many companies of mountains of valuable data so it ain't ever happening, right?
---
Also, am I missing something obvious here? If Backblaze -- or anyone else really -- wants analytics, what do they need the Facebook pixel for? There are so many good analytics services out there.
[+] [-] dleslie|5 years ago|reply
I'm paying for Backblaze, now I'm wondering if that was a mistake.