Keep in mind that a substantial portion of users now use ad blockers such that a lot of URLs used for analytics like this are blocked.
Consequently, you can't actually expect to capture 100% of these analytics events nor even expect the percentage captured to stay the same over time since the filter lists are very regularly updated and users enable/disable different ad blockers over time.
More broadly speaking, once you have sent a webpage to the user, you should not expect anything from the user's browser. They may or may not allow whatever arbitrary JS you have on the page. They may even intentionally give you bad data (e.g. hijack the payload to give you intentionally malformed data).
edit: even more broadly speaking, there's additional reasons why you can't expect to receive these kinds of callbacks: consider what happens if a user loses connectivity between loading the page and them navigating away (e.g. their phone loses service because they went into an elevator before navigating away)
> Keep in mind that a substantial portion of users now use ad blockers such that a lot of URLs used for analytics like this are blocked.
How sure are we about this? I'm pretty sure it depends on which market specifically you're in, and the data I'm about to show is of course not perfect, but it seems that not so many users actually do use adblockers today. Although I don't know a single developer who doesn't, and in some web applications I'm running, the majority of users do use adblockers as they are focused on developers.
Chrome is assumed to be the most popular browser (by a large margin last time I checked, so I won't bother to check again) and a quick search puts the user base around 2-3 billion users. Searching for "adblock" in the Chrome Web Store (https://chrome.google.com/webstore/search/adblock?hl=en&_cat...) shows that the most popular adblocker has a user base of ~300,000 users.
That makes 0.015% to 0.01% of Chrome users having the "AdBlock" extension installed. Not that substantial.
If someone has some more accurate numbers than my slightly-educated guess, I'd be happy to be proven wrong.
Edit: The above user base of the adblock extension is wrong. As Jabbles pointed out, I was seeing the number of reviews, not number of users.
So instead, the page lists "10,000,000+ users" so we can assume the true number to be above that, but below "100,000,000+ users" users.
That would put the amount of Chrome users using the "AdBlock" extension between 0.3% and 5% more of less. Closer to "substantial", but not sure if it would impact businesses choice regarding ads/tracking or not.
For those wondering --- despite the domain of the site seeming to imply so, this does not use CSS.
On the other hand, the more I read about stuff like this, the more disenchanted I am with the "modern web". When something as seemingly innocent as a link can "stealthily" do something else when it's clicked, but otherwise looks exactly the same as a benign one (and moreover, behaves entirely as expected with JS off), it really brings into question whose interests such features are designed to serve.
From an honest analytics parsing question, how do you decipher 4 different iOS users on the same device model from a Starbucks or the VPN exit node sharing the same IP from access_log?
Just thinking here. If you did not have SPA, couldn't you just I don't know log when they request those pages? Like, aren't those done with HTTP request anyway... So why the hell you need to do the logging on client side...
Then again that would be fun to send enough nice bogus log information... As clearly that is what they want.
What's wrong with doing something like this? If it contains PII, then I agree, the additional hassle of dealing with everything that comes with handling PII (like GDPR) becomes too much, easier to just don't do it. But if it doesn't contain PII, it can be useful to see how many people drop off a form VS submitting it for example.
Will any of these techniques inform the site owners that I close the tab the instant they put up a big “sign up for our mailing list” modal? And maybe that would help train them to not design sites that way?
Maybe that could be one good outcome of such an API.
I think popups on page load are more of a symptom of short-term, myopic thinking than anything else. This type of thinking is strongly encouraged by always trying to live up to quarterly projections - "Did Sales meet their targets?"
Popups exchange long-term trust in your brand for short-term profits. Of course it's the case that if you pop a modal in front of everyone, your conversion stats will go up. But potential life-time customers who have faith in your brand and believe in your mission will lose trust in you quickly and probably never sign up in the first place.
For me, it always comes down to the primary rule of UX: if you wouldn't do it IRL, don't do it online.
You can ask for someone's email if they look like they're interested in something you're selling, but don't ask for it the moment they enter your store!
Since you were never going to join their mailing list, no, you are simply saving them bandwidth.
What you don't realise is that x number of people join the list and x number of people tolerate being spammed with 'did you know we sell stuff' emails every day for the rest of their lives. x number of those people actually do buy something; likley the thing they intended to buy originally but joined the email list in case a discount code would be the subject of the first 'welcome' email because this is what they have been trained to expect.
All this has been A/B tested and, thus, has been 'proven' to be great business practice.
Those pop-ups are usually shown on "exit intent" which is simply when you move your mouse outside the viewport (i.e. the address bar or hover over another tab).
Firefox is unlucky that its remaining users furiously hate everything tracking-related, and are ready to burn Mozilla to the ground for even tiniest suspicion of any transgression. Therefore, Mozilla can't choose a lesser evil here, because it's still seen as evil.
What use-cases are there for this feature other than tracking users?
Anyway, the best solution is to pass any links through a redirect which is responsible for logging the visit. That's how Google tracks what search results get clicked on. And it doesn't require any JavaScript.
I'm surprised they didn't even mention it as a possibility.
> What use-cases are there for this feature other than tracking users?
Social services where users want to signal if they are online/offline
Users who want to know how much time they've spent on a platform.
Closing down sessions for things that can be expensive to just run "forever". Think VMs and similar that gets started when the user visits the page, that you want to turn off when they leave.
The use cases are many, and there are also many other alternative solutions, this is obviously not the only one. But as always, all of them come with their own tradeoffs, so this technique might be the best for some of them.
Let us not forget that tracking users is not a single use case.
Closing off cached resources when a user vanishes is a good thing, it saves resources for other users and could, if I may stretch a little, have environmental impact. Noting that sessions tend to end after a particular set of circumstances could mean spoting something in your application that is annoying people, something that particularly in an SPA you might otherwise not easily detect.
Sometimes tracking is legitimately benign. It is only when unnecessary PII is included, and kept to track each specific user over time rather than just over a session, that it gets stalky to the point of begin in my "I won't do that" list (sharing the PII around further gets you on my "you are morally wrong for doing that" list).
> surprised they didn't even mention it as a possibility
The article is focused on detecting when a user leaves. While a fair amount if that will be detectable because they've clicked a link, it doesn't cover the large case set that is closing tabs and windows.
Something I might have to investigate (or search to see if someone else has): do any of these techniques work reliably on controlled OS shutdown.
"Tracking users" has lots of uses. For example, we were trying to do OP on gwern.net for a while, using several of those tricks. It failed miserably and we could never figure out why.
What evil things were we doing? Well, I wanted to know if people were using the links I put inside popups, like to Google Scholar. Was anyone using them? If not, they were just so much clutter and should be removed for the users' benefit. I also wanted to get a list of outbound links by popularity, so I could write summaries/annotations for the most popular unannotated links, and rank dead links by priority for fixing (there are too many unannotated or dead links to 'just fix', so a ranking would've helped a lot).
But the link tracking never worked, so, I couldn't do any of that.
This is an interesting take, I would assume at some point it would end up costing you time, because for all its faults, the modern web has some great tools (apps) that can improve your life in a number of ways. I’m also unsure how they’d earn your trust without giving them that essential unblocking js trust in the first place.
> Access to client side scripts on my machine has to be earned in trust
I agree that it's all about trust. Why would you ever enter information on a website you don't trust? What extra information can be gathered using JS that can't be otherwise? Or what is the exact harm done if they send extra requests to a tracking API?
My point is that once you access a specific website you consent with that website sending HTML/CSS/JS to your computer and executing it. The biggest problem those days is when that website is sending your information to other entities, 3rd parties, for other purposes than improving your experience. I think having "tracking" to detect errors, loading time issues and improving the user experience is perfectly reasonable if implemented in a proper way.
What happens if the user has more than one tab/window open an closes one?
Weirdest case I've observed in the wild is a site that works perfectly without referer header, except for logout. As usual for anything requiring referer it fails without a user-readable message without referer. If referer is sent, the logout procedure switches through multiple redirects and some waiting while js is doing something, server backend probably, too.
Some sites make weird assumptions about the network, the browser, and the user.
IE11 is the last browser that can reliably send a request before the tab closes (by using synchrous).
I swear telling the business guys that we had no way to run an action when someone closed the page had them nearly running back to IE. And for that specific functionality, I find it hard to disagree.
In our case, we lock entities when someone starts working on them, but now we have no way to unlock them if they fat-finger the close button, or press one too many backspaces.
I'm surprised that browsers nowadays include specific API for better user tracking.
At first thought I don't like it.
But at second thought I think that I like it. Issue is, webmasters will still track you, like it or not. But with those dedicated APIs tracking will be less annoying and those who can control their browsers, can disable that kind of tracking more easily (provided that they're even aware of those features).
Anyway I don't like the direction web standards are evolving. Why have so many custom APIs and attributes? There's already fetch priority flag, that should be enough. This is exponential explosion of APIs. Modern API designers don't know about concept of orthogonal APIs (or I don't know something).
Also <a ping attribute seems to track user even if he disabled JavaScript. Bad bad thing.
We have attempted to use sendBeacon as a last resort signal for releasing an exclusive lock granted to a user for editing a record. Only one user can be granted this at a time for a given record to avoid contention, but there are so many ways a user might abandon an edit-in-progress in web applications (close browser, inactivity, loss of network or power).
I realize there are a lot of different strategies for addressing this situation, and I'd be curious to hear how others have solved this problem.
In the specific case of using sendBeacon, it is a little disheartening that once a basic programming tool gets abused by the analytics/advertising/tracking behemoths then the ad blockers' only recourse at times is to disable it completely. I completely support the efforts made by the ad blockers, but also find it sad how legitimate uses of certain tools end up getting blocked in the process.
> However, this is extremely unreliable. In many situations, especially on mobile, the browser will not fire the unload, beforeunload, or pagehide events.
Man, I had no idea that this stuff could happen. Having a built-in, non-disableable pingback method in the browser engine is scary stuff for privacy-oriented people. I don't know why more people don't know about this. Happily, I found and installed https://github.com/vijithassar/pingkiller to help mitigate the `ping` attribute (in case my ad blocker isn't doing that already).
Interesting. I am curious what browsers were tested in this experiment, and if the different flavors behave differently.
From my casual observation of Firefox (I use the network debugging tools in Firefox frequently) there isn't a column for priority. Firefox also does not show a status of pending, rather it just omits a pending status until a status becomes available.
Navigator.sendBeacon() doesn't reliably follow 30x redirects in my experience. About 30% of the time, for no obvious reason, it stops following redirect chains after hitting the first redirect in the chain, even though it's supposed to follow those for analytics purposes.
before any JS has a chance to run. You can also trap "beforeunload" listeners, or just delete them document.body.onbeforeunload=null document.onbeforeunload=null
Wow there are a lot of new features in web browsers lately just targeting analytics and ads. Makes me wonder what other use cases have beacons as the solution.
[+] [-] kevindong|3 years ago|reply
Consequently, you can't actually expect to capture 100% of these analytics events nor even expect the percentage captured to stay the same over time since the filter lists are very regularly updated and users enable/disable different ad blockers over time.
More broadly speaking, once you have sent a webpage to the user, you should not expect anything from the user's browser. They may or may not allow whatever arbitrary JS you have on the page. They may even intentionally give you bad data (e.g. hijack the payload to give you intentionally malformed data).
edit: even more broadly speaking, there's additional reasons why you can't expect to receive these kinds of callbacks: consider what happens if a user loses connectivity between loading the page and them navigating away (e.g. their phone loses service because they went into an elevator before navigating away)
[+] [-] capableweb|3 years ago|reply
How sure are we about this? I'm pretty sure it depends on which market specifically you're in, and the data I'm about to show is of course not perfect, but it seems that not so many users actually do use adblockers today. Although I don't know a single developer who doesn't, and in some web applications I'm running, the majority of users do use adblockers as they are focused on developers.
Chrome is assumed to be the most popular browser (by a large margin last time I checked, so I won't bother to check again) and a quick search puts the user base around 2-3 billion users. Searching for "adblock" in the Chrome Web Store (https://chrome.google.com/webstore/search/adblock?hl=en&_cat...) shows that the most popular adblocker has a user base of ~300,000 users.
That makes 0.015% to 0.01% of Chrome users having the "AdBlock" extension installed. Not that substantial.
If someone has some more accurate numbers than my slightly-educated guess, I'd be happy to be proven wrong.
Edit: The above user base of the adblock extension is wrong. As Jabbles pointed out, I was seeing the number of reviews, not number of users.
So instead, the page lists "10,000,000+ users" so we can assume the true number to be above that, but below "100,000,000+ users" users.
That would put the amount of Chrome users using the "AdBlock" extension between 0.3% and 5% more of less. Closer to "substantial", but not sure if it would impact businesses choice regarding ads/tracking or not.
[+] [-] dfas23|3 years ago|reply
[+] [-] ForHackernews|3 years ago|reply
Interesting. Are there tools we can install to send malicious payloads to surveillance companies?
[+] [-] userbinator|3 years ago|reply
On the other hand, the more I read about stuff like this, the more disenchanted I am with the "modern web". When something as seemingly innocent as a link can "stealthily" do something else when it's clicked, but otherwise looks exactly the same as a benign one (and moreover, behaves entirely as expected with JS off), it really brings into question whose interests such features are designed to serve.
[+] [-] jbverschoor|3 years ago|reply
No need for that. Stop hoarding data!
[+] [-] soylentgraham|3 years ago|reply
[+] [-] dylan604|3 years ago|reply
[+] [-] Ekaros|3 years ago|reply
Then again that would be fun to send enough nice bogus log information... As clearly that is what they want.
[+] [-] capableweb|3 years ago|reply
[+] [-] unknown|3 years ago|reply
[deleted]
[+] [-] unknown|3 years ago|reply
[deleted]
[+] [-] ninkendo|3 years ago|reply
Maybe that could be one good outcome of such an API.
[+] [-] panphora|3 years ago|reply
Popups exchange long-term trust in your brand for short-term profits. Of course it's the case that if you pop a modal in front of everyone, your conversion stats will go up. But potential life-time customers who have faith in your brand and believe in your mission will lose trust in you quickly and probably never sign up in the first place.
For me, it always comes down to the primary rule of UX: if you wouldn't do it IRL, don't do it online.
You can ask for someone's email if they look like they're interested in something you're selling, but don't ask for it the moment they enter your store!
[+] [-] dazc|3 years ago|reply
What you don't realise is that x number of people join the list and x number of people tolerate being spammed with 'did you know we sell stuff' emails every day for the rest of their lives. x number of those people actually do buy something; likley the thing they intended to buy originally but joined the email list in case a discount code would be the subject of the first 'welcome' email because this is what they have been trained to expect.
All this has been A/B tested and, thus, has been 'proven' to be great business practice.
[+] [-] slig|3 years ago|reply
[+] [-] XCSme|3 years ago|reply
[+] [-] bhaney|3 years ago|reply
Little sad that Firefox not supporting an API only downgrades that API's browser support to "good" instead of "unusable" these days
[+] [-] danuker|3 years ago|reply
Another example is trimming referrer URLs (just keeping the domain).
[+] [-] vbezhenar|3 years ago|reply
There's Chrome and there's Safari. Rest of browsers are rounding errors. And even those rounding errors are mostly chromiums. Sad indeed.
[+] [-] pornel|3 years ago|reply
[+] [-] TheAceOfHearts|3 years ago|reply
Anyway, the best solution is to pass any links through a redirect which is responsible for logging the visit. That's how Google tracks what search results get clicked on. And it doesn't require any JavaScript.
I'm surprised they didn't even mention it as a possibility.
[+] [-] capableweb|3 years ago|reply
Social services where users want to signal if they are online/offline
Users who want to know how much time they've spent on a platform.
Closing down sessions for things that can be expensive to just run "forever". Think VMs and similar that gets started when the user visits the page, that you want to turn off when they leave.
The use cases are many, and there are also many other alternative solutions, this is obviously not the only one. But as always, all of them come with their own tradeoffs, so this technique might be the best for some of them.
[+] [-] dspillett|3 years ago|reply
Closing off cached resources when a user vanishes is a good thing, it saves resources for other users and could, if I may stretch a little, have environmental impact. Noting that sessions tend to end after a particular set of circumstances could mean spoting something in your application that is annoying people, something that particularly in an SPA you might otherwise not easily detect.
Sometimes tracking is legitimately benign. It is only when unnecessary PII is included, and kept to track each specific user over time rather than just over a session, that it gets stalky to the point of begin in my "I won't do that" list (sharing the PII around further gets you on my "you are morally wrong for doing that" list).
> surprised they didn't even mention it as a possibility
The article is focused on detecting when a user leaves. While a fair amount if that will be detectable because they've clicked a link, it doesn't cover the large case set that is closing tabs and windows.
Something I might have to investigate (or search to see if someone else has): do any of these techniques work reliably on controlled OS shutdown.
[+] [-] gwern|3 years ago|reply
What evil things were we doing? Well, I wanted to know if people were using the links I put inside popups, like to Google Scholar. Was anyone using them? If not, they were just so much clutter and should be removed for the users' benefit. I also wanted to get a list of outbound links by popularity, so I could write summaries/annotations for the most popular unannotated links, and rank dead links by priority for fixing (there are too many unannotated or dead links to 'just fix', so a ranking would've helped a lot).
But the link tracking never worked, so, I couldn't do any of that.
[+] [-] daveoc64|3 years ago|reply
[+] [-] OtomotO|3 years ago|reply
Access to client side scripts on my machine has to be earned in trust.
Most websites never get that far, because they are broken without js.
Doesn't matter, saves me a lot of time ;)
[+] [-] PUSH_AX|3 years ago|reply
[+] [-] XCSme|3 years ago|reply
I agree that it's all about trust. Why would you ever enter information on a website you don't trust? What extra information can be gathered using JS that can't be otherwise? Or what is the exact harm done if they send extra requests to a tracking API?
My point is that once you access a specific website you consent with that website sending HTML/CSS/JS to your computer and executing it. The biggest problem those days is when that website is sending your information to other entities, 3rd parties, for other purposes than improving your experience. I think having "tracking" to detect errors, loading time issues and improving the user experience is perfectly reasonable if implemented in a proper way.
[+] [-] vbezhenar|3 years ago|reply
Actually I think that's the whole reason of introducing this attribute.
[+] [-] someweirdperson|3 years ago|reply
Weirdest case I've observed in the wild is a site that works perfectly without referer header, except for logout. As usual for anything requiring referer it fails without a user-readable message without referer. If referer is sent, the logout procedure switches through multiple redirects and some waiting while js is doing something, server backend probably, too.
Some sites make weird assumptions about the network, the browser, and the user.
[+] [-] Aeolun|3 years ago|reply
IE11 is the last browser that can reliably send a request before the tab closes (by using synchrous).
I swear telling the business guys that we had no way to run an action when someone closed the page had them nearly running back to IE. And for that specific functionality, I find it hard to disagree.
In our case, we lock entities when someone starts working on them, but now we have no way to unlock them if they fat-finger the close button, or press one too many backspaces.
[+] [-] vbezhenar|3 years ago|reply
At first thought I don't like it.
But at second thought I think that I like it. Issue is, webmasters will still track you, like it or not. But with those dedicated APIs tracking will be less annoying and those who can control their browsers, can disable that kind of tracking more easily (provided that they're even aware of those features).
Anyway I don't like the direction web standards are evolving. Why have so many custom APIs and attributes? There's already fetch priority flag, that should be enough. This is exponential explosion of APIs. Modern API designers don't know about concept of orthogonal APIs (or I don't know something).
Also <a ping attribute seems to track user even if he disabled JavaScript. Bad bad thing.
[+] [-] glitcher|3 years ago|reply
I realize there are a lot of different strategies for addressing this situation, and I'd be curious to hear how others have solved this problem.
In the specific case of using sendBeacon, it is a little disheartening that once a basic programming tool gets abused by the analytics/advertising/tracking behemoths then the ad blockers' only recourse at times is to disable it completely. I completely support the efforts made by the ad blockers, but also find it sad how legitimate uses of certain tools end up getting blocked in the process.
[+] [-] janci|3 years ago|reply
[+] [-] openplatypus|3 years ago|reply
Check here: https://developer.mozilla.org/en-US/docs/Web/API/Navigator/s...
> However, this is extremely unreliable. In many situations, especially on mobile, the browser will not fire the unload, beforeunload, or pagehide events.
[+] [-] XCSme|3 years ago|reply
[+] [-] LorenDB|3 years ago|reply
[+] [-] zelon88|3 years ago|reply
From my casual observation of Firefox (I use the network debugging tools in Firefox frequently) there isn't a column for priority. Firefox also does not show a status of pending, rather it just omits a pending status until a status becomes available.
[+] [-] taf2|3 years ago|reply
[+] [-] arbuge|3 years ago|reply
[+] [-] Phelinofist|3 years ago|reply
[+] [-] rasz|3 years ago|reply
[+] [-] z3t4|3 years ago|reply
[+] [-] jtchang|3 years ago|reply
[+] [-] nokya|3 years ago|reply