PostHog cofounder here. This affected users that did not have a specific version of the JS library pinned and deployed a new version, or were using the snippet, and had network capture enabled, (a feature we introduced very recently and is only enabled on 3% of projects), and had recordings enabled on that particular session (for most customers, only a small percentage of sessions are recorded due to sampling or billing limits)
This outage was definitely disruptive and we shouldn't have let this happen. We will be doing a full post mortem write up, but this affected a small percentage of our users, so the comparison with Crowdstrike isn't fair.
- Used your recommended way of implementing PostHog [1]
- Used a feature of the product
- Used a feature of the product
The comparison to CrowdStrike is not fair, you're right. But this attempt to shed responsibility still leaves a sour taste.
[1] See "This is the simplest way to get PostHog up and running. It only takes a few minutes." from your website, which is the first method suggested when clicking the "installation" tab
>This affected users that did not have a specific version of the JS library pinned and deployed a new version
Par for the course honestly. The amount of garbage that gets called "production" these days is mindboggling. No blue/green or canary deployments, shipping code that has nothing pinned, no clear rollback, etc. This is what happens when anyone can become an EngineerTM after a two week JavaScript boot camp.
You're trying to phrase this as if those conditions make it any less bad, but they don't. This affected users that were using the latest version and used... features? Give me a break. Every product has bugs, but trying to downplay the issue after you've just read a distressed user of yours struggle with it is definitely not what you should be doing.
timgl|1 year ago
This outage was definitely disruptive and we shouldn't have let this happen. We will be doing a full post mortem write up, but this affected a small percentage of our users, so the comparison with Crowdstrike isn't fair.
ziddoap|1 year ago
- Used your recommended way of implementing PostHog [1]
- Used a feature of the product
- Used a feature of the product
The comparison to CrowdStrike is not fair, you're right. But this attempt to shed responsibility still leaves a sour taste.
[1] See "This is the simplest way to get PostHog up and running. It only takes a few minutes." from your website, which is the first method suggested when clicking the "installation" tab
VBprogrammer|1 year ago
I personally would have like to see a bit more contrition rather than trying to minimise the issue.
anonfordays|1 year ago
Par for the course honestly. The amount of garbage that gets called "production" these days is mindboggling. No blue/green or canary deployments, shipping code that has nothing pinned, no clear rollback, etc. This is what happens when anyone can become an EngineerTM after a two week JavaScript boot camp.
Etheryte|1 year ago