> We’ve identified your Wyze account as one that was affected. This means that thumbnails from your Events were visible in another Wyze user’s account and that a thumbnail was tapped. Most taps enlarged the thumbnail, but in some cases it could have caused an Event Video to be viewed.
Kudos to Wyze for doing the things noted in the thread like being honest and prompt with notification etc, but "thumbnails from your Events were visible... and that thumbnail was tapped" is a pretty mealy-mouthed way to say "another person saw your private pictures and videos taken with your Wyze cameras"
"like being honest and prompt with notification etc,"
On the Wyze subreddit, people have been griping about the fact that it took them days to even acknowledge something happened. When I read the Wyze email about it, there was no new info in it, it had pretty much all already been discussed online.
For those that want their Wyze cams 100% local (with RTSP), you can use wz_mini_hacks[0]. I've set up v2 and v3 cams using this, and they've never touched the internet or wyze app.
I've use the official RTSP firmware in the past on some v2 cams, but I remember it having some problems and not being as good as this solution.
Another in a long line of reasons to avoid low price, off-the-shelf, unauditable, cloud-enabled cameras.
I continue to be amazed that there is not a reasonably priced, open source, audited, local-first solution, which doesn’t require a significant personal investment of time to install and maintain.
I swear sometimes you people post this ignorance bait to get product suggestions.
Local-first cameras are super cheap and easy to find. Most just run their own local RTSP server, which you can connect to live with VLC, homeassistant, whatever. OpenIPC is like ddwrt for ip cameras, here is the supported devices page: https://github.com/OpenIPC/wiki/blob/master/en/guide-support...
But unless you're Richard Stallman, go with the closed source Reolink RTSP camera, which is about $100 and used by big corporate installs. It can integrate with the cloud, but you can set it up to just have each camera run a RTSP server with user/pass auth. You have to secure your own network.
But there isn't a really great open source platform for the kind of multi cam security that businesses might need. You have to do your own storage. But grab four reolinks, send the feeds to homeassistant, and most homeowners will be fine.
Personally I'm using Shinobi[1], but I've heard good things about frigate too if you're more into the ai detection and HA integration. There's a proxmox helper script one liner guided installation of you're into that [2]
I cringe every time I browse Amazon and see the random cam/cloud service/Play store apps that I know so many people are installing without questioning anything about them.
Not open-source/audited, but Apple HomeKit Secure Video is light years ahead. Encrypted in-house on your own hardware (ATV or HomePod), they don't have the keys, excellent UX, hosted service, super easy to use. I'm all about the self host, but assuming Apple isn't straight up lying, they have build something that's too good and too easy. I buy cheapish HKSV cameras, and block them from accessing the WAN, so there's minimal Apple tax outside one hub.
NAS companies like synology have an offering too. The main problem is that they still live in 2012 when it comes to cpu power and sell vastly underpowered boxes
This is one of the things Apple does right. HomeKit working local is a pretty great setup and just works. I put my HomeKit cameras on a VLAN without internet and device isolation and they still work seamlessly. The hard part is getting cameras that are wireless. I use scrypted but even then, getting ONVIF or RTSP isn’t as straight forward nowadays. I also have a local frigate backup which works great too. You can pipe in the detection to scrypted with MQTT.
The V3 models need to be downgraded to a specific firmware first and patching it exposes RSTP streams using https://github.com/AlexxIT/go2rtc. Everything doable without ever installing Wyze app on an environment air gapped environment with no internet.
I'm having great success with half a dozen v3's in tandem -- for $30 a camera, the quality is really unbeatable -- setup / notes below.
1. all cameras (firmware v4.36.9.139) have 64gb+ micro SD cards and record to local storage -- many people seem to have issues with anything greater than 32gb in v3's but I've found that this Verbatim tool [0] formats FAT32 at high capacity with no problems
2. all cameras have wz_mini_hacks [1] on the SD card with RTSP enabled
3. all cameras are connected via ethernet instead of wifi using this adapter [2] and wz_mini_hacks config
4. network blocks all outgoing internet connections for all cameras to keep them LAN-only -- this means I have to connect to VPN to review video when outside the house, but I'm cool with that
5. all RTSP streams are also recorded over the network via Agent DVR [3] to a NAS
6. the Wyze app (free tier, not paid) works normally with all of the above in place -- I find it much more intuitive to review recent videos in-app (streamed off the SD card), and then review the very occasional older video from a computer off the NAS (scrubbing through in VLC on a computer)
For what it's worth, I don't use them like a Ring camera where you're responding to realtime video events / talking through the camera to a delivery person -- this is mostly just for 24/7 recording. I have all object/motion detection events turned off, just a straight uninterrupted feed recording local and on the network.
I finally gave up trying to use the mini hacks to make RTSP work reliably. I ended up using Wyze Bridge [0] instead, and it has been far more stable. Using Frigate for the web UI. It doesn't make for a local-only solution, but I don't use my cameras to record anything that would bother me if other people saw it.
As time and motivation permit, I've been converting the cameras I care about over to POE. But having to run cable across the house for each one means I haven't done them all.
This is one of the reasons why you want end-to-end encryption wherever possible.
Even a bad implementation with cloud-synced encryption keys (which defeats most of the benefits of e2e) would have stopped this.
The response in this case (notifying customers and specifically stating whether they were affected or not) is excellent, but this seems to be a repeat of a previous incident from September 2023: https://www.theverge.com/2023/9/8/23865255/wyze-security-cam...
I wonder how many people would continue to so casually use these services if they understood that, for the most part, there is rarely proper end-to-end encryption of their data with these services. It is awfully disingenuous when these companies' marketing materials describe their services as "encrypted" when it usually just means there are two independent TLS pipes, which both terminate in their "cloud"; this surely gives a false sense of security to end-users who may not understand the implications of such a setup.
“Don’t use Wyze” seems like the wrong takeaway from this.
I’d go with “don’t put internet-connected cameras in your house if you don’t want those images on the internet”. I’ve got a Wyze in my garage looking over my mountain bikes, and for $35 I don’t really care if somebody else sees that image. But I’d never put one in my living space, regardless of their security track record.
What is the point of filming your mountain bikes? Do you watch them from your office with fondness of your most recent ride? Will that prevent them from being stolen? I doubt so.
This actually looks like a concurrency bug in their request handling code that may have stored the user id and camera id in shared variables, under load the wrong camera id is seen by a user. At least based on the description of what they say happened.
I would have liked them going more into the details of the caching issue. It sounds like they think the cache library was responsible for the issue, but a more technical analysis of what exactly went wrong within that library would be great.
My work encountered this same sort of thing after an outage. Our Redis instance or client got confused. If a=b and c=d in our cache, a request for a returned d randomly.
We quickly realized that cache is fast but not infallible. Use proper security on all your resources. Don’t rely on UUIDs to obfuscate your data as security.
> The incident was caused by a third-party caching client library that was recently integrated into our system. This client library received unprecedented load conditions caused by devices coming back online all at once. As a result of increased demand, it mixed up device ID and user ID mapping and connected some data to incorrect accounts.
That seems like enough of a line of bullshit to steer me away from ever using wyze.
Do you think the issue was something else? "People randomly see other people's content" is an issue that would immediately make me think some issue with caching is the culprit.
Given their openness in the rest of the communications, I don't see why they would make this part up.
Edit: Of course, I'm also curious what the actual bug was. A discussion below is suggesting several plausible ways (e.g. concurrency issues, insufficient entropy in some key) how a problem could happen under load (although many of these would also lead to the problem happening with less load, just much less often).
How so? I've seen caching clients exhibit some really weird behaviour under heavy load. It's not beyond the pale that, eg, the caching library doesn't do proper locking before writing, resulting in writes stomping all over each other.
Caching is normally read heavy, not write heavy, so it's plausible it wouldn't be something you'd see much under typical operation. After an outage, they'd be dealing with a thundering herd level of traffic as everything tries to reconnect, that'd be very different from normal write loads, even different than the write load they'd have seen when they first enabled caching.
Yeah I would very much likely to know what caching library has a failure mode of returning content for the wrong keys, that seems pretty bad if not a highly suspect explanation
I would at least want to know the client library so I can never use it for anything. Also never trust the client and I hope they don't mean the actual client app such that it can access other user id without server validation...
Coincidentally, I just cancelled my wyze service because the product and support are so terrible. I wanted a simple way to see if there was a package on my doorstep but instead I got something that alerts me when any dog, person, or vehicle goes down my street, and all I’ve gotten from support is robotic responses suggesting I update my firmware and ignoring my direct questions, running out the clock on my ability to return the thing. At this point I’m not surprised their engineering is bad and amused that it’s caused two different security incidents.
There's a bunch of things in here that don't really make sense:
> The incident was caused by a third-party caching client library that was recently integrated into our system. This client library received unprecedented load conditions caused by devices coming back online all at once. As a result of increased demand, it mixed up device ID and user ID mapping and connected some data to incorrect accounts.
What? How does load on the system affect correctness?
> The outage originated from our partner AWS
What does this mean? Was there an AWS outage for a service they use, or was this just a normal loss of an instance?
It's interesting that they blame external entities for the root causes of the incident and don't take responsibility for what is ultimately on them.
I assume the code was always incorrect, but only exhibits the problem in practice under high load. This could be a race-condition/data-race, or treating short hashes as unique.
It’s just a WAG, but I bet someone used a timestamp as a unique key, or at least part of one, so you were unlikely to get collisions except under load.
> What? How does load on the system affect correctness?
Seen this happen quite often with code that is not multi-thread safe, especially in languages like c# and java, such as using a static class property for data that should be request-scoped, or not using the appropriate concurrent collection classes etc.
It’s 2024. Everything is connected to the internet. Dropbox, Google, and Apple all offer multiple terabyte level plans. The default today is to store in the cloud. We are all storing data in someone else’s computer.
Instead of blaming the users, we must hold the companies responsible. Data privacy laws must be stricter and these incidents must be taken more seriously.
Or buying cheap, no-brand or upstart brand cameras with cloud capabilities.
I had a heck of a time finding a proper POE recording DVR camera system for my mom's house without online or cloud bullshit, but still I isolated it on the network to not take any chances of UPnP port opening or dial-home crap.
The only system I would trust would be one that laid out their security model, source to their apps, and had a self-hosted server DVR option. The captological signals of 99.9% of security system websites do not instill confidence in my mind.
Wyze cameras can actually be used very securely, as long as you bother to jump through some hoops.
First of all, google "Wyze RTSP firmware". It's the official firmware from the vendor that enables the RTSP protocol. Now you can enable RTSP via the app and give the camera a fixed IP address in your DHCP server.
RTSP is a pretty standard protocol, so you can now view the feed via VNC player, record it 24/7 via ffmpeg, use tools like motion, etc.
The camera will still try to connect to cloud, but you can move it to a local-only Wi-Fi network, or outright block it from reaching the outer world on the router side.
And if you want advanced stuff (multiple streams, organized recording, etc), there is a plethora of free/open-source security camera tools (iSpy for instance). It all takes time to learn and configure, but you can have your own fully closed-circuit surveillance network, while still using the Wyze's rather cheap hardware.
Instead of patching, you can also just use PoE cameras that are designed for this use case (local RTSP) and are only a little more expensive than Wyze. I’ve installed an Amcrest doorbell that works well with Scrypted and HomeKit, and plan on adding some Amcrest cameras like these soon: https://www.amazon.com/dp/B083G9KT4C
>The outage originated from our partner AWS and took down Wyze devices for several hours early Friday morning.
... As we worked to bring cameras back online, we experienced a security issue. Some users reported seeing the wrong thumbnails and Event Videos in their Events tab.
... The incident was caused by a third-party caching client library that was recently integrated into our system. This client library received unprecedented load conditions caused by devices coming back online all at once. As a result of increased demand, it mixed up device ID and user ID mapping and connected some data to incorrect accounts.
As an software engineer who's dealt with caches for large high throughput services, this does not make sense to me why they are blaming a caching client. It's your own code that will decide what is the cache key, and what value to pass as the cache key. Did the caching library have a bug where when you ask for a given key, it returned results for a different key? Or more likely did your own code have a bug where you mixed up the keys? I think we need more details on what went wrong in here.
Lot of blaming others - for architecture, stack, configuration, and operational choices that are likey/should be own decisions that should come with taking ownership.
This company non stop spams me, might just have to ditch the cam. I have grown used to throwing away hardware due to infinite fees, self bricking, or hacked out of the box. Consumer electronics have taken a painful dive in quality control. And hey Wyze unsubscribe me already !!!
Of course, the reason this keeps happening is the infrastructure is designed to let it happen in certain cases. Notice how they explicitly say, they need to fix it in the front end. They can't fix it in the backend because that would break eavesdropping.
This is the sort of thing that makes me salty that Unifi Protect is basically cloud locked in. No direct IP connection with "local" account support on the mobile app.
I'm wondering about the probability that, out of all the affected customers, at least one had the research skills and social skills to identify another customer and successfully ask to meet. Like, for an essay about "His schnauzer needed a mom. WyZettle: the amazing story of a pivot from a home camera service to a dating app."
A little off topic, but how is it possible that a tech startup named itself “Wyze” and didn’t get sued by Google over the “Waze” trademark? In some accents it sounds exactly the same, and they’re sort of in an adjacent product space.
Trademarks are about confusing names in a similar product market. Just having a similar-ish sounding name doesn't mean it violates the trademark. Self-driving cars are pretty different market to home security cameras.
3rd party libraries are a serious security problem that nobody wants to talk about, because there is no easy solution. I once lost $300k to a hack related to that issue.
Initially I thought this was about the money transfer company. Curious how there are two software companies with the same name and one hasn’t sued the other.
People want incredibly cheap products that are internet connected. Your average home user should not have to worry (and won't) about cybersecurity concerns, so this will continue to happen. The only out I can foresee is government regulation stepping in to make these incidents actually hurt for the companies, but America has basically no appetite for that.
jasongill|2 years ago
Kudos to Wyze for doing the things noted in the thread like being honest and prompt with notification etc, but "thumbnails from your Events were visible... and that thumbnail was tapped" is a pretty mealy-mouthed way to say "another person saw your private pictures and videos taken with your Wyze cameras"
tripdout|2 years ago
I actually appreciate the more specific details on how the private pictures and videos were actually viewed using terminology from the Wyze app.
thekevan|2 years ago
On the Wyze subreddit, people have been griping about the fact that it took them days to even acknowledge something happened. When I read the Wyze email about it, there was no new info in it, it had pretty much all already been discussed online.
KryptoKnight|2 years ago
frognumber|2 years ago
https://support.wyze.com/hc/en-us/articles/360026245231-Wyze...
A good response to this might be to put it back, and to extend other devices to be dual-use (Wyze Cloud or HA).
SparkyMcUnicorn|2 years ago
I've use the official RTSP firmware in the past on some v2 cams, but I remember it having some problems and not being as good as this solution.
https://github.com/gtxaspec/wz_mini_hacks/
voakbasda|2 years ago
I continue to be amazed that there is not a reasonably priced, open source, audited, local-first solution, which doesn’t require a significant personal investment of time to install and maintain.
Cheer2171|2 years ago
Local-first cameras are super cheap and easy to find. Most just run their own local RTSP server, which you can connect to live with VLC, homeassistant, whatever. OpenIPC is like ddwrt for ip cameras, here is the supported devices page: https://github.com/OpenIPC/wiki/blob/master/en/guide-support...
But unless you're Richard Stallman, go with the closed source Reolink RTSP camera, which is about $100 and used by big corporate installs. It can integrate with the cloud, but you can set it up to just have each camera run a RTSP server with user/pass auth. You have to secure your own network.
But there isn't a really great open source platform for the kind of multi cam security that businesses might need. You have to do your own storage. But grab four reolinks, send the feeds to homeassistant, and most homeowners will be fine.
hughesjj|2 years ago
[1] https://gitlab.com/Shinobi-Systems/Shinobi
[2] https://tteck.github.io/Proxmox/
UberFly|2 years ago
NelsonMinar|2 years ago
I don't have time to mess around with hacking a camera to do RTSP and then figure out how to set up and use something unfriendly like ZoneMinder.
scosman|2 years ago
ClumsyPilot|2 years ago
syntaxing|2 years ago
iAMkenough|2 years ago
radredgreen|2 years ago
[deleted]
firefalcon222|2 years ago
The V3 models need to be downgraded to a specific firmware first and patching it exposes RSTP streams using https://github.com/AlexxIT/go2rtc. Everything doable without ever installing Wyze app on an environment air gapped environment with no internet.
kevinsync|2 years ago
1. all cameras (firmware v4.36.9.139) have 64gb+ micro SD cards and record to local storage -- many people seem to have issues with anything greater than 32gb in v3's but I've found that this Verbatim tool [0] formats FAT32 at high capacity with no problems
2. all cameras have wz_mini_hacks [1] on the SD card with RTSP enabled
3. all cameras are connected via ethernet instead of wifi using this adapter [2] and wz_mini_hacks config
4. network blocks all outgoing internet connections for all cameras to keep them LAN-only -- this means I have to connect to VPN to review video when outside the house, but I'm cool with that
5. all RTSP streams are also recorded over the network via Agent DVR [3] to a NAS
6. the Wyze app (free tier, not paid) works normally with all of the above in place -- I find it much more intuitive to review recent videos in-app (streamed off the SD card), and then review the very occasional older video from a computer off the NAS (scrubbing through in VLC on a computer)
For what it's worth, I don't use them like a Ring camera where you're responding to realtime video events / talking through the camera to a delivery person -- this is mostly just for 24/7 recording. I have all object/motion detection events turned off, just a straight uninterrupted feed recording local and on the network.
Links:
0. https://www.verbatim.com/index/search.php?words=fat32+tool
1. https://github.com/gtxaspec/wz_mini_hacks
2. https://www.amazon.com/dp/B07M5X9795
3. https://www.ispyconnect.com/docs/agent/about
rootusrootus|2 years ago
As time and motivation permit, I've been converting the cameras I care about over to POE. But having to run cable across the house for each one means I haven't done them all.
[0] https://github.com/mrlt8/docker-wyze-bridge
tgsovlerkhgsel|2 years ago
Even a bad implementation with cloud-synced encryption keys (which defeats most of the benefits of e2e) would have stopped this.
The response in this case (notifying customers and specifically stating whether they were affected or not) is excellent, but this seems to be a repeat of a previous incident from September 2023: https://www.theverge.com/2023/9/8/23865255/wyze-security-cam...
avg_dev|2 years ago
ls65536|2 years ago
notatoad|2 years ago
I’d go with “don’t put internet-connected cameras in your house if you don’t want those images on the internet”. I’ve got a Wyze in my garage looking over my mountain bikes, and for $35 I don’t really care if somebody else sees that image. But I’d never put one in my living space, regardless of their security track record.
fullstop|2 years ago
prmoustache|2 years ago
spullara|2 years ago
bhhaskin|2 years ago
jabiko|2 years ago
illusive4080|2 years ago
We quickly realized that cache is fast but not infallible. Use proper security on all your resources. Don’t rely on UUIDs to obfuscate your data as security.
twisteriffic|2 years ago
That seems like enough of a line of bullshit to steer me away from ever using wyze.
tgsovlerkhgsel|2 years ago
Given their openness in the rest of the communications, I don't see why they would make this part up.
Edit: Of course, I'm also curious what the actual bug was. A discussion below is suggesting several plausible ways (e.g. concurrency issues, insufficient entropy in some key) how a problem could happen under load (although many of these would also lead to the problem happening with less load, just much less often).
Twirrim|2 years ago
Caching is normally read heavy, not write heavy, so it's plausible it wouldn't be something you'd see much under typical operation. After an outage, they'd be dealing with a thundering herd level of traffic as everything tries to reconnect, that'd be very different from normal write loads, even different than the write load they'd have seen when they first enabled caching.
hn_20591249|2 years ago
achille|2 years ago
https://news.ycombinator.com/item?id=35294082
t0mas88|2 years ago
Very little ownership on Wyze's side.
wutwutwat|2 years ago
-- Phil Karlton
SigmundA|2 years ago
hamburglar|2 years ago
pylua|2 years ago
unknown|2 years ago
[deleted]
belter|2 years ago
woodrowbarlow|2 years ago
sigden|2 years ago
[deleted]
psanford|2 years ago
> The incident was caused by a third-party caching client library that was recently integrated into our system. This client library received unprecedented load conditions caused by devices coming back online all at once. As a result of increased demand, it mixed up device ID and user ID mapping and connected some data to incorrect accounts.
What? How does load on the system affect correctness?
> The outage originated from our partner AWS
What does this mean? Was there an AWS outage for a service they use, or was this just a normal loss of an instance?
It's interesting that they blame external entities for the root causes of the incident and don't take responsibility for what is ultimately on them.
CodesInChaos|2 years ago
skipkey|2 years ago
0x0|2 years ago
Seen this happen quite often with code that is not multi-thread safe, especially in languages like c# and java, such as using a static class property for data that should be request-scoped, or not using the appropriate concurrent collection classes etc.
sneak|2 years ago
LeafItAlone|2 years ago
Instead of blaming the users, we must hold the companies responsible. Data privacy laws must be stricter and these incidents must be taken more seriously.
1letterunixname|2 years ago
I had a heck of a time finding a proper POE recording DVR camera system for my mom's house without online or cloud bullshit, but still I isolated it on the network to not take any chances of UPnP port opening or dial-home crap.
The only system I would trust would be one that laid out their security model, source to their apps, and had a self-hosted server DVR option. The captological signals of 99.9% of security system websites do not instill confidence in my mind.
matrix_overload|2 years ago
First of all, google "Wyze RTSP firmware". It's the official firmware from the vendor that enables the RTSP protocol. Now you can enable RTSP via the app and give the camera a fixed IP address in your DHCP server.
RTSP is a pretty standard protocol, so you can now view the feed via VNC player, record it 24/7 via ffmpeg, use tools like motion, etc.
The camera will still try to connect to cloud, but you can move it to a local-only Wi-Fi network, or outright block it from reaching the outer world on the router side.
And if you want advanced stuff (multiple streams, organized recording, etc), there is a plethora of free/open-source security camera tools (iSpy for instance). It all takes time to learn and configure, but you can have your own fully closed-circuit surveillance network, while still using the Wyze's rather cheap hardware.
sgerenser|2 years ago
Shadowmist|2 years ago
creativeSlumber|2 years ago
As an software engineer who's dealt with caches for large high throughput services, this does not make sense to me why they are blaming a caching client. It's your own code that will decide what is the cache key, and what value to pass as the cache key. Did the caching library have a bug where when you ask for a given key, it returned results for a different key? Or more likely did your own code have a bug where you mixed up the keys? I think we need more details on what went wrong in here.
albert_e|2 years ago
KryptoKnight|2 years ago
n89nanda|2 years ago
siliconc0w|2 years ago
gerwim|2 years ago
Yes, of course. Blame a third party library which was probably created by an open source maintainer instead of testing your own systems.
_obviously|2 years ago
summarity|2 years ago
unknown|2 years ago
[deleted]
darknavi|2 years ago
wredue|2 years ago
zzyzxd|2 years ago
but even with this, it's still an security camera app that can't send push notification without cloud access.
pledess|2 years ago
t8sr|2 years ago
sqlacid|2 years ago
vel0city|2 years ago
fatkam|2 years ago
unknown|2 years ago
[deleted]
_obviously|2 years ago
EVa5I7bHFq9mnYK|2 years ago
urbandw311er|2 years ago
ra|2 years ago
whatever1|2 years ago
In the era of cloud and microservices why each user does not have their own dedicated resources?
NegativeK|2 years ago
Because race to the bottom.
People want incredibly cheap products that are internet connected. Your average home user should not have to worry (and won't) about cybersecurity concerns, so this will continue to happen. The only out I can foresee is government regulation stepping in to make these incidents actually hurt for the companies, but America has basically no appetite for that.
seanarnold|2 years ago
X6S1x6Okd1st|2 years ago
uconnectlol|2 years ago
lofaszvanitt|2 years ago
scubadude|2 years ago
BugsJustFindMe|2 years ago
arealaccount|2 years ago
mahbran10|2 years ago
[deleted]
gvanmeter|2 years ago
[deleted]
gvanmeter|2 years ago
[deleted]
gvanmeter|2 years ago
[deleted]
micola4151|2 years ago
[deleted]
Amehle|2 years ago
[deleted]
gvanmeter|2 years ago
[deleted]