top | item 40677424

Microsoft to delay release of Recall AI feature on security concerns

308 points| mfiguiere | 1 year ago |reuters.com | reply

468 comments

order
[+] digging|1 year ago|reply
This is confusing and vague to me, which I believe is exactly the intent. It focuses on security, reiterates that security is their top priority (and we know that this is untrue). What were the security problems? They don't even allude to the existence or detection of any specific security problems.

It sounds to me like they're figuring out a new marketing approach, or they're softening the blow by "listening to users" and then rolling out more slowly, when outrage has died down and people will just accept it.

[+] segasaturn|1 year ago|reply
My takeaway is that Microsoft has been trying to boil the frog, but slipped and turned the temperature up too quickly. They're retreating for now, but make no mistake that Recall will slowly trickle back into Windows under another name. Every major power broker wants something like Recall to become the norm - bosses to spy on their employees, governments to spy on their citizens/enemies, and tech CEO's to collect training data for AI and target more ads at end users.
[+] swatcoder|1 year ago|reply
There's a much more mundane read:

They invested a bunch of effort into a product the market loudly rejected.

They're now withdrawing the product while they figure out what they can salvage from the effort.

Key stakeholders may have a few ideas about how to proceed (ranging from "try again later" through "repurpose it" to "forget it"), but enterprises of Microsoft size make decisions very slowly so of course it's vague about what's next. Collectively, they almost certainly don't know!

[+] rvense|1 year ago|reply
Security is a mindset and some people don't have it.

I used to work for a company that made a rather popular database for mobile applications. An easy API to store data on your phone and have it synced to a server with no effort on the developers part.

Two of my co-workers spent a few weeks making a nice looking chat application which worked by syncing messages from many users to different devices, and they wanted to publish it as a demo. Until somebody else pointed out that there was no security at all. The server just accepts the latest state from the client. This was fine for most of the current use cases, but for chat basically meant that any client could rewrite the entire history and the server would just say "thanks!" on next sync and distribute the changes to everyone else. These were adult humans with degrees from respectable institutions, and this hadn't crossed their minds at all.

Basically, I think a combination of Hanlon's razor and nobody wanting to be a naysayer is a perfectly adequate explanation for this Recall thing. I think it's obvious that a lot of people would like their computer to work like that, and I can see them wanting to get it out without having listened to any internal criticism (if they even have a culture that allows that).

[+] pjmlp|1 year ago|reply
Currently I am still looking forward to when the Secure Future Initiative (SFI) will actually mean more .NET and Rust and less COM and C++ love by Windows team.

So until this changes, take with a grain of salt how much secure Recall is actually going to be.

Contrast this with Apple Inteligence, where not only are most local APIs made available via Swift, they have created special hardware and a unikernel like OS with sandboxed layers, exposing only what OS capabilities required for AI processing and cluster communication.

Versus "Thrust us, we are going to do the right thing".

[+] 1vuio0pswjnm7|1 year ago|reply
"It sounds to me like they're figuring out a new marketing approach, or they're softening the blow by "listening to users" and then rolling out more slowly, when outrage has dies down ad people will just accept it."

Of course "listening to users" really means "listening in on users". Or just "bad press".

Microsoft does not consult with users before adding code into Windows. Nor do users contact Microsoft to tell the company what code they want or don't want.

Even if they did, the company does not operate based on user suggestions.

The reaction to "Recall" by journalists, bloggers and commenters is not that they think it should be "delayed". They think it is a bad idea.

Microsoft will do as it pleases. As it always has done.

[+] patmorgan23|1 year ago|reply
Per one of the ars Technica articles, All the information collected was stored locally completely unencrypted, and would be accessible by anyone with local administrator rights.
[+] jmholla|1 year ago|reply
> It focuses on security, reiterates that security is their top priority (and we know that this is untrue).

I think that messaging is a direct response to their hearing in from of the House yesterday. They were being grilled on their numerous security lapses and Brad Smith (president of Microsoft) constantly reiterated that they are refocusing their priorities to be security. They were also questioned about Recall specifically so it's not surprising to see this as one of the first places where they are putting out that messaging.

[+] Jedd|1 year ago|reply
The specific security problem was that their enterprise customers said no, and not in a 'no thanks' way, but a more vehement 'no fucking way', way.

They could conceivably push to SOHO users, but a) there's no revenue there (and this stuff is expensive), and b) it's really bad optics.

"We're going to offer you a feature that your workplace refused to run on their network."

I'm sure there's ways to spin that, but it'd be a challenge.

[+] pcloadletter_|1 year ago|reply
Or maybe they have to figure out how to actually make it work
[+] mihaaly|1 year ago|reply
My recollection is that the CEO stated no security problem with the product, security was their utmost and first the toppest priority all the time and into eternity, they wouldn't dare trying to release anything with security concerns.

Apparently there are security concerns afterall. Did they lie before or now or just completely clueless about what is a security concern or what? I am confused.

[+] HumblyTossed|1 year ago|reply
They're totally waiting for the negative press to die down, then they'll try again.
[+] xnx|1 year ago|reply
People should not get over this (but probably will). There was an uproar (decades ago) about GMail "reading all your email". This was overblown, but Microsoft building the infrastructure to view a history of everything on your screen is much much worse. There's a lot more private things that get displayed on a screen (and of course all of your email would be a subset) that no one has a right to see.
[+] ranger_danger|1 year ago|reply
> What were the security problems?

I would argue there really weren't away, apart from the usual disaster/lack of security that desktop systems have.

It wasn't uploaded anywhere, so the only threat would be from programs that would run locally and steal it, which is already the same for any other (even third-party) program stealing your local files, which they have always been able to do.

[+] AceJohnny2|1 year ago|reply
You're assuming Microsoft acts as a singular, cohesive entity, which like any company it is not.
[+] godelski|1 year ago|reply
> What were the security problems?

> They don't even allude to the existence or detection of any specific security problems

Arguably the product itself. Which is another reason they might be vague about it. Because to talk about those security problems would taint the entire product and they can't do that if they aren't willing to completely scrap it.

People have been talking about how the data in here is similar to what may be already existing but that's far from the truth. Yes, these companies have a lot of data on us, but this is a significant step forwards in the granularity of that data. It's also worth noting that hackers could not get into your computer and assume that your computer not only has a keylogger that they can access to further compromise your system (and other systems/accounts) but that they can also obtain screenshots. These increase user risk significantly and greatly reduce the requisite technical skill needed for those infiltrating machines.

Similarly, many have pointed out the potential connections to Chat Control[0] and how such systems can likely be used by many companies to be exploitative of workers. While you may trust your company/partner/significant others/government and so on, it is important to remember that not everyone has such luxuries. It is also important to remember that such things can change. Even in the US there are high risks of potential abuse: such as police obtaining a warrant to get this data to see if someone is trying to obtain abortion medication. Regardless on where you fall on that specific issue, you can replace it with any other concerning issue and I'm sure you wouldn't like that (guns, religion, gender identity, political affiliations, and so on). So even if you trust Microsoft to not give away this type of information nor to provide authorities access (which often includes authorities not in your home country), then you must ask if the benefits are worth the costs. And not just for you, but for others.[1]

> It sounds to me like they're figuring out a new marketing approach

I suspect this is correct and as segasaturn suggested, turned up the heat too fast. I also suspect that this type of data invasion can be much more easily understood by the general public, who often struggle with understanding what metadata is and how it is/can be used. It does require technical knowledge for this and is often non-obvious, even for people who are well above average in technical literacy (as is the average HN user).

[0] Specifically we should note here that Chat Control would force Microsoft to use this system in a much more invasive way. We lambasted Apple over their proposal for CSAM detection, including the potential risks of abuse even if it were theoretically impossible to avoid hash collisions. Having Relay would require Microsoft to implement such a system and that's why there are many conspiracies arising that Relay is specifically intended for Chat Control, because true or not it would likely have similar outcomes. We'll see if Apple revisits the idea, and the recent WWDC doesn't rule out such a possibility https://www.patrick-breyer.de/en/posts/chat-control/

[1] https://www.youtube.com/watch?v=goQ4ii-zBMw

[+] neogodless|1 year ago|reply
I'm a bit confused by the headline chosen for the submission (but the update doesn't do much to clarify).

The original is this:

Update on the Recall preview feature for Copilot+ PCs

> Recall will now shift from a preview experience broadly available for Copilot+ PCs on June 18, 2024, to a preview available first in the Windows Insider Program (WIP) in the coming weeks.

To be clear, it may be delayed for public release, but it is still shipping to Insiders (possibly on June 18, 2024 but in the coming weeks indicates later).

> With that in mind we are announcing updates that will go into effect before Recall (preview) ships to customers on June 18.

Further...

> ...we plan to make Recall (preview) available for all Copilot+ PCs coming soon.

[+] Hasu|1 year ago|reply
The headline is correct. I have seen people believe that "indefinite" means "permanent", but it just means "undetermined". It is delayed, but we (and perhaps Microsoft) do not know for how long, so the delay is indefinite.
[+] kmlx|1 year ago|reply
> Copilot+ PCs

not being accustomed with microsoft products i initially read this as copilot “plus” pcs.

[+] jjcm|1 year ago|reply
Recall suffered from a classic Microsoft mistake they've made time and again, but never learned from - how to correctly market and package your feature.

Microsoft always tends to "go big" with their integrations, often to their detriment, in order to increase adoption of new features. One notable time was with Windows 8. They really, REALLY wanted people to try out the new Metro UI, so they deeply integrated it into the OS, pushed it in every marketing campaign, and made it the first screen you saw on login. There were some great features in it - better performance and better search results, but it wasn't opt in. The reaction from customers who took a casual look was, "they removed the desktop!". It wasn't true, but because of how overzealous MS was to push the new feature, that became the takeaway.

The same thing is happening here - Microsoft pushed what objectively is a great tool, but they did so in a way that never gave users a choice of whether or not they wanted it. They've also framed the messaging and marketing in a way that's confusing to what is actually happening. Look at the amount of talk in this blogpost dedicated to mentioning how important security is for them, without ever actually going into what the security issues are or how they're addressing them.

Sloppy marketing + forced integration has bit Microsoft so many times now. I'm always shocked that they never learn from this.

[+] hn_throwaway_99|1 year ago|reply
The problem is not marketing. The problem is the tool is fundamentally not secure, and in my opinion, fundamentally not securable without major changes.

The core issue is that everyone has things on their computer that they want to be transient. I don't ever want my computer taking screenshots when I'm entering, say, my credit card number. More importantly, though, I oftentimes have text editors containing "scratch pads" that may contain sensitive data that I never want to persist.

Microsoft just never thought through the security implications of this feature.

[+] plopilop|1 year ago|reply
How is this objectively a great feature? This is a spyware that stores screenshots unencrypted (and thus accessible to any other spyware). I am also not convinced that the AI tools would have been offline, thus effectively sharing your whole data with Microsoft (even more than before).

From a privacy perspective, this feature is an abomination

[+] IAmNotACellist|1 year ago|reply
What's funny is if they had marketed it as Apple does (and had as much credibility as Apple does among their fans) then everyone would love it. I seriously doubt they intend to do much different than "Apple Intelligence." I.e., local access to all your data and uploads of data you use on cloud apps.
[+] Kwpolska|1 year ago|reply
With Windows 8, Microsoft thought that tablets and touchscreens were the future, and Metro was designed for those. Tablets being the future of computing meant they made the new experience the default. Turns out keyboards and mice are still vastly more popular a decade later.
[+] cubefox|1 year ago|reply
> The same thing is happening here - Microsoft pushed what objectively is a great tool, but they did so in a way that never gave users a choice of whether or not they wanted it.

Citation needed. I highly doubt this is true.

[+] tacocataco|1 year ago|reply
Can't users just not want a feature?

Why bother using psychological tricks to fool the user into compliance when you can just use that time and energy to make a better product?

[+] mihaaly|1 year ago|reply
> Microsoft pushed what objectively is a great tool

... excuse me!? Complete surveillance being a great tool?! Objectively great tool?! Maybe in China, yes.

[+] SimianSci|1 year ago|reply
For those who have not been keeping up with recent events. The United States government, is currently reevaluating its relationship with Microsoft due to recent security issues related to Russian and Chinese state-funded attacks.

[Microsoft Storm-0558 Incident, cited as a recent example] https://www.microsoft.com/en-us/security/blog/2023/07/14/ana...

Microsoft recently pledged to improve its security practices through incentives to executive pay and other initiatives.

[Microsoft Blog on recent Commitment] https://blogs.microsoft.com/on-the-issues/2024/06/13/microso...

Despite these pledges, several members of Congress are making it known that they dont see Microsoft as being serious about their recent commitments around security. It is worth noting that several of these members of congress influence how much Microsoft gets paid. The Recall feature is often used as a lightning rod to bring to light the rushed rollout of Microsoft's features without concern for security.

[Video with timestamp of Microsoft's President being questioned by Florida Congresswoman, Recall mentioned] https://youtu.be/kB2GCmasH4c?t=8217

While I suspect there may not be any sole reason for the release delay, it would seem to me that having Microsoft's biggest customer using Recall this way, may greatly influence the company's decision to hold off on the release.

[+] akira2501|1 year ago|reply
> improve its security practices through incentives to executive pay

Oh! It was lax executive pay that led to the problems.

[+] MP_1729|1 year ago|reply
Satya Nadella's Microsoft is such a weird company. It's like there's one side of it that is running with Zuckerberg's "move fast and break things" and the other side is saying "wait, we're the most important software company in the world! Things can't break!"
[+] nimbius|1 year ago|reply
In summary: the only customers that matter --corporations paying site licenses-- declared this to be an unacceptable business risk.

Anyone who is still using windows in 2024 and isnt a multinational business or llc gets what they deserve.

[+] 999900000999|1 year ago|reply
What if you can't afford a Mac, and you're not technically literate enough to install Ubuntu ?

Speaking for myself, I dual boot mint and windows because I really like playing games and making music. Both of those are absolutely subpar on Linux.

Outside of our nerd bubble, most normal people don't really want to run desktop Linux. Macs are great, but I can't really game on them.

[+] UberFly|1 year ago|reply
I'm neither of those things and Windows 10 Enterprise is working fine for me. Many of us (for now) are still able to corral our OS.
[+] frithsun|1 year ago|reply
What's interesting to me is that AI hype accidentally got non technical people thinking and talking more about their privacy and security concerns relating to software.

There's nothing sinister about LLMs relative to the kind of data collection big tech has been up to for years and years. It's just that all the AGI spin has triggered a defensive response in people.

Positive, in my opinion. People should be approaching tech privacy concerns with fear, uncertainty, and doubt.

[+] hbn|1 year ago|reply
There did not previously exist screenshots of everything my monitor displays any time I'm using my computer, and I don't want that data to exist. Sure, a lot of my activity could be pieced together from various other things that track my activity, but constant screenshots of everything that was on my monitor is a centralized goldmine of data that I don't want anyone to have access to.

I'd say that is more sinister than most other data collection.

[+] wruza|1 year ago|reply
Recall uses local AI models built into Windows 11 to screenshot mostly everything you see or do on your computer and then give you the ability to search and retrieve items you’ve seen. An explorable timeline lets you scroll through these snapshots to look back on what you did on a particular day on your PC. Everything in Recall is designed to remain local and private on-device, so no data is used to train Microsoft’s AI models.

https://www.theverge.com/2024/6/13/24178144/microsoft-window...

Had to look it up, sharing to save someone a minute.

[+] ChicagoDave|1 year ago|reply
This is only the beginning of AI-centric offerings that were oversold and will be delayed or quietly abandoned.

LLMs are nice for simple things, but they’ve already reached their limits. No amount of data will solve the iteration and complexity problems.

[+] visarga|1 year ago|reply
Meanwhile Apple Intelligence recalls across all apps with no backlash. I personally like this idea, should be done in a thoughtful and safe way, but recalling your logs is more useful than searching anew.

I see the same double standard with Google's generative search vs OpenAI's chatGPT with search - when Google gets it wrong, it's a big issue, but not for the other.

[+] porcoda|1 year ago|reply
I’m not sure Microsoft will ever achieve the level of trust they’d need to make things like this feature ever be acceptable. I’m sure in parts of the company they care about user trust quite a bit, but those people will never be able to counter the actions that the “maximize revenue at all costs” people take that undermine trust left and right. I don’t see them putting “build and maintain user trust” as a corporate goal that they ACTUALLY try to achieve (not just use as a corporate feel good statement), since “maximize shareholder value and revenue” will always win.
[+] 7thpower|1 year ago|reply
This is not a must have feature for me, but I am interested to see how it unfolds and I can definitely see it being useful in the future.

I do think they bungled the launch by not thinking through the security implications, and particularly how many sensitive threads this crosses.

That being said, they took a risk, it did not go over well, and they’re adjusting. I am sure I will get flamed, but I appreciate the approach.

[+] hnpolicestate|1 year ago|reply
You know what would be catastrophically bad? A Recall AI feature being baked into Android.

Like most people don't actually use personal computers anymore, even laptops aren't common among demos younger than millennials. I can tolerate switching to Linux or buying a steam deck.

But if this became a hard coded feature of android or iOS I'd have to give up smartphones entirely.

[+] Rinzler89|1 year ago|reply
What a dumb feature. They had to get all that backlash to understand why everyone wouldn't want it. Is someone at Microsoft taking crazy pills to think consumers would be into that?

They pulled the exact same shit 11 years ago when they launched the Xbox One as a "home media center" instead of a gaming console and it came with mandatory always-on internet connection, disc games DRM tied to a single console unable to lend them to a friend, and with Kinect camera, and just like this time, it took community backlash to get them to roll back on this shit while Sony was having the time of their lives seeing how the succes of the PS4 was already in the bag from the start before they evens started.

What is wrong with them? Does Microsoft think consumers are stupid masochists who enjoy being shit on by megacorporations while paying for the privilege? Does Nadella not look into the stupid decisions his execs are making and make necessary organizational adjustments to prevent stuff like this?

People shit on Steve Balmer but I don't remember Microsoft's products having that level of anti-consumer disrespect during his tenure. Sure Microsoft Zune and Window Phone 7-10 eventually flopped, by not because they had anti-consumer features but because they were too late and not very popular. And the Xbox 360, despite the Red ring of death was still smash hit. Now, Microsoft is an even richer company that during Balmer's tenure but it's products seem way more anti-consumer.

Edit: sorry for the overuse of the word shit, I'm just angry

[+] cedws|1 year ago|reply
If Apple did something like Recall, they'd run the whole thing on a separate secure chip somehow and encrypt the data with a (actual secure) enclave.

Microsoft are doing this the quick, dirty, lazy way by just embedding it into the OS. Lack of vertical integration is also haunting them.

[+] mrandish|1 year ago|reply
Once they do deploy it, I'll immediately disable it. And not even (primarily) because it's invasive and intrusive but because it'll be mostly useless to me AND it'll further clog my still-occasionally-laggy 4B cycle/sec PC with more unnecessary background tasks while sucking battery life and storage space for little benefit.

I resent that they continue to invest significant resources in buzzy new features like this (which no user asked for) to drive conceptual agendas decreed by MSFT leadership while continuing to ignore fixing the core Windows feature in this area: Search. The Windows built-in search function has always been so slow and limited it's nearly useless. It's so bad MSFT should be embarrassed. Meanwhile, the free Everything add-on shows exactly how it should be done - delivering global file search that's blisteringly fast, flexible and deeply capable. And it's written by one guy in his spare time (https://www.voidtools.com/).

Note: I assume above that MSFT will be forced to offer a way for enterprise IT departments to disable this via Policy Manager (because NSA and others with sensitive intel or IP will balk without a way to disable it).

[+] RcouF1uZ4gsC|1 year ago|reply
What is interesting is the contrast to Apple’s AI announcements.

Apple’s announcements were accompanied by an acknowledgment of the risk of privacy and a thorough analysis of the threat model and detailed design and specific steps taken to mitigate them. You can tell people with deep expertise spent time looking at the problem and coming up with solutions.

Microsoft Recall on the hand had the feeling of - Oh my, this has privacy implications, we never would have guessed???

That approach my Microsoft erodes trust. Apples’s approach builds trust.

[+] lemonlime0x3C33|1 year ago|reply
I have been dual booting for years but this has been my motivation to officially abandon windows at home, just need to figure out how to play civ 7 when it comes out next year...
[+] mihaaly|1 year ago|reply
Delay?!

Will recording and storing all your activity ever be secure? Suddenly in the future it will flip and become secure or what do they expect from the delay?

Ah, yes, I see, they want to delay until people are not concerned anymore. Cooking them slowly in increasingly worse privacy violations than throwing them into a hot one at once. Works with frogs. And users of modern gadgets.

Instead of waiting they'd better making it a product that those have no concern can download now, and perhaps even paid for in exchange for the mass of data they provide. It could even be a separate operating system lets say Windows 1984 that users are paid for using it. With webcam on all the time and no stars in password fields. Alternatively it could be called Windwos Beijing and have mass orders from the folks there.

[+] chx|1 year ago|reply
> Recall snapshots will only be decrypted and accessible when the user authenticates.

Question is, do you need to auth every time you try to access past snapshots? If not then this is still the mother lode for any infostealer.

And I do not think the danger Recall poses in an abusive relationship especially to women is adequately answered by "You can disable saving snapshots, pause them temporarily, filter applications and websites from being in snapshots, and delete your snapshots at any time" -- you'd need to know this thing exists and figure out how to pause. And I wonder whether the pause itself would leave a trace...