The proposed attack on Apple's protocol doesn't work. The user's device adds randomness when generating an outer encryption key for the voucher. Even if an adversary obtains both the hash set and the blinding key, they're just in the same position as Apple—only able to decrypt if there's a hash match. The paper could do a better job explaining how the ECC blinding scheme works.
This is one of the concerns in the OP, have an AI generate millions of variations of a certain kind of images and check the hashes. In this case it boils down to how common false positives neural hashes are.
Given some CP image, an attacker could perhaps morph it into an innocent looking image while maintaining the hash. Then spread this image on the web, and incriminate everybody.
It won't be long until these type of systems are mandated. Combined with a hardware root of trust it's not inconceivable that modifying your hardware not to report home will also be made a crime. It never stops with CSAM either, pretty soon it's terrorism and whatever vague new definition they use.
The focus on CSAM seems extremely hypocritical when authorities make such little effort to stop ongoing CSA. I would encourage everyone to research the Sophie Long case. Unless there is image or video evidence the police make little effort to investigate CSA because it's resource intensive.
Total surveillance is definitely the end goal of policing forces. It's in their very nature of getting their job done (what better way to catch criminals than a computer constantly scanning every move of everyone) and why people need to always push back against these "think of the children" scapegoats they use to get their foot in the door and get more control.
> It never stops with CSAM either, pretty soon it's terrorism and whatever vague new definition they use.
But PhotoDNA has been scanning cloud photos (Google, Dropbox, Microsoft, etc.,) to detect CSAM content for a decade now and this "pretty soon it's terrorism" slippery slope hasn't yet manifested, has it?
If the slope was going to be slippery, wouldn't we have seen some evidence of that by now?
Regardless of whether this attack works or not, you'd assume this scheme produces a wider attack surface against pictures in iCloud and against iCloud users. One attack I could imagine is a hacker uploading child porn to a hacked device to trigger immediate enforcement against a user (and sure, maybe there are more controls involved but would you carry around a very well-protected, well-designed hand grenade in your wallet just so you're bad, it'll explode).
Or even a hash collision with a banned image. Actually, if that could be generated this thing could fall apart pretty quickly if such collisions could be widely distributed.
For some reason, after reading the initial reporting on this system, I thought it was running against any photos on your iPhone, but now I read the actual paper, it seems like it only applies to photos destined to be uploaded to iCloud? So users can opt out by not using iCloud?
Much of the discussion is about how trivial it would be for Apple to start scanning any photos on the phone at a later date.
Right now they are able to bill this as doing what they currently do server side, but client side. Later, they can say they are simply applying the same "protections" to all photos instead of merely the ones being uploaded to iCloud.
Friendly reminder: until ios source code is closed all privacy claims is only backed by trust. They easily can do whatever they want if you're not compiling from source.
If Apple is to keep their word about guaranteeing the privacy of non-CSAM photos (which this whole discussion is about them not doing a very good job of), then they would only be able to do that with photos stored in iCloud because of this technical specification as to how the identification process works. That being said, other photos across your device are still monitored in a different way. For example Apple will scan photos that you send or receive via iMessage to automatically detect if they're nudes, and if you're underage, they will block them/send a notification to your parents.
As far as I know apple plans to put up 2 systems, one focused on phones of people age < 13 which filters "more or less" any photos and uses AI to detect explicit photos and one which looks for known child pornographic photos and for now seems to not necessary apply to all photos.
Yeah pretty much. Another way of thinking about it, is that to upload an image to iCloud, your phone must provide a cryptographic safety voucher to prove the image isn’t CSAM.
The question presumes the database leak also comes with the server side secret for blinding the CSAM database, which is unlikely (that’s not how HSMs work) and would be a general catastrophe (it would leak the Neural Hashes of photos in the NCMEC database, which are supposed to remain secret).
Yeah, I've worked with HSMs in the past and to say that it's a challenge to get key material out of them is an understatement. That said, a lot of this depends on the architecture surrounding the HSM - if the key material leaves the HSM at any point, you've basically increased your attack surface from an incredibly secure box to whatever your surrounding interfaces are. At Apple's scale, I have to imagine it's more economical to have some kind of envelope encryption - maybe this is the right attack vector for a malicious actor to hit?
The question doesn't presume that, as the the secret for blinding the CSAM database would only be helpful if a third party were also looking to see which accounts contained CSAM.
In this case, the question assumes that an attacker would more or less be creating their own database of hashes and derived keys (to search for and decrypt known photos and associate them with user accounts, or to bruteforce unknown photos), and would therefore have no need to worry about acquiring the key used for blinding the CSAM hash database.
Why does Apple even bother with encryption? They should just skip all of the warrant requirements etc and use their iCloud keys to unlock our content and store it unencrypted at rest.
Maybe they can also build an api so that governments can search easily for dissidents without the delays that the due process of law causes.
Facetious as this is, I can't imagine this is anything other than Apple's endgame here.
The best of both worlds: keep advertising their privacy chops to the masses, while also allowing any and every government agency a programmatic way to hash-verify the data passing through their systems in real-time.
[+] [-] jonathanmayer|4 years ago|reply
The proposed attack on Apple's protocol doesn't work. The user's device adds randomness when generating an outer encryption key for the voucher. Even if an adversary obtains both the hash set and the blinding key, they're just in the same position as Apple—only able to decrypt if there's a hash match. The paper could do a better job explaining how the ECC blinding scheme works.
[+] [-] jobigoud|4 years ago|reply
This is one of the concerns in the OP, have an AI generate millions of variations of a certain kind of images and check the hashes. In this case it boils down to how common false positives neural hashes are.
[+] [-] amelius|4 years ago|reply
Given some CP image, an attacker could perhaps morph it into an innocent looking image while maintaining the hash. Then spread this image on the web, and incriminate everybody.
[+] [-] kfprt|4 years ago|reply
The focus on CSAM seems extremely hypocritical when authorities make such little effort to stop ongoing CSA. I would encourage everyone to research the Sophie Long case. Unless there is image or video evidence the police make little effort to investigate CSA because it's resource intensive.
[+] [-] stjohnswarts|4 years ago|reply
[+] [-] zimpenfish|4 years ago|reply
But PhotoDNA has been scanning cloud photos (Google, Dropbox, Microsoft, etc.,) to detect CSAM content for a decade now and this "pretty soon it's terrorism" slippery slope hasn't yet manifested, has it?
If the slope was going to be slippery, wouldn't we have seen some evidence of that by now?
[+] [-] joe_the_user|4 years ago|reply
[+] [-] selsta|4 years ago|reply
[+] [-] mnd999|4 years ago|reply
[+] [-] jl6|4 years ago|reply
[+] [-] foerbert|4 years ago|reply
Right now they are able to bill this as doing what they currently do server side, but client side. Later, they can say they are simply applying the same "protections" to all photos instead of merely the ones being uploaded to iCloud.
[+] [-] tandav|4 years ago|reply
[+] [-] RegnisGnaw|4 years ago|reply
[+] [-] NTroy|4 years ago|reply
[+] [-] sharikone|4 years ago|reply
[+] [-] dathinab|4 years ago|reply
But I haven't looked to closely into it.
[+] [-] avianlyric|4 years ago|reply
[+] [-] shuckles|4 years ago|reply
[+] [-] gorgonzolachz|4 years ago|reply
[+] [-] NTroy|4 years ago|reply
In this case, the question assumes that an attacker would more or less be creating their own database of hashes and derived keys (to search for and decrypt known photos and associate them with user accounts, or to bruteforce unknown photos), and would therefore have no need to worry about acquiring the key used for blinding the CSAM hash database.
[+] [-] ashneo76|4 years ago|reply
But think of the children and security of the society. Couple that with constant monitoring of your car and you can be monitored anywhere
[+] [-] kaba0|4 years ago|reply
[+] [-] kook_throwaway|4 years ago|reply
[+] [-] floatingatoll|4 years ago|reply
[+] [-] whatever1|4 years ago|reply
Maybe they can also build an api so that governments can search easily for dissidents without the delays that the due process of law causes.
[+] [-] laurent92|4 years ago|reply
[+] [-] x2r|4 years ago|reply
[+] [-] gorgonzolachz|4 years ago|reply
The best of both worlds: keep advertising their privacy chops to the masses, while also allowing any and every government agency a programmatic way to hash-verify the data passing through their systems in real-time.
[+] [-] roody15|4 years ago|reply