> Neural hash generated here might be a few bits off from one generated on an iOS device. This is expected since different iOS devices generate slightly different hashes anyway. The reason is that neural networks are based on floating-point calculations. The accuracy is highly dependent on the hardware. For smaller networks it won't make any difference. But NeuralHash has 200+ layers, resulting in significant cumulative errors.
This is a little unexpected. I'm not sure whether this has any implication on CSAM detection as whole. Wouldn't this require Apple to add multiple versions of NeuralHash of the same image (one for each platform/hardware) into the database to counter this issue? If that is case, doesn't this in turn weak the threshold of the detection as the same image maybe match multiple times in different devices?
This may explain why they (weirdly), only announced it for iOS and iPadOS, as far as I can tell they didn't announce it for macOS.
My first thought was that they didn't want to make the model too easily accessible by putting it on macOS, in order to avoid adversarial attacks.
But knowing this now, Intel Macs are an issue as (not as I previously wrote because they differ in floating point implementation to ARM, thanks my123 for the correction) they will have to run the network on a wide variety of GPUs (at the very least multiple AMD archs and Intel's iGPU), so maybe that also factored in their decision ? They would have had to deploy multiple models and (I believe, unless they could make the models exactly converge ?) multiple distinct database server side to check back.
To people knowledgeable on the topic, would having two versions of the models increase the attack surface ?
Edit: Also, I didn't realise that because of how perceptual hashes worked, they would need to have their own threshold to matching, independent of the "30 pictures matched to launch a human review". Apple's communication push implied exact matches. I'm not sure they used the right tool here (putting aside the fact for now that this is running client side).
This basically invalidates any claims Apple made about accuracy, and brings up an interesting point about the hashing mechanism: it seems two visually similar images will also have similar hashes. This is interesting because humans quickly learn such patterns: for example, many here will know what dQw4w9WgXcQ is without thinking about it at all.
> Wouldn't this require Apple to add multiple versions of NeuralHash of the same image (one for each platform/hardware) into the database to counter this issue?
Not if their processor architectures are all the same, or close enough that they can write (and have written) an emulation layer to get bit-identical behaviour.
I think it would just require generating the table of hashes once on each type of hardware in use (whether CPU or GPU), then doing the lookup only in the table that matches the hardware that generated it.
Now that the model is known, I wonder how hard it is to create "adversarial collisions": given an image and a target hash, perturb the image in a way that is barely perceptible for a human, so that it matches the target hash.
It might be useful to read the threat model document. Associated data from client neural hash matches are compared with the known CSAM database again on the server using a private perceptual hash before being forwarded to human reviewers, so all such an attack would do is expose non-private image derivatives to Apple. It would likely not put an account at risk for referral to NCMEC. In this sense, privacy is indeed preserved versus other server scanning solutions where an adversarial perceptual hash (PhotoDNA is effectively public as well, per an article shared on HN) would trigger human review of all the account’s data.
See https://news.ycombinator.com/item?id=28105849, which shows a POC to generate adversarial collisions for any neural network based perceptual hash scheme. The reason it works is because "(the network) is continuous(ly differentiable) and vulnerable to (gradient-)optimisation based attack".
Spy agencies could add CSAM images adversarially modified to match legit content they want to find. Then they need to have someone in Apple's team to intercept the reports. This way they can scan for any image.
So, these adversarial collisions are the images that I need to send to my enemies so that they go to prison when they upload those images to iCloud? It seems trivially easy to exploit.
An interesting tidbit: "Believe it or not, [the NeuralHash algorithm for on-device CSAM detection] already exists as early as iOS 14.3, hidden under obfuscated class names."
In addition to generate the adversarial collisions, someone mentioned that it can also be used to train a decoder network to reverse any NeuralHash back to its input image.
> ONNX is an open format built to represent machine learning models. ONNX defines a common set of operators - the building blocks of machine learning and deep learning models - and a common file format to enable AI developers to use models with a variety of frameworks, tools, runtimes, and compilers.
- onnx.ai
I have never heard of this before and had to look it up. Is it widely used? Can I define a model in onnx and run it "everywhere", instead of learning pytorch or tensorflow?
Yes. Could be that you have to check what operators is available when you move from one framework to another using ONNX. One example is ONNX.js that make it possible to run the model using JavaScript, made by Microsoft. If the operator is not available (ex. ArgMax for WebGL) you have to switch to something equivalent and retrain. List of supported operators for ONNX.js:
https://github.com/microsoft/onnxjs/blob/master/docs/operato...
Onnx has really poor support by microsoft. I suspect they've basically abandoned it internally (their onnxjs variant is orders of magnitude slower than tfjs[1]). It's a good 'neutral standard' for the moment but we should all eventually probably move away from it long term.
ONNX is a representation and interchange format that training/inference frameworks can use to represent models. However, ONNX does not have a tensor engine of its own. So you would still define and train your model in tensorflow, pytorch, etc, and then save the model in ONNX format, at which point it can be ported to any inference service that supports the ONNX format.
I think you need to use pytorch or tensorflow to create the network and train it. After training you export it to ONNX. I suggest you to convert from ONNX to Openvino.
For those wondering why you can't just delete these files. On Mac OS Big Sur and later, the system OS is signed by apple and in order to actually delete these files you need to go through a whole bunch of steps that apparently requires completely disabling FileVault (full disk encryption).
On Apple Silicon Macs this isn’t true, as files are encrypted on disk with “data protection” (same thing as on iOS). You can enable FileVault but it’s just extra.
Also I’ve edited /etc without disabling FileVault, is it just /System which is protected this way?
Here's an interesting adversarial attack. If we reverse engineer the Apple NeuralHash format and replace these files, we could create a system that does a DDoS attack on Apple's manual verification system by flooding the system with false positives caused from a faulty NeuralHash. This would overload Apple's manual review system and effectively make it uneconomical to run.
Winning would always be easy if you didn't have an adversary. Apple (having access to the original low resolution photo) could build a relatively simple mechanism add a filter in their verification pipeline.
Even if they didn't have access to the original (for whatever reason), they train their own learning algorithm (supervised by their manual verification checkers) to detect the fake submissions.
How does this actually help society? After this announcement child abusers won't user their iPhone for this stuff (can't believe they did in the first place).
Their approach is so poorly targeted at the claimed problem, and so ineffectual, I don’t think it is reasonable to take them at face value re: what they say they are trying to accomplish.
For the folks who are interested in stopping abuse of children, there are many other approaches that would break the market for new abuse and new CSAM. This just isn’t going to move the needle and I have to assume they know that.
I’ve completely lost trust in Apple because I can’t understand what their motivations are. I _do_ understand the technology, so I’m pretty tired of articles suggesting this is some sort of misunderstanding and not Apple taking a giant leap towards enabling authoritarianism, and of course building literal thoughtcrime enforcement into end user devices, which is beyond even what 1984 imagined.
Why do so many people get caught sharing CSAM on FB then?
I remember reading about the CSAM ring the FBI (IIRC) infiltrated and ran for a period of time, that group had strict rules on how to access and share material that, if followed, would have completely protected them but the majority of them were sloppy and got caught. Criminals really aren't that smart by and large. Will this catch the smartest of them? Probably not, but it will catch a good number I'm sure.
All that said, I'm not a fan of these changes, I just dislike arguments that don't hold water against it.
If I’m reading this right it seems like the neural hash runs on both MacOS and iOS, since the weights can be found on both systems. I though the neuralhash was only running on iOS?
Does apple even encrypt the actual image data on device? Their system document says "payload = visual derivative + neural hash" and only that is encrypted with a secondary level encryption. And they didn't go through with e2ee for icloud last I heard. This elaborate system makes no sense if they very well could have done it in cloud.
It feels like elaborate privacy theatre trojan horse to introduce in device surveillance.
Simplicity is a good thing. One of the perception hashes that I found in an URL on HN was literally just compressing the images, converting to grescale, calculating the hamming distance and coalescing that into an n bit hash.
Several articles stated that given a perceptual hash you could somehow reverse it into a (very) low resolution image. However the README provides an example hash and it's only 24 characters. How is that possible?
Really disgusting idea: I wonder if it's possible for someone to use this as a 'discriminator' in a GAN to configure a generator to recreate the CP this is trying to avoid distributing in the first place.
welp. i am no iphone user, nor a dev. just a random guy trying to wrap my head around the concept of this trojan horse of "CSAM" which WILL be used by tyrants and crony governments to spy and persecute their citizens and apple will "have to obey the law", something they had no business interfering with the rights of customers earlier, now they are active participants.
How much do you want to bet google will bring something similar to this to "keep up with the industry demands and partners requests". That would be the day either i go full lineageOS if they decide to not join the party or a dumb flip phone for ever. i will not subject myself to this because i know the government "WILL" hunt me down for being a dissident.
[+] [-] dang|4 years ago|reply
Apple defends anti-child abuse imagery tech after claims of ‘hash collisions’ - https://news.ycombinator.com/item?id=28225706
Hash collision in Apple NeuralHash model - https://news.ycombinator.com/item?id=28219068
[+] [-] xucheng|4 years ago|reply
This is a little unexpected. I'm not sure whether this has any implication on CSAM detection as whole. Wouldn't this require Apple to add multiple versions of NeuralHash of the same image (one for each platform/hardware) into the database to counter this issue? If that is case, doesn't this in turn weak the threshold of the detection as the same image maybe match multiple times in different devices?
[+] [-] cwizou|4 years ago|reply
My first thought was that they didn't want to make the model too easily accessible by putting it on macOS, in order to avoid adversarial attacks.
But knowing this now, Intel Macs are an issue as (not as I previously wrote because they differ in floating point implementation to ARM, thanks my123 for the correction) they will have to run the network on a wide variety of GPUs (at the very least multiple AMD archs and Intel's iGPU), so maybe that also factored in their decision ? They would have had to deploy multiple models and (I believe, unless they could make the models exactly converge ?) multiple distinct database server side to check back.
To people knowledgeable on the topic, would having two versions of the models increase the attack surface ?
Edit: Also, I didn't realise that because of how perceptual hashes worked, they would need to have their own threshold to matching, independent of the "30 pictures matched to launch a human review". Apple's communication push implied exact matches. I'm not sure they used the right tool here (putting aside the fact for now that this is running client side).
[+] [-] eurasiantiger|4 years ago|reply
[+] [-] wizzwizz4|4 years ago|reply
Not if their processor architectures are all the same, or close enough that they can write (and have written) an emulation layer to get bit-identical behaviour.
[+] [-] csmpltn|4 years ago|reply
[+] [-] therealcamino|4 years ago|reply
[+] [-] varispeed|4 years ago|reply
[+] [-] enriquto|4 years ago|reply
[+] [-] y7|4 years ago|reply
[+] [-] misterdata|4 years ago|reply
See also here: https://gist.github.com/unrealwill/c480371c3a4bf3abb29856c29...
[+] [-] shuckles|4 years ago|reply
[+] [-] xucheng|4 years ago|reply
[+] [-] visarga|4 years ago|reply
[+] [-] tvirosi|4 years ago|reply
[+] [-] dsign|4 years ago|reply
[+] [-] xucheng|4 years ago|reply
[+] [-] gary17the|4 years ago|reply
[+] [-] xucheng|4 years ago|reply
[+] [-] wodenokoto|4 years ago|reply
- onnx.ai
I have never heard of this before and had to look it up. Is it widely used? Can I define a model in onnx and run it "everywhere", instead of learning pytorch or tensorflow?
[+] [-] punnerud|4 years ago|reply
[+] [-] tvirosi|4 years ago|reply
[1] https://github.com/microsoft/onnxjs/issues/304
[+] [-] atty|4 years ago|reply
[+] [-] Loranubi|4 years ago|reply
[+] [-] red2awn|4 years ago|reply
[+] [-] figomore|4 years ago|reply
[+] [-] toxik|4 years ago|reply
> Resizing the image yields same hash, even down to below 200x100. Cropping or rotating the image yields different hashes.
[+] [-] spuz|4 years ago|reply
[+] [-] mlindner|4 years ago|reply
https://apple.stackexchange.com/questions/395508/can-i-mount...
So in the end we'll be left with a choice.
1. Allow Apple to scan your files.
2. Disable any kind of encryption letting anyone who steals your laptop access all your files.
[+] [-] helen___keller|4 years ago|reply
Maybe I'm missing context, what files are you referring to?
[+] [-] easton|4 years ago|reply
Also I’ve edited /etc without disabling FileVault, is it just /System which is protected this way?
[+] [-] djrogers|4 years ago|reply
[+] [-] mlindner|4 years ago|reply
[+] [-] scoopertrooper|4 years ago|reply
Even if they didn't have access to the original (for whatever reason), they train their own learning algorithm (supervised by their manual verification checkers) to detect the fake submissions.
[+] [-] mrits|4 years ago|reply
[+] [-] n8cpdx|4 years ago|reply
For the folks who are interested in stopping abuse of children, there are many other approaches that would break the market for new abuse and new CSAM. This just isn’t going to move the needle and I have to assume they know that.
I’ve completely lost trust in Apple because I can’t understand what their motivations are. I _do_ understand the technology, so I’m pretty tired of articles suggesting this is some sort of misunderstanding and not Apple taking a giant leap towards enabling authoritarianism, and of course building literal thoughtcrime enforcement into end user devices, which is beyond even what 1984 imagined.
[+] [-] joshstrange|4 years ago|reply
I remember reading about the CSAM ring the FBI (IIRC) infiltrated and ran for a period of time, that group had strict rules on how to access and share material that, if followed, would have completely protected them but the majority of them were sloppy and got caught. Criminals really aren't that smart by and large. Will this catch the smartest of them? Probably not, but it will catch a good number I'm sure.
All that said, I'm not a fan of these changes, I just dislike arguments that don't hold water against it.
[+] [-] doctoboggan|4 years ago|reply
[+] [-] laughingman2|4 years ago|reply
It feels like elaborate privacy theatre trojan horse to introduce in device surveillance.
[+] [-] c7DJTLrn|4 years ago|reply
[+] [-] isatty|4 years ago|reply
[+] [-] hendersoon|4 years ago|reply
[+] [-] tvirosi|4 years ago|reply
[+] [-] ryeguy_24|4 years ago|reply
[+] [-] ryeguy_24|4 years ago|reply
[+] [-] _fat_santa|4 years ago|reply
[+] [-] gjsman-1000|4 years ago|reply
[+] [-] suifbwish|4 years ago|reply
[+] [-] 2Gkashmiri|4 years ago|reply
How much do you want to bet google will bring something similar to this to "keep up with the industry demands and partners requests". That would be the day either i go full lineageOS if they decide to not join the party or a dumb flip phone for ever. i will not subject myself to this because i know the government "WILL" hunt me down for being a dissident.
[+] [-] simondotau|4 years ago|reply