Any new "defense" that claims to use adversarial perturbations to undermine GenAI training should have to explain why this paper does not apply to their technique: https://arxiv.org/pdf/2406.12027
The answer is, almost unfailingly, "this paper applies perfectly to our technique because we are just rehashing the same ideas on new modalities". If you believe it's unethical for GenAI models to train on people's music, isn't is also unethical to trick those people into posting their music online with a fake "defense" that won't actually protect them?
You are assuming input-transformation based defenses in the image domain transfer to the music recognition domain, when we know they don't automatically even transfer to the speech recognition domain.
But 'protection' of any one song isn't the entire point. It only takes less than a fraction of a percent of corpus data to have persistent long term effects in the final model, or increase costs and review requirements to those stealing their content.
As most training is unsupervised, because the cost and limited access to quality, human labeled data, it wouldn't take much if even some obscure, limited market, older genres which still have active fan bases, like Noise rock to start filtering into recommendation engines and impact user satisfaction.
Most of the speech protections, just force attacks to be in the perceptible audio range, with lo-fi portions like those of TripHop, that would be non-detectable without the false positive rate going way up. With bands like Arab On Radar, Shellac, or The Oxes, it wouldn't be detectable.
But it is also like WAFs/AV software/IDS. The fact that it can't help with future threats today is immaterial. Any win of these leaches has some value.
Obviously any company intentionally applying even the methods in your linked paper to harvest protected images would be showing willful intent to circumvent copyright protections and I am guessing most companies will just toss any file that it thinks has active protections just because how sensitive training is.
Most musicians also know that copyright only protects the rich.
I am ignorant here, this is a genuine question - is there any reason to assume that a paper solely about image mimicry can be blanket-applied, as OP is doing, to audio mimicry?
Some of the sibling comments had questions around purposefully releasing defenses which don’t work. I think Carlini’s (one of the paper authors) post can add some important context: https://nicholas.carlini.com/writing/2024/why-i-attack.html.
TLDR: Once these defenses are broken, all previously protected work is perpetually unprotected, so they are flawed at a foundational level.
Ignoring these arguments and pretending they don’t exist is pretty unethical.
I like Benn Jordan because he’s clearly got a grasp on a functional understanding of machine learning, but that’s not his primary background. He comes from a music production background, so his focus is more practical and results-oriented.
It will be really interesting as this knowledge percolates into more and more fields, what domain experts do with it. I see ML as more of a bag of tricks that can be applied to many fields.
>He comes from a music production background, so his focus is more practical and results-oriented
It's his art and his livelihood too, so it's also personal. These people want to steal his art and create a world full of soulless cheap muzak, while simultaneously putting him out of work.
Benn is one of my fave subscriptions on YouTube--both for the (now more occasional) music gear stuff and for the in-depth music industry education. The fact that he has been hacking away at IP and AI stuff for ages is just icing on the cake.
All this stuff is snake oil, either already, or eventually.
There's new models showing up regularly. Civitai recognizes 33 image models at this point, and audio will also see multiple developments. Any successful attack on a model isn't guaranteed to apply to another one, not even yet invented. There's also a multitude of possible pre-processing methods and their combinations for any piece of media.
There's also the difficulty of attacking a system that's not well documented. Not every model out there is open source and available for deep analysis.
And it's hard to attack something that doesn't yet exist, which means countermeasures will come up only after a model was already successfully created. This is I'm sure of some academic interest, but the practical benefits seem approximately none.
Since information is trivially stored, anyone having any trouble could just download the file today and sit on it for a year or two not doing anything at all, just waiting for a new model to show up.
The problem is that copyright is the law of the land, and it demands our participation.
Because of that reality, every artist who wants to make money must either participate in it, or completely isolate themselves from it.
These models have become an incredible opportunity for giant corporations to circumvent the law. By training a model on a copyrighted work, you can launder that work into your own new work, and make money from it without sharing that money with the original artists. Obviously, this is an incredibly immoral end to copyright as we know it.
So what are we going to do about this situation? Are we really going to keep pretending that copyright can work? It wasn't even working before all the AI hype! Ever heard the words "starving artist"? Of course you have!
We need a better system than copyright. I'm convinced that no system at all (anarchy) would be a superior option at this point. If not now, then when?
> By training a model on a copyrighted work, you can launder that work into your own new work, and make money from it without sharing that money with the original artists.
Not sure if "you" refers to model developers, hosting company or end users. But let's see each one of them in turn
- model development is a cost center, there is no profit yet
- model deployment brings little profit, they make cents per million tokens
- applying the model to your own needs - that is where the benefit goes.
So my theory is that benefits follow the problem, it is in the application layer. Have a need, you can benefit from AI, don't need it, no benefit. Like Linux. You got to use it for something. And that usage, that problem - is personal. You can't sell your problems, they remain yours. It is hard to quantify how people benefit from AI, it could be for fun, for learning, professional use, or for therapy.
Most gen-AI usage is seen by one person exactly once. Think about that. It's not commercial, it's more like augmented imagination. Who's gonna pay for AI generated stuff when it is so easy to make your own.
adversarial noise is very popular in the media but imo is a complete dead end for the desired goals - representations do not transfer between different models this easily
IP is such a stupid concept. How does it make any sense of that an artist could own the right to let people learn from his music. The idea of an artist getting to choose who can and can't learn from their song is so patently absurd.
I hope that the adversarial attacks can be easily detected and circumvented, just like other IP protection measures have been subverted successfully.
Exclusive rights over their published work encourages artists and inventors to publish their work, which is a clear benefit to society at large. The period of time it should remain exclusive and the specific rights that are made exclusive can be debated, but the utility of IP rights in general is obvious.
And generative AI is not a person in the first place, so I don't think the appeal to learning makes much sense here.
This is such a radical take on IP rights and AI "learning" that I can only assume you're consciously choosing to misunderstand both.
On the off chance that you are not: IP-rights does not cover "learning from" a source. What ML does is not in any way akin to human learning in methodology. When we call it learning that's an analogy. You can not argue a legal case from analogy alone.
> artist could own the right to let people learn from his music.
They don't, what's happening here is their music is being fed to a computer program in a for-profit venture.
This anthropomorphism of LLMs is concerning. What you're actually implying here is that you believe some computer programs should be awarded the same rights as humans. You can't just skip that like it's some kind of foregone conclusion. You have to defend it. And, it's not easy.
I had a client train an AI on images I created without extended usage and so added adversarial noise to the images next time around. The models I tested with misclassified the images and image generation seemed broken, so Im curious how it impacted their attempts, if they even attempted it again, I don't know. I don't expect them to come to me and ask why their model is so interested in ducks...
Benn Jordan is a musician who is probably one of the most critical of the current copyright regime in his space. For context, see https://www.youtube.com/watch?v=PJSTFzhs1O4
Copyright exists to enrich the interests of the publishers of a work, not the artists they funded. A long time ago, copyright was a sufficient legal tool to bring publishers to artists' heels, but no longer. Long copyright terms and the imbalance of power between different wealthy interests allowed publishers to usurp and alienate artists' ownership over their work. And the outsized amount of commercial interest in current generative AI tools comes down to the fact that publishers believe they can use them to strip what little ownership interest authors have left. What Benn is doing is looking for new tools to bring publishers to heel.
IP is fundamentally a social contract, subject to perpetual renegotiation through action and counter-action. If you told any game publisher in the early 2000s, during the height of the Napster Wars, that they'd be proudly allowing randos on the Internet to stream video of their games being played, they'd laugh in your face. But people did it anyway, and everyone in the games biz realized it's not worth fighting people who are adding to your game. Even Nintendo, notorious IP tightwads as they are, tried scraping micropennies off the top of streamers and realized it's a fool's errand.
The statement Benn is making is pretty clear. You can either...
- Negotiate royalties for, and purchase training data from, actual artists, who will then in exchange give you high-quality training data, or,
- Spend increasing amounts of time fighting to filter an increasingly polluted information ecosystem to have a model that only sorta kinda replicates the musical landscape of the late 2010s.
A lot of us are reflexively inclined to hate on anything "copyright-shaped" because of our experiences over the past few decades. Publishers wanted to go back to the days of copyright being a legal tool of arbitrary and capricious punishment. But that doesn't mean that everything that might fall afoul of copyright law is automatically good or that generative AI companies are trying to liberate creativity. They're trying to monopolize it, just like Web 2.0 "disintermediation" was turned into "here's five websites with screenshots of the other four". That's why so much money is being poured into these companies and why a surprisingly nonzero amount of copyright reformists also have deeply negative opinions of AI.
It's been struggling since the internet became a thing. People got more content than they can consume. For any topic there are 1000 alternative sites, most of them free. Any new work competes against decades of backlog. Under this attention scarcity mode, artists devolve into enshittification because they hunt ad money, while royalties are a joke.
On the other hand, people stopped being passive consumers, we like to interact now. Online games, social networks, open source, wikipedia and scientific publication - they all run in a permissive mode. How could we do anything together if we all insisted on copyright protection?
We like to make most of our content ourselves, we don't need the old top-down model of content creation. We attach "reddit" to our searches because we value comments more than official sources. It's an interactive world where LLMs fit right in, being interactive and contextually adaptive.
Even if we completely toss out IP, isn't an artist free to release whatever version of their content they want? I find that AI poisoning seems to be a speed bump at best, but I don't see an issue with artists using it if they wish. If anything, it gives a fun challenge for machine learning developers to try.
Imnimo|10 months ago
The answer is, almost unfailingly, "this paper applies perfectly to our technique because we are just rehashing the same ideas on new modalities". If you believe it's unethical for GenAI models to train on people's music, isn't is also unethical to trick those people into posting their music online with a fake "defense" that won't actually protect them?
nyrikki|10 months ago
But 'protection' of any one song isn't the entire point. It only takes less than a fraction of a percent of corpus data to have persistent long term effects in the final model, or increase costs and review requirements to those stealing their content.
As most training is unsupervised, because the cost and limited access to quality, human labeled data, it wouldn't take much if even some obscure, limited market, older genres which still have active fan bases, like Noise rock to start filtering into recommendation engines and impact user satisfaction.
Most of the speech protections, just force attacks to be in the perceptible audio range, with lo-fi portions like those of TripHop, that would be non-detectable without the false positive rate going way up. With bands like Arab On Radar, Shellac, or The Oxes, it wouldn't be detectable.
But it is also like WAFs/AV software/IDS. The fact that it can't help with future threats today is immaterial. Any win of these leaches has some value.
Obviously any company intentionally applying even the methods in your linked paper to harvest protected images would be showing willful intent to circumvent copyright protections and I am guessing most companies will just toss any file that it thinks has active protections just because how sensitive training is.
Most musicians also know that copyright only protects the rich.
tptacek|10 months ago
https://securitycryptographywhatever.com/2025/01/28/cryptana...
jjulius|10 months ago
nickpadge|10 months ago
TLDR: Once these defenses are broken, all previously protected work is perpetually unprotected, so they are flawed at a foundational level.
Ignoring these arguments and pretending they don’t exist is pretty unethical.
nemomarx|10 months ago
unknown|10 months ago
[deleted]
janalsncm|10 months ago
It will be really interesting as this knowledge percolates into more and more fields, what domain experts do with it. I see ML as more of a bag of tricks that can be applied to many fields.
dingnuts|10 months ago
It's his art and his livelihood too, so it's also personal. These people want to steal his art and create a world full of soulless cheap muzak, while simultaneously putting him out of work.
Get 'em, Benn! I should go buy one of his albums.
rcarmo|10 months ago
propter_hoc|10 months ago
The Flashbulb - Parkways: https://youtu.be/C6pzg7I61FI
dale_glass|10 months ago
There's new models showing up regularly. Civitai recognizes 33 image models at this point, and audio will also see multiple developments. Any successful attack on a model isn't guaranteed to apply to another one, not even yet invented. There's also a multitude of possible pre-processing methods and their combinations for any piece of media.
There's also the difficulty of attacking a system that's not well documented. Not every model out there is open source and available for deep analysis.
And it's hard to attack something that doesn't yet exist, which means countermeasures will come up only after a model was already successfully created. This is I'm sure of some academic interest, but the practical benefits seem approximately none.
Since information is trivially stored, anyone having any trouble could just download the file today and sit on it for a year or two not doing anything at all, just waiting for a new model to show up.
ben_w|10 months ago
Seems like an awful risk to deliberately strip such markings. It's a kind of DRM, and breaking DRM is illegal in many countries.
thomastjeffery|10 months ago
Because of that reality, every artist who wants to make money must either participate in it, or completely isolate themselves from it.
These models have become an incredible opportunity for giant corporations to circumvent the law. By training a model on a copyrighted work, you can launder that work into your own new work, and make money from it without sharing that money with the original artists. Obviously, this is an incredibly immoral end to copyright as we know it.
So what are we going to do about this situation? Are we really going to keep pretending that copyright can work? It wasn't even working before all the AI hype! Ever heard the words "starving artist"? Of course you have!
We need a better system than copyright. I'm convinced that no system at all (anarchy) would be a superior option at this point. If not now, then when?
visarga|10 months ago
Not sure if "you" refers to model developers, hosting company or end users. But let's see each one of them in turn
- model development is a cost center, there is no profit yet
- model deployment brings little profit, they make cents per million tokens
- applying the model to your own needs - that is where the benefit goes.
So my theory is that benefits follow the problem, it is in the application layer. Have a need, you can benefit from AI, don't need it, no benefit. Like Linux. You got to use it for something. And that usage, that problem - is personal. You can't sell your problems, they remain yours. It is hard to quantify how people benefit from AI, it could be for fun, for learning, professional use, or for therapy.
Most gen-AI usage is seen by one person exactly once. Think about that. It's not commercial, it's more like augmented imagination. Who's gonna pay for AI generated stuff when it is so easy to make your own.
discardedrefuse|10 months ago
https://www.youtube.com/watch?v=PJSTFzhs1O4
KennyBlanken|10 months ago
https://www.youtube.com/watch?v=xMYm2d9bmEA
emsign|10 months ago
throw_m239339|10 months ago
whimsicalism|10 months ago
dijksterhuis|10 months ago
the [transferability] rates just drop off significantly for audio (always felt it was a similar vibe to RNN ‘vanishing gradients’)
edit — specifically mention transferability
unknown|10 months ago
[deleted]
constantcrying|10 months ago
I hope that the adversarial attacks can be easily detected and circumvented, just like other IP protection measures have been subverted successfully.
charonn0|10 months ago
And generative AI is not a person in the first place, so I don't think the appeal to learning makes much sense here.
delusional|10 months ago
On the off chance that you are not: IP-rights does not cover "learning from" a source. What ML does is not in any way akin to human learning in methodology. When we call it learning that's an analogy. You can not argue a legal case from analogy alone.
const_cast|10 months ago
They don't, what's happening here is their music is being fed to a computer program in a for-profit venture.
This anthropomorphism of LLMs is concerning. What you're actually implying here is that you believe some computer programs should be awarded the same rights as humans. You can't just skip that like it's some kind of foregone conclusion. You have to defend it. And, it's not easy.
spacecadet|10 months ago
kmeisthax|10 months ago
Copyright exists to enrich the interests of the publishers of a work, not the artists they funded. A long time ago, copyright was a sufficient legal tool to bring publishers to artists' heels, but no longer. Long copyright terms and the imbalance of power between different wealthy interests allowed publishers to usurp and alienate artists' ownership over their work. And the outsized amount of commercial interest in current generative AI tools comes down to the fact that publishers believe they can use them to strip what little ownership interest authors have left. What Benn is doing is looking for new tools to bring publishers to heel.
IP is fundamentally a social contract, subject to perpetual renegotiation through action and counter-action. If you told any game publisher in the early 2000s, during the height of the Napster Wars, that they'd be proudly allowing randos on the Internet to stream video of their games being played, they'd laugh in your face. But people did it anyway, and everyone in the games biz realized it's not worth fighting people who are adding to your game. Even Nintendo, notorious IP tightwads as they are, tried scraping micropennies off the top of streamers and realized it's a fool's errand.
The statement Benn is making is pretty clear. You can either...
- Negotiate royalties for, and purchase training data from, actual artists, who will then in exchange give you high-quality training data, or,
- Spend increasing amounts of time fighting to filter an increasingly polluted information ecosystem to have a model that only sorta kinda replicates the musical landscape of the late 2010s.
A lot of us are reflexively inclined to hate on anything "copyright-shaped" because of our experiences over the past few decades. Publishers wanted to go back to the days of copyright being a legal tool of arbitrary and capricious punishment. But that doesn't mean that everything that might fall afoul of copyright law is automatically good or that generative AI companies are trying to liberate creativity. They're trying to monopolize it, just like Web 2.0 "disintermediation" was turned into "here's five websites with screenshots of the other four". That's why so much money is being poured into these companies and why a surprisingly nonzero amount of copyright reformists also have deeply negative opinions of AI.
jeremyjh|10 months ago
ForHackernews|10 months ago
SCdF|10 months ago
visarga|10 months ago
It's been struggling since the internet became a thing. People got more content than they can consume. For any topic there are 1000 alternative sites, most of them free. Any new work competes against decades of backlog. Under this attention scarcity mode, artists devolve into enshittification because they hunt ad money, while royalties are a joke.
On the other hand, people stopped being passive consumers, we like to interact now. Online games, social networks, open source, wikipedia and scientific publication - they all run in a permissive mode. How could we do anything together if we all insisted on copyright protection?
We like to make most of our content ourselves, we don't need the old top-down model of content creation. We attach "reddit" to our searches because we value comments more than official sources. It's an interactive world where LLMs fit right in, being interactive and contextually adaptive.
aezart|10 months ago
SkyBelow|10 months ago
nkrisc|10 months ago
nemomarx|10 months ago
whimsicalism|10 months ago