Claiming they have the unrestricted right to scrape whatever information they want off the internet but complaining about it when others do to you and bringing out the 'China bad' card, just ironic
Somebody’s nervous about the competition, so they’re running a PR campaign. To me, though, this campaign hurts Anthropic more than anyone else. How can I trust the output of a model they can modify at will for certain clients? And who are they, after scraping the entire web, to complain about theft?
This violates the ToS, but I don't think it's distillation. Distillation requires knowing the logits, which current API does not provide. This is just synthetic data generation. Anthropic definitely knows the difference.
Yes, it is annoying that companies keep calling it “distillation” when it’s really imitation learning. In fact the closest analogy is probably more like “scraping” which is pretty ironic.
I find this extremely concerning: "Countermeasures. We are developing Product, API and model-level safeguards designed to reduce the efficacy of model outputs for illicit distillation, without degrading the experience for legitimate customers."
I often ask Claude to reason out loud, and this indicates that instead of explicitly blocking flagged requests the model output will be purposefully degraded.
Whatever you think of the ethics of doing this, it does hurt the reputation of the follower labs in my mind. If their capabilities can't exist without the work of the frontier labs, they're less equal competitors and more the guys trying to sell you a shoddy knockoff. Not that there's no use case for shoddy knockoffs.
It’s not that capabilities could not exist without the original work. It’s more that the shortest path between A and B isn’t repeating all of the same work.
Further, although media likes to depict Chinese labs as “just copying” I think there’s a ton of hubris involved. First of all, American labs are filled with Chinese who are trained at the very same schools as Chinese labs. Second, if you look at the contributions from Chinese labs many have pushed the state of the art.
Zooming out, data is kind of an arbitrary line to draw. Anthropic didn’t invent the neural network, back propagation, or the transformer. They didn’t invent all of the post training techniques they are using. They might even be using some pretrained open models during pre training and data prep. They got all of those for free because those things are shared openly.
With OAI and Gemini already having anti-distillation measures for quite a while now, I thought Anthropic was purposefully letting Chinese labs distill in hopes that it would improve their safety and alignment by default (at least closer to Claude's level).
Apparently not. (Or not anymore.)
It's not like they can actually prevent distillation anyways even by hiding the thinking output, since you can just turn extended thinking off and all current Claude models will switch to thinking in the open (non-reasoning output) instead whenever it encounters a hard agentic task. So all it takes for distillation to continue to happen is for some real users to sell a competing AI lab their real usage trajectory data which is entirely undetectable by definition, and many people would probably be glad to do it.
One consequence of creating a country of geniuses in a data center is that you now have a country of geniuses who can potentially help your competitors catch up on research, coding, and data labeling. It's a tough problem for the industry and, more importantly, for long-term safety.
We're obviously nowhere close now, but if we get to a world AI becomes powerful, and powerful AI can be used to create misaligned powerful AI, you may have to start regulating powerful AI like refined uranium processing tech, which is regulated more heavily than refined uranium itself.
Not too long ago Anthropic started hiding chain of thought in Claude Code. Returning the 'thinking' all at once for only a second or two.
There's a lot of good insight inside of there and you can immediately identify any misconceptions and stop an LLM from going in the wrong direction or doing damage.
A lot of their decisions as of late make sense now. The experience has suffered and some of the examples they cite as signs of distillation point to things only getting worse.
I think they are exposing how fragile and vulnerable in reality they are, and I wonder when it will happen that a group of highly motivated individuals will organize to create a truly community driven distilled models.
This is an exmaple of a potentially problematic prompt:
"You are an expert data analyst combining statistical rigor with deep domain knowledge. Your goal is to deliver data-driven insights — not summaries or visualizations — grounded in real data and supported by complete and transparent reasoning."
And they say:
"This includes detection of chain-of-thought elicitation used to construct reasoning training data."
...
"We are developing Product, API and model-level safeguards designed to reduce the efficacy of model outputs for illicit distillation, without degrading the experience for legitimate customers."
It's going to be very hard to generate outputs that people need but that also can't be used for distillation. For example, it's a good practice for many reasons including audibility to ask for the chain of thought. In fact, I'd argue it's essentially impossible to modify the outputs in a way that makes them less useful for distillation without degrading quality for legitimate users.
So then their only viable option is to try to identify the traffic. However, that is very hard because: "In one case, a single proxy network managed more than 20,000 fraudulent accounts simultaneously, mixing distillation traffic with unrelated customer requests to make detection harder."
First of all this is not technically distillation, it is more imitation learning.
Second, you could do something like asking Claude to create 1 million prompt, offensive response, non offensive response triplets. Then train a model with DPO to prefer the offensive responses.
cherryteastain|6 days ago
amunozo|5 days ago
direwolf20|6 days ago
WiSaGaN|6 days ago
janalsncm|6 days ago
k1musab1|6 days ago
I often ask Claude to reason out loud, and this indicates that instead of explicitly blocking flagged requests the model output will be purposefully degraded.
joshribakoff|6 days ago
atultw|6 days ago
resfirestar|6 days ago
janalsncm|6 days ago
Further, although media likes to depict Chinese labs as “just copying” I think there’s a ton of hubris involved. First of all, American labs are filled with Chinese who are trained at the very same schools as Chinese labs. Second, if you look at the contributions from Chinese labs many have pushed the state of the art.
Zooming out, data is kind of an arbitrary line to draw. Anthropic didn’t invent the neural network, back propagation, or the transformer. They didn’t invent all of the post training techniques they are using. They might even be using some pretrained open models during pre training and data prep. They got all of those for free because those things are shared openly.
2001zhaozhao|6 days ago
Apparently not. (Or not anymore.)
It's not like they can actually prevent distillation anyways even by hiding the thinking output, since you can just turn extended thinking off and all current Claude models will switch to thinking in the open (non-reasoning output) instead whenever it encounters a hard agentic task. So all it takes for distillation to continue to happen is for some real users to sell a competing AI lab their real usage trajectory data which is entirely undetectable by definition, and many people would probably be glad to do it.
tedsanders|6 days ago
We're obviously nowhere close now, but if we get to a world AI becomes powerful, and powerful AI can be used to create misaligned powerful AI, you may have to start regulating powerful AI like refined uranium processing tech, which is regulated more heavily than refined uranium itself.
xyzsparetimexyz|6 days ago
F7F7F7|5 days ago
There's a lot of good insight inside of there and you can immediately identify any misconceptions and stop an LLM from going in the wrong direction or doing damage.
A lot of their decisions as of late make sense now. The experience has suffered and some of the examples they cite as signs of distillation point to things only getting worse.
noravux|6 days ago
unknown|6 days ago
[deleted]
trexmux|6 days ago
titaniumrain|4 days ago
ReDeiPirati|5 days ago
SteveVeilStream|6 days ago
And they say: "This includes detection of chain-of-thought elicitation used to construct reasoning training data." ... "We are developing Product, API and model-level safeguards designed to reduce the efficacy of model outputs for illicit distillation, without degrading the experience for legitimate customers."
It's going to be very hard to generate outputs that people need but that also can't be used for distillation. For example, it's a good practice for many reasons including audibility to ask for the chain of thought. In fact, I'd argue it's essentially impossible to modify the outputs in a way that makes them less useful for distillation without degrading quality for legitimate users.
So then their only viable option is to try to identify the traffic. However, that is very hard because: "In one case, a single proxy network managed more than 20,000 fraudulent accounts simultaneously, mixing distillation traffic with unrelated customer requests to make detection harder."
nitros|6 days ago
nebezb|6 days ago
janalsncm|6 days ago
Second, you could do something like asking Claude to create 1 million prompt, offensive response, non offensive response triplets. Then train a model with DPO to prefer the offensive responses.
ncb9094|6 days ago
amai|5 days ago
titaniumrain|5 days ago
shablulman|6 days ago
[deleted]