top | item 34680761

(no title)

convexfunction | 3 years ago

It's not quite the same, and might not be possible to make rigorous enough that it really proves anything, but something sort of similar would actually be practical to at least attempt in many cases. Stable Diffusion checkpoints of the same major version, along with other families of model weights, have the IMO fascinating property that you can do element-wise arithmetic with them, and the resulting model will actually sort of function like you'd naively expect. Recent paper on the topic (in LLMs, not diffusion models) here: https://arxiv.org/abs/2212.04089

So, if you take a Stable Diffusion checkpoint (call it "A") which is only lightly trained on some subset of an artist's work, then fine tune it on the full corpus of that artist's work to a point where it's still coherent/"good" and just shy of actually memorizing the fine tuning data (call the resulting model "B"), then define model "C" as 2A-B (i.e. A + (A-B), where A-B is the artist's task vector multiplied by -1), can you still produce qualitatively similar images with model C? Whether with the exact same prompt, or the same prompt with "in the style of Kinkade" removed (which doesn't mean as much if Kinkade's task vector was subtracted), or with any prompt whatsoever?

Lots of issues with this as laid out -- it's definitely not quite the same as "forgetting" Kinkade from the training data, and "any prompt whatsoever" introduces tons of leeway, and most good AI-assisted art is not just an unmodified single text-to-image output anyway -- but it might be a promising direction to explore.

(Strongly disagree with the "copyright laundry" characterization, by the way.)

discuss

order

No comments yet.