[jplrssn]

I also wouldn't be surprised if labs were starting to mix in a few pelican SVGs into their training data.

[diggan]

Even "accidentally" it makes sense that "SVGs of pelicans riding bikes" are now included into datasets used for training as it has spread as a wildfire on the internet, making it less useful as a simple benchmark.

This is why I keep all my benchmarks private and don't share anything about them publicly, as soon as you write about them anywhere publicly they'll stop being useful in some months.

[toyg]

> This is why I keep all my benchmarks private

This is also why, if I were an artist or anyone commercially relying on creative output of any kind, I wouldn't be posting anything on the internet anymore, ever. The minute you make anything public, the engines will clone it to death and turn it into a commodity.

[simonw]

I'll believe they are doing that when one of the models draws me an SVG that actually looks like a pelican.

[__mharrison__]

Someone needs to craft a beautifully bike donned by a pelican, throw in some seo, and see how long it takes a model to replicate it.

Simon probably wouldn't be happy about killing his multi-year evaluation metric though...

[quantumHazer]

SVG benchmarking is a thing since GPT-4, so probably all major labs are overfitting on some dataset ov svg images for sure