[Manabu-eo]

How likely this problem is already on the training set by now?

[simonw]

If anyone trains a model on https://simonwillison.net/tags/pelican-riding-a-bicycle/ they're going to get some VERY weird looking pelicans.

[suddenlybananas]

Why would they train on that? Why not just hire someone to make a few examples.

[throwup238]

For every combination of animal and vehicle? Very unlikely.

The beauty of this benchmark is that it takes all of two seconds to come up with your own unique one. A seahorse on a unicycle. A platypus flying a glider. A man’o’war piloting a Portuguese man of war. Whatever you want.

[recursive]

No, not every combination. The question is about the specific combination of a pelican on a bicycle. It might be easy to come up with another test, but we're looking at the results from a particular one here.

[verdverm]

I've heard it posited that the reason the frontier companies are frontier is because they have custom data and evals. This is what I would do too

[zarzavat]

You can always ask for a tyrannosaurus driving a tank.