[nmaley]

The NFL theorem means nothing if all the learning tasks have a common underlying structure. In the real world, they do. The laws of physics and chemistry create emergent causal relationships. Any SSL learning algorithm that learns to exploit causal relationships will consistently perform well over a variety of real world tasks.

[cs702]

Yes, I agree: In practice, all learning tasks have a common underlying structure because our physical reality can behave only in certain ways, determined by the symmetries in our universe. However, as far as I know, no one has formally specified or proved this commonality in learning tasks.

The most persuasive argument I've seen for this view -- our view -- is laid out in this paper by Lin, Tegmark, and Rolnick, written between 2016 and 2017 (although it feels like it was written a century ago!): https://arxiv.org/abs/1608.08225

[medo-bear]

you say this as if it is trivial. for example, optimization of the lagrangian function is a recurring theme in physics and is pretty much a model of physical causation. once you have the lagrangian you pretty much have the domain knowledge. if you can come up with an ssl that discovers relevant lagrangians from observing the real world, that would be HUGE. however nfl might still stand in your way ;)