[queuebert]

You seem to be intuiting some kind of chi squared minimization. It is true that fewer free parameters constrain models, but there is nothing in nature that prefers simplicity. That is probably the most annoying thing to us physicists. Even thermodynamics is always shoving us toward disorder. Just look at plasma physics some time for deterministically intractable problems stemming from four little equations (one if you like tensors).

I think it's better to think of most real world models as being low dimensional-ish, where there is a decaying power law of eigenvalues, and most are quite small, though not zero. You can get quite far by looking at the largest modes and ignoring small ones, but you're not exact, so you're not seeing The Truth, or whatever. Forcing your self to use fewer parameters is a way of denoising, however, that is quite effective.