[joker3]

Concentration of measure. If you have a quantity that depends on a large number of random variables but not too strongly on any small subset of them, it tends to behave like a constant. That's the intuition behind the law of large numbers, the central limit theorem, a bunch of concentration inequalities, and model averaging.

[marcosdumay]

And, of course, there is nothing that says this would work for intermediate layers, since the sample dimensions may get there from any input.

What works is averaging similar networks and averaging your networks a lot of times.