[LPisGood]

One thing I wonder is why design of experiments (DOE) methodology is so seldom used for these things.

Statisticians and operations researchers have spent a hundred years deciding how to do as few experiments as possible to tweak parameters in the ways that give the highest impact with statistical basis that the selections are good.

In the language of information and decision trees, these experiments are trying to in some sense “branch” on the entropy minimizing variables.

[agalunar]

SPRT is used religiously in engine development today. There is enormous incentive to test efficiently.

https://github.com/official-stockfish/fishtest/wiki/Fishtest...

[mpolson64]

DOE is still very useful in many contexts, but when it's possible do use a sequential design these adaptive techniques really start to pull away in terms of optimization quality.

There's simply a lot of sample efficiency to gain by adapting the experiment to incoming data in a regime where one can repeatedly design n candidates, observe their effects, and repeat m times compared to a setting where one must design a fixed experiment with n*m samples.