[jmcnulty]

Depends what you want to do. If you're happy for changes to dribble in over time then size your puppetmaster pool to how many hosts you want to be able to run puppet simultaneously and stagger client execution to avoid a stampeding herd. Accept that sometimes there will be individual failures due to load and clients will just have to wait till the next time.

Alternatively, in real life, many teams have Change Management to consider and Maintenance windows. If there's a need to update thousands of systems on a saturday morning then expect teams to start puppet runs manually. You'd better have a seriously big pool of puppetmasters ready and waiting to manage the load, and don't forget Puppet DB, that has to be scaled up too to avoid lock ups. Even then, if teams start too many puppet runs at once, you'll get flattened.

We ended up scrapping all the puppetmasters in individual DCs and consolidating them in an AWS EC2 Autoscaling group. The number of puppetmasters started at 70 and just went up. That came with problems of its own. e.g. ensuring that all puppetmasters share the same copy of role versions at the same time. Being able to spin up new puppetmasters fast enough to meet spikes in demand. Various other corner case tuning issues.

It's taken a dedicated team years to get to grips with puppet, tame it and master it. Very glad I'm not involved in that any more.