[Anon1096]

What you're missing is that for these massive systems there's never enough capacity. You can go look at datacenter buildouts YOY if you'd like. Any and all compute power that can be used is being used.

For individual services what that means is that for something like Google Search there will be dozens of projects in the hopper that aren't being worked on because there's just not enough hardware to supply the feature (for example something may have been tested already at small scale and found to be good SEO ranking wise but compute expensive). So a team that is able to save 1% CPU can directly repurpose that saved capacity and fund another project. There's whole systems in place for formally claiming CPU savings and clawing back those savings to fund new efforts.

[bongodongobob]

I've never in my life (25 years in the business) seen a system that was so utilized things needed to be cut. Every single cluster/DC I've worked with has been at 50-85% utilization tops. I mean, they might hit 100% during a report generation period or something, but the 95% avg I've seen has never exceeded maybe 80%.