[jeffreygoesto]

We are in such a situation, not for web but embedded and tools. We found getting the split between build system and CI right is one of the most important things. Don't model as CI dependencies what in essence is a build dependency. Then local builds can predict CI outcome well.

Went from Jenkins/Conan/CMake to GH/Bazel with tons of stuff learned, the most important learning being "declarative eats imperative for breakfast" and that hermetic builds enable remote caching. Build duration on average went down from hours to minutes (because our PRs try to be small incremental updates) for everybody including CI.