[slightwinder]

In this specific case, it's because of creative conflicts. Volker Wertich, the creator of Settlers 1&3, was also supposed to be creator for the newest Game, Settler: New Allies (part 8(?) of the mainline). He left after Ubisoft was not convinced from his Vision, considered it too complex, ambiguous. They release the reworked final version some years later, and it flopped hard. He then went on to create Pagonia after it came to light how much the new part sucks.

I don't think we know in detail what his original vision was, we can only assume if Pagonia is aiming to manifest it. And there is the theory that Ubisoft didn't like to have two similar games in their catalogue. They already have the Anno-Franchise, which is very similar to settlers, and had a promising new game released around the time Wertich left Ubisoft.

So the answer is probably a mix between internal politics of big companies, risk-avoidance and creative minds being too creative for the average manager. The more money you push, the more the system strifes to controlled outcomes. And Ubisoft was pushing very big around that time, to the point that they killed themselves, it seems.