[tweetle_beetle]

I think you might be looking at the film through rose tinted glasses without the broader context. Kubrick's films had been nominated for 9 Academy Awards and won 1 (he was personally nominated 3 times) by the time Barry Lyndon started filming. (He had also directed a certain Spartacus.)

Warner Brothers were keen to bankroll whatever he wanted to do, even tolerating moving the country of production due to the Troubles.

He was given some artistic freedom due to previous commercial success - ie. a "data- and money-driven approach". He also really wanted to be making a Napoleon biopic, but financing was pulled when a similar film failed at the box office, so he didn't get it all his own way.

Barry Lyndon was only a modest commercial succes. So much so that Warner Brothers hooked him up with a much safer bet for them for their next venture. He was given unfinished manuscript of The Shining, from the wildy popular best seller King for his next project, which was also simpler to produce ie. "relentless pursuit of revenue".

TLDR Making films is expensive and needs to be a commercial activity, but every now and then there's a fortunate crossover of quality and funding. This still happens but you need to look out for it.