[haddr]

I think this is the case: when you run your pipelines at scale you want to standardize and simplify some repeatable aspects to lower the cost of managing them. You may also want to be orthogonal to orchestrator engines (or triggering engines) and avoid getting too opinionated and inflexible in the future. So this framework is exploring some sweet spot between raw spark pipelines and low code etl engines.

[steveBK123]

yeah though a lot of these fall for a variant of the "universal standard" conceit joked about in xkcd. All these low-code solutions suck, so we'll build our own in-house that surely won't have the same pitfalls..