[nooorofe]

> if you want to convert some scheduled pipeline to some event-driven architecture

Airflow has sensors and triggers. https://airflow.apache.org/docs/apache-airflow/stable/author...

But in the core it is built around data pipeline concept, event driven pipeline will much more fragile. Airflow intentionally doesn't manage business logic, it works with "tasks".

[annexrichmond]

Yes, but that means you are forced to build EDA on top of Airflow, which may not be ideal for many cases. You are stuck managing your pools/workers within Airflow's paradigm, which means all workload must (a) be written in Python and (b) have Airflow installed on the venv (very heavy pkg) and (c) be k8s pod or Celery (unless you write your own).

[nyrikki]

Only because you have chosen to introduce configuration and maintenance complexity by using airflow as enterprise wide middleware.

In a modern even based SOA, products like airflow are a sometimes food while pub/sub is the default.

Perhaps a search for images of the zachman framework would help conceptualize how you are tightly coupling to the implementation.

But also research SOA 2.0, or event based SOA, the Enterprise Service Bus concept of the original SOA is as dead as COBRA.

ETA: the minimal package load for airflow isn't bad, are you installing all of the plugins and their dependencies?