> Airflow and Dagster, along with similar tools like Flyte, Luigi, Argo Workflows, and Prefect, are bucketed under a set of systems you could utilize for the orchestration of jobs.
Quite the list of tools, nice. Interesting space to me.
I'm not sure we can label one solution to be "better than" the other just because it "is leveraging containerization to entirely solve..." anything. Reasons:
1. Containerization addresses the monolith pattern issues yes, but introduces other challenges (data persistency, image management, immutable infra, etc)
2. It takes much more than just containers and executors to design a reliable platform for ML/Data.
I think what can set projects apart is how/if they make use of Software Engineering patterns for what matters to ML as a workload. An orchestrator that could leverage a control loop (reconciler) pattern, similar to what K8s does for containers, would provide the eventual consistency and reliability required by most ML workloads.
rektide|2 years ago
Quite the list of tools, nice. Interesting space to me.
davidmirror|2 years ago
1. Containerization addresses the monolith pattern issues yes, but introduces other challenges (data persistency, image management, immutable infra, etc) 2. It takes much more than just containers and executors to design a reliable platform for ML/Data.
I think what can set projects apart is how/if they make use of Software Engineering patterns for what matters to ML as a workload. An orchestrator that could leverage a control loop (reconciler) pattern, similar to what K8s does for containers, would provide the eventual consistency and reliability required by most ML workloads.