Dolphin Scheduler

[jackneary]

I try DolphinScheduler online demo: http://106.75.43.194:8888/

easy to use.

[jackneary]

mail to [email protected], and tell him you want to try online demo.

[Xiali]

DolphinScheduler ranks among top 10 most valuable projects in OSChina GVP(Gitee Most Valuable Project)

[Ozzie_osman]

How does this compare to something like Airflow or Luigi?

[luckypeter]

may be you can refer their proposal. https://cwiki.apache.org/confluence/display/INCUBATOR/Dolphi...

[foo_barington]

This may be of interest:

https://github.com/mikub/titanoboa

[newcrobuzon]

Thanks for linking! There seem to be similarities, but looking (very) briefly at the DolphinScheduler these are the potential differences:

- number of contributors :D

- titanoboa can process even a potentially cyclic graph

- in titanoboa you can write step functions directly in high level programming languages such as clojure and java (so not just bash or python) and deploy them directly during runtime

- the clustering setup in titanoboa is master-less

- titanoboa does not have such direct integration with Spark as it employs some map-reduce patterns internally

But all-in-all I have to say that DolphinScheduler seems quite nice! Also would have to compcomplement it on the nice documentation (again, just briefly skimming through it).

(edit: formatting)

[eitland]

That looks interesting but remind me more about Apache Camel.

[rb808]

Also interesting that its one of the first open source applications from China I've seen.

[y4mi]

I've actually stopped paying attention to https://github.com/vitalets/github-trending-repos because there are so many Chinese repositories each week.

It's just rare to get them on HN, because it's a nightmare to go through their docs and they're usually not even attempting to write their code in English. Basically unusable for all intents and purposes, even if it were quality software.

[rb808]

This looks great, I've always wanted something like this. I've always had autosys or controlm at work and they both suck.

I'd just prefer if it had been around longer. Any other open source alternatives out there? I only know of Airflow, k8s Cronjobs.

[massive]

There you go https://github.com/meirwah/awesome-workflow-engines

[hartzell]

> Any other open source alternatives out there?

Here's a big handful of them: https://github.com/pditommaso/awesome-pipeline

[daveFNbuck]

There's also Luigi. https://github.com/spotify/luigi

[elcritch]

I was just reading up on Broadway, written in Elixir, (https://hexdocs.pm/broadway/Broadway.html) that provides the fundamentals of batching/job control. It’s by the creator of Elixir and is based on 7 years of libraries in the area so the fundamentals are pretty well honed.

[monstrado]

I've had alot of success using Apache NiFi as a distributed scheduler / general purpose workflow tool.

[tbrock]

The only other one I can think of off the top of my head is dagster: https://github.com/dagster-io/dagster

It’s made by Nick Schrock of graphql fame, among others. I’m sure there are 100s of these projects though.

[pgoggijr]

This looks almost exactly like Airflow - I wonder what Apache’s plan is for both of these to coexist.