top | item 41123400

(no title)

fs111 | 1 year ago

Well spark was really a showcase Project for mesos when it was created. Now everyone knows a lot more

discuss

order

bigcat12345678|1 year ago

Spark was never a showcase for mesos

Mesos was a derivative idea from some sporadic idea from 2 level scheduling from inside Google based on mapreduce

Mesos was wrong from day one, they thought they have the right idea, but was really caught up by a Small group of Google engineers who happened to be from academia. These engineers were very good at having theoretically nice idea

In the time mesos was invented, Google had its own mesos, which is a similar project by learning wrong lesson from Borg. That thing is Omega.

Eventually everyone decided that Borg was right all along, thus the kubernetes

fs111|1 year ago

I have no idea what any of the google tech has to do with anything here.

Quoting from the original spark paper:

> Spark is built on top of Mesos [16, 15], a “cluster operat- > ing system” that lets multiple parallel applications share > a cluster in a fine-grained manner and provides an API > for applications to launch tasks on a cluster

https://people.csail.mit.edu/matei/papers/2010/hotcloud_spar...

Note how Matei Zaharia - the inventor of spark - is also on the mesos paper:

https://people.eecs.berkeley.edu/~alig/papers/mesos.pdf