These MPI-based scientific computing applications make up a bulk of the compute hours on hpc clusters, but there is a crazy long tail of scientists who have workloads that can’t (or shouldn’t) run on their personal computers. The other option is HPC. This sucks for a ton of reasons, but I think the biggest one is that it’s more or less impossible to set up a persistent service of any kind. So no databases; if you want spark, be ready to spin it up from nothing every day (also no HDFS unless you spin that up in your SLURM job too). This makes getting work done harder but it also means that it makes integrating existing work so much harder because everyone’s workflow involves reinventing everything, and everyone does it in subtly incompatible ways; there are no natural (common) abstraction layers because there are no services.
No comments yet.