[112233]

But how about not scheduling swapped out processes if there currently is no free ram for their current RSS? of course kernel cannot know that a new process will balloon to eat all RAM, but once it has done so, is there a way to let it run to completion without being swapped out to "improve responsivity"?

[man8alexd]

There is no actual swapping in the modern kernels. Nowadays, it is paging, when the kernel pages out individual unused memory pages, not entire processes, so it keeps all non-blocked processes running, but only necessary memory pages in the memory.