[Hawkenfall]

A more in-depth paper about this found the Swish activation often outperformed other functions: https://arxiv.org/abs/1710.05941

[osipov]

Most of the recent research is moving to GELU (Gaussian Error Linear Units) activation functions: https://arxiv.org/pdf/1606.08415.pdf

[excessive]

That's interesting. I didn't read the paper closely, but skipping to the pictures, it looks like ReLU, but smoothed out so the derivative is continuous. Intuitively, that seems useful.

[rickdeveloper]

I wasn’t aware of that one. Definitely interesting, thanks for sharing!