Slightly faster softplus

Sleort · web-flow · commit 7eea8b8a7618 · 2022-11-18T13:27:30.000+01:00
A few percent faster `softplus` using `relu(x)` instead of `max(x, 0)`. Ref. #347.
diff --git a/src/activations.jl b/src/activations.jl
@@ -615,7 +615,7 @@ julia> softplus(16f0)
 16.0f0
 ```
 """
-softplus(x) = log1p(exp(-abs(x))) + max(x, 0)
+softplus(x) = log1p(exp(-abs(x))) + relu(x)
 
 """
     logcosh(x)
