theanets.activations.LGrelu¶
-
class
theanets.activations.
LGrelu
(*args, **kwargs)¶ Rectified linear activation with learnable leak rate and gain.
This activation is characterized by two linear pieces joined at the origin. For negative inputs, the unit response is a linear function of the input with slope \(r\) (the “leak rate”). For positive inputs, the unit response is a different linear function of the input with slope \(g\) (the “gain”):
\[\begin{split}f(x) = \left\{ \begin{eqnarray*} rx &\qquad& \mbox{if } x < 0 \\ gx &\qquad& \mbox{otherwise} \end{eqnarray*} \right.\end{split}\]This activation allocates a separate leak and gain rate for each unit in its layer.
-
__init__
(*args, **kwargs)¶
Methods
__init__
(*args, **kwargs)-