theanets.regularizers.HiddenL1

class theanets.regularizers.HiddenL1(pattern=None, weight=0.0)

Penalize the activation of hidden layers under an L1 norm.

Notes

This regularizer implements the loss() method to add the following term to the network’s loss function:

\[\frac{1}{|\Omega|} \sum_{i \in \Omega} \|Z_i\|_1\]

where \(\Omega\) is a set of “matching” graph output indices, and the L1 norm :math`|cdot|_1` is the sum of the absolute values of the elements in the corresponding array.

This regularizer tends to encourage the hidden unit activations in a model to be zero. Nonzero activations are used only when they are able to reduce the other components of the loss (e.g., the squared reconstruction error).

This regularizer acts indirectly to force a model to cover the space of its input dataset using as few features as possible; this pressure often causes features to be duplicated with slight variations to “tile” the input space in a very different way than a non-regularized model.

References

[Ng11]A. Ng. (2011). “Sparse Autoencoder.” Stanford CS294A Lecture Notes http://web.stanford.edu/class/cs294a/sae/sparseAutoencoderNotes.pdf

Examples

This regularizer can be specified at training or test time by providing the hidden_l1 or hidden_sparsity keyword arguments:

>>> net = theanets.Regression(...)

To use this regularizer at training time:

>>> net.train(..., hidden_sparsity=0.1)

By default all hidden layer outputs are penalized. To include only some graph outputs:

>>> net.train(..., hidden_sparsity=dict(weight=0.1, pattern='hid3:out'))

To use this regularizer when running the model forward to generate a prediction:

>>> net.predict(..., hidden_sparsity=0.1)

The value associated with the keyword argument can be a scalar—in which case it provides the weight for the regularizer—or a dictionary, in which case it will be passed as keyword arguments directly to the constructor.

__init__(pattern=None, weight=0.0)

Methods

__init__([pattern, weight])
log() Log some diagnostic info about this regularizer.
loss(layers, outputs)
modify_graph(outputs) Modify the outputs of a particular layer in the computation graph.