theanets.layers.recurrent.MRNN

class theanets.layers.recurrent.MRNN(factors=None, **kwargs)[source]

A recurrent network layer with multiplicative dynamics.

Notes

The formulation of MRNN implemented here uses a factored dynamics matrix. To understand the motivation for a factored dynamics, imagine for a moment a vanilla recurrent layer with one binary input, whose hidden dynamics depend on the input, so that \(W_{hh}^0\) is used if the input is 0, and \(W_{hh}^1\) is used if the input is 1:

\[h_t = \sigma(h_{t-1} W_{hh}^{x_t} + x_t W_{xh} + b)\]

This generalizes to the idea that there might be an entire collection of \(W_{hh}^i\) matrices that govern the hidden dynamics of the network, one for each \(0 \le i < N\). But in the general case, it would be prohibitively expensive to store this weight tensor; in addition, there are probably many shared hidden dynamics that one might want to learn across all of these runtime “modes.”

The MRNN solves this problem by factoring the weight tensor idea into two 2–dimensional arrays. The hidden state is mapped to and from “factor space” by \(W_{hf}\) and \(W_{fh}\), respectively, and the latent factors are modulated by the input using \(W_{xf}\).

The overall hidden activation for the MRNN model, then, looks like:

\[h_t = \sigma((x_t W_{xf} \odot h_{t-1} W_{hf}) W_{fh} + x_t W_{xh} + b)\]

where \(odot\) represents the elementwise product of two vectors.

Parameters

  • b — vector of bias values for each hidden unit
  • xf — matrix connecting inputs to factors
  • xh — matrix connecting inputs to hiddens
  • hf — matrix connecting hiddens to factors
  • fh — matrix connecting factors to hiddens

Outputs

  • out — the post-activation state of the layer
  • pre — the pre-activation state of the layer
  • factors — the activations of the latent factors

References

[Sut11]I. Sutskever, J. Martens, & G. E. Hinton. (ICML 2011) “Generating text with recurrent neural networks.” http://www.icml-2011.org/papers/524_icmlpaper.pdf
__init__(factors=None, **kwargs)[source]

x.__init__(…) initializes x; see help(type(x)) for signature

Methods

__init__([factors]) x.__init__(…) initializes x; see help(type(x)) for signature
add_bias(name, size[, mean, std]) Helper method to create a new bias vector.
add_weights(name, nin, nout[, mean, std, …]) Helper method to create a new weight matrix.
bind(graph[, reset, initialize]) Bind this layer into a computation graph.
connect(inputs) Create Theano variables representing the outputs of this layer.
find(key) Get a shared variable for a parameter by name.
full_name(name) Return a fully-scoped name for the given layer output.
log() Log some information about this layer.
log_params() Log information about this layer’s parameters.
resolve_inputs(layers) Resolve the names of inputs for this layer into shape tuples.
resolve_outputs() Resolve the names of outputs for this layer into shape tuples.
setup() Set up the parameters and initial values for this layer.
to_spec() Create a specification dictionary for this layer.
transform(inputs) Transform the inputs for this layer into an output for the layer.

Attributes

input_name Name of layer input (for layers with one input).
input_shape Shape of layer input (for layers with one input).
input_size Size of layer input (for layers with one input).
output_name Full name of the default output for this layer.
output_shape Shape of default output from this layer.
output_size Number of “neurons” in this layer’s default output.
params A list of all parameters in this layer.
setup()[source]

Set up the parameters and initial values for this layer.

to_spec()[source]

Create a specification dictionary for this layer.

Returns:
spec : dict

A dictionary specifying the configuration of this layer.
transform(inputs)[source]

Transform the inputs for this layer into an output for the layer.

Parameters:
inputs : dict of Theano expressions

Symbolic inputs to this layer, given as a dictionary mapping string names to Theano expressions. See Layer.connect().
Returns:
output : Theano expression

The output for this layer is the same as the input.

updates : list

An empty updates list.