theanets.layers.recurrent.SCRN¶
-
class
theanets.layers.recurrent.
SCRN
(rate='vector', s_0=None, context_size=None, **kwargs)[source]¶ Structurally Constrained Recurrent Network layer.
Notes
A Structurally Constrained Recurrent Network incorporates an explicitly slow-moving hidden context layer with a simple recurrent network.
The update equations in this layer are largely those given by [Mik15], pages 4 and 5, but this implementation adds a bias term for the output of the layer. The update equations are thus:
\[\begin{split}\begin{eqnarray} s_t &=& r \odot x_t W_{xs} + (1 - r) \odot s_{t-1} \\ h_t &=& \sigma(x_t W_{xh} + h_{t-1} W_{hh} + s_t W_{sh}) \\ o_t &=& g\left(h_t W_{ho} + s_t W_{so} + b\right). \\ \end{eqnarray}\end{split}\]Here, \(g(\cdot)\) is the activation function for the layer and \(\odot\) is elementwise multiplication. The rate values \(r\) are computed using \(r = \sigma(\hat{r})\) so that the rate values are limited to the open interval (0, 1). \(\sigma(\cdot)\) is the logistic sigmoid.
Parameters
w
— matrix connecting inputs to [hidden, state] units (this is a concatenation of parameters A and B in the paper)sh
— matrix connecting state to hiddens (P)hh
— matrix connecting hiddens to hiddens (R)ho
— matrix connecting hiddens to output (U)so
— matrix connecting state to output (V)b
— vector of output bias values (not in original paper)
Additionally, if
rate
is specified as'vector'
(the default), then we also have:r
— vector of learned rate values for the state units
Outputs
out
— the overall output of the layerhid
— the output from the layer’s hidden unitsstate
— the output from the layer’s state unitsrate
— the rate values of the state units
References
[Mik15] (1, 2) T. Mikolov, A. Joulin, S. Chopra, M. Mathieu, & M. Ranzato (ICLR 2015) “Learning Longer Memory in Recurrent Neural Networks.” http://arxiv.org/abs/1412.7753 -
__init__
(rate='vector', s_0=None, context_size=None, **kwargs)[source]¶ x.__init__(…) initializes x; see help(type(x)) for signature
Methods
__init__
([rate, s_0, context_size])x.__init__(…) initializes x; see help(type(x)) for signature add_bias
(name, size[, mean, std])Helper method to create a new bias vector. add_weights
(name, nin, nout[, mean, std, …])Helper method to create a new weight matrix. bind
(graph[, reset, initialize])Bind this layer into a computation graph. connect
(inputs)Create Theano variables representing the outputs of this layer. find
(key)Get a shared variable for a parameter by name. full_name
(name)Return a fully-scoped name for the given layer output. log
()Log some information about this layer. log_params
()Log information about this layer’s parameters. resolve_inputs
(layers)Resolve the names of inputs for this layer into shape tuples. resolve_outputs
()Resolve the names of outputs for this layer into shape tuples. setup
()Set up the parameters and initial values for this layer. to_spec
()Create a specification dictionary for this layer. transform
(inputs)Transform the inputs for this layer into an output for the layer. Attributes
input_name
Name of layer input (for layers with one input). input_shape
Shape of layer input (for layers with one input). input_size
Size of layer input (for layers with one input). output_name
Full name of the default output for this layer. output_shape
Shape of default output from this layer. output_size
Number of “neurons” in this layer’s default output. params
A list of all parameters in this layer. -
resolve_inputs
(layers)[source]¶ Resolve the names of inputs for this layer into shape tuples.
Parameters: - layers : list of
Layer
A list of the layers that are available for resolving inputs.
Raises: - theanets.util.ConfigurationError :
If an input cannot be resolved.
- layers : list of
-
to_spec
()[source]¶ Create a specification dictionary for this layer.
Returns: - spec : dict
A dictionary specifying the configuration of this layer.
-
transform
(inputs)[source]¶ Transform the inputs for this layer into an output for the layer.
Parameters: - inputs : dict of Theano expressions
Symbolic inputs to this layer, given as a dictionary mapping string names to Theano expressions. See
Layer.connect()
.
Returns: - output : Theano expression
The output for this layer is the same as the input.
- updates : list
An empty updates list.