theanets.losses.KullbackLeiblerDivergence

class theanets.losses.KullbackLeiblerDivergence(target, weight=1.0, weighted=False, output_name='out')

The KL divergence loss is computed over probability distributions.

Notes

The KL divergence loss is intended to optimize models that generate probability distributions. If the outputs \(x_i\) of a model represent a normalized probability distribution (over the output variables), and the targets \(t_i\) represent a normalized target distribution (over the output variables), then the KL divergence is given by:

\[\mathcal{L}(x, t) = \frac{1}{d} \sum_{i=1}^d t_i \log \frac{t_i}{x_i}\]

Here the KL divergence is computed as a mean value over the output variables in the model.

__init__(target, weight=1.0, weighted=False, output_name='out')

Methods

__init__(target[, weight, weighted, output_name])
log() Log some diagnostic info about this loss.

Attributes

variables A list of Theano variables used in this loss.