Getting Started

The theanets package provides tools for defining and optimizing several common types of neural network models. It uses Python for rapid development, and under the hood Theano provides graph optimization and fast computations on the GPU.

This page provides a quick overview of the theanets package. It is aimed at getting you up and running with a few examples. Once you understand the basic workflow, you will be able to extend the examples to your own datasets and modeling problems. After you’ve finished reading through this document, have a look at Creating a Network, Training a Model, and Using a Model for more detailed documentation.

Installation

If you haven’t already, the first thing you should do is download and install theanets. The easiest way to do this is by using pip:

pip install theanets

This command will automatically install all of the dependencies for theanets, including numpy and theano.

If you’re feeling adventurous, you can also check out the latest version of theanets and run the code from your local copy:

git clone https://github.com/lmjohns3/theanets
cd theanets
python setup.py develop

This can be risky, however, since theanets is in active development—the API might change in the development branch from time to time.

To work through the documentation you should also install a couple of supporting packages:

pip install skdata
pip install matplotlib

These will help you obtain the example dataset described below, and also help in making plots of various things.

Package Overview

At a high level, the theanets package is a tool for (a) defining and (b) optimizing cost functions over a set of data. The workflow in theanets typically involves three basic steps:

  1. First, you define the structure of the model that you’ll use for your task. For instance, if you’re trying to classify MNIST digits, you’ll want something that takes in pixels and outputs digit classes (a “classifier”). If you’re trying to model the unlabeled digit images, you might want to use a model that uses the same data for input and output (an “autoencoder”).

  2. Second, you train or adjust the parameters in your model so that it has a low cost or performs well with respect to some task. For classification, you might want to adjust your model parameters to minimize the negative log-likelihood of the correct image class given the pixels, and for autoencoders you might want to minimize the reconstruction error.

    A significant component of this step usually involves preparing the data that you’ll use to train your model.

  3. Finally, you use the trained model in some way, probably by predicting results on a test dataset, visualizing the learned features, and so on.

The theanets package provides a helper class, Experiment, that performs these tasks with relatively low effort on your part. You typically define a model in theanets by creating an experiment with a number of model hyperparameters that define the specific behavior of your model, and then train your model using another set of optimization hyperparameters that define the behavior of the optimization algorithm.

The skeleton of your code will usually look something like this:

import matplotlib.pyplot as plt
import skdata.mnist
import theanets

# create an experiment to define a model.
exp = theanets.Experiment(
    Model,
    hyperparam1=value1,
    hyperparam2=value2,
    # ...
)

# train the model.
exp.train(
    training_data,
    validation_data,
    algorithm='foo',
    # ...
)

# use the trained model.
model = exp.network
model.predict(test_data)

This quickstart document shows how to implement these stages by following a couple of examples.

Classifying MNIST Digits

A standard benchmark for neural network classification is the MNIST digits dataset, a set of 70,000 28×28 images of hand-written digits. Each MNIST digit is labeled with the correct digit class (0, 1, ... 9). This example shows how to use theanets to create and train a model that can perform this task.

_images/mnist-digits-small.png

Networks for classification map a layer of continuous-valued inputs, through one or more hidden layers, to an output layer that is activated through the softmax function. The softmax generates output that can be treated as a categorical distribution over the digit labels given the input image.

Defining the model

Now that you know which model to use for this task, you’ll need to define some hyperparameters that determine the structure of your network. The most important of these is choosing a set of layer sizes that you want in your model.

The first (“input”) and last (“output”) layers in your network must match the size of the data you’ll be providing. For an MNIST classification task, this means your network must have 784 inputs (one for each image pixel) and 10 outputs (one for each class).

Between the input and output layers, on the other hand, can be any number of so-called “hidden” layers, in almost any configuration. Models with more than about two hidden layers are commonly called “deep” models and have been quite popular recently due to their success on a variety of difficult machine learning problems. For now, though, to keep things simple, let’s start out with a model that just has one hidden layer with 100 units.

Once you’ve chosen the layers you want in your model, the easiest way to use theanets is to create an Experiment to construct your model:

exp = theanets.Experiment(
    theanets.Classifier,
    layers=(784, 100, (10, 'softmax')))

This is all that’s required to get started. There are many different hyperparameters that can also be useful when constructing a model; see Creating a Network for more information. Particularly useful to know will be the different ways of creating layers; see Specifying Layers for details.

Preparing the data

In theanets, the parameters of a model are initialized randomly. To improve the model’s performance on the task, you’ll need to train the model parameters. This training process requires a dataset to compute gradient and loss function values.

In the case of the MNIST digits, our classifier model will consume a dataset consisting of two parts—”samples” (image pixels) and corresponding “labels” (integer class values). Each of these parts is provided as a numpy array: the samples are a two-dimensional array, with vectorized MNIST pixels arranged along the first axis and pixel data arranged along the second axis; the labels are a one-dimensional array, with one integer value per MNIST image.

For easy access to the MNIST digits dataset, we’ll use the skdata package and write a little bit of glue code to get the data into the desired format:

def load_mnist():
    mnist = skdata.mnist.dataset.MNIST()
    mnist.meta  # trigger download if needed.
    def arr(n, dtype):
        # convert an array to the proper shape and dtype
        arr = mnist.arrays[n]
        return arr.reshape((len(arr), -1)).astype(dtype)
    train_images = arr('train_images', 'f') / 255.
    train_labels = arr('train_labels', np.uint8)
    test_images = arr('test_images', 'f') / 255.
    test_labels = arr('test_labels', np.uint8)
    return ((train_images[:50000], train_labels[:50000, 0]),
            (train_images[50000:], train_labels[50000:, 0]),
            (test_images, test_labels[:, 0]))

Here we’ve rescaled the image data so that each pixel lies in the interval [0, 1] instead of the default [0, 255]. (In general, it’s a good idea to standardize the data for your problem so that each dimension has approximately the same scale.) We’ve also reshaped the data as described above.

Note

Because theanets uses Theano for its computations, most datasets need to be cast to a value that is compatible with your setting for Theano’s “floatX” configuration parameter. Unless you have a really expensive GPU, this is likely to mean that you need to use 32-bit floats.

The load function returns a training split (the first 50000 examples), a validation split (the remainder of the training data from skdata, containing 10000 examples), and a test split (the test split from skdata, containing 10000 examples). The training dataset is used to compute parameter updates, and the validation dataset is used to determine when the model has stopped improving during training.

There are other ways to provide data to your model during training; for a more complete description, see Providing Data.

Training the model

Now that you have a model and some data, you’re ready to train the model so that it performs the classification task as well as possible. The Experiment class handles the general case of training with fairly little work.

The main decision to make during training is to choose the training algorithm to use, along with values for any associated hyperparameters. This is most naturally accomplished using the Experiment.train() method:

train, valid, test = load_mnist()

exp.train(train,
          valid,
          algorithm='nag',
          learning_rate=1e-3,
          momentum=0.9)

The first positional argument to this method is the training dataset, and the second (if provided) is a validation dataset. (These positional arguments can also be passed to Experiment.train() using the train_set and valid_set keywords, respectively.) If a validation dataset is not provided, the training dataset will be used for validation.

The algorithm keyword argument specifies an algorithm to use for training. If you do not provide a value for this argument, RmsProp is used as the default training algorithm. Any subsequent keyword arguments will be passed to the training algorithm; these arguments typically specify hyperparameters of the algorithm like the learning rate and so forth.

The available training methods are described in Available Trainers; here we’ve specified Nesterov’s Accelerated Gradient, a type of stochastic gradient descent with momentum.

Visualizing features

Once you’ve trained a classification model for MNIST digits, it can be informative to visually inspect the features that the model has learned. Because the model was trained using the MNIST digits, you can reshape the learned features and visualize them as though they were 28×28 images:

img = np.zeros((28 * 10, 28 * 10), dtype='f')
for i, pix in enumerate(exp.network.find('hid1', 'w').get_value().T):
    r, c = divmod(i, 10)
    img[r * 28:(r+1) * 28, c * 28:(c+1) * 28] = pix.reshape((28, 28))
plt.imshow(img, cmap=plt.cm.gray)
plt.show()

In this example, the weights in layer 1 connect the inputs to the first hidden layer; these weights have one column of 784 values for each hidden node in the network, so we can iterate over the transpose and put each column—properly reshaped—into a giant image.

The trained model can also be used to predict the class for a new MNIST digit:

predicted_class = exp.network.predict(new_digit)

For more information on the things you can do with a model, see Using a Model.

Remembering Network Inputs

Recurrent neural networks are a family of network models whose computation graph contains a cycle—that is, there are some layers in a recurrent network whose outputs at a certain time step depend not only on the inputs at that time step, but also on the state of the network at some previous time step as well.

Recurrent networks, while often quite tricky to train, can be used to solve difficult modeling tasks. Thanks to recent advances in optimization algorithms, recurrent networks are enjoying a resurgence in popularity and have been shown to be quite effective at a number of different temporal modeling tasks.

In this section we consider a classic task for a recurrent network: remembering data from past inputs. In this task, a network model receives one input value at each time step. The network is to remember the first \(k\) values, then wait for \(t\) time steps, and then reproduce the first \(k\) values that it saw. Effectively the model must ignore the inputs after time step \(k\) and start producing the desired output at time step \(k + t\).

Defining the model

We’ll set up a recurrent model by creating an Experiment with the appropriate model class and layers:

exp = theanets.Experiment(
    theanets.recurrent.Regressor,
    layers=(1, ('lstm', 10), 1))

Here we’ve specified that we’re using a recurrent regression model. Our network has three layers: the first just has one input unit, the next is a Long Short-Term Memory (LSTM) recurrent layer with ten units, and the output is a linear layer with just one output unit. This is just one way of specifying layers in a network; for more details see Specifying Layers.

Training the model

The most difficult part of training this model is creating the required data. To compute the loss for a recurrent regression model in theanets, we need to provide two arrays of data—one input array, and one target output array. Each of these arrays must have three dimensions: the first is time, the second is the batch size, and the third is the number of inputs/outputs in the dataset.

For the memory task, we can easily create random arrays with the appropriate shape. We just need to make sure that the last \(k\) time steps of the output are set to the first \(k\) time steps of the input:

T = 20
K = 3
BATCH_SIZE = 32

def generate():
    s, t = np.random.randn(2, T, BATCH_SIZE, 1).astype('f')
    s[:K] = t[-K:] = np.random.randn(K, BATCH_SIZE, 1)
    return [s, t]

In theanets, data can be provided to a trainer in several ways; here we’ve used a callable that generates batches of data for us. See training-using-callables for more information.

Having set up a way to create training data, we just need to pass this along to our training algorithm:

exp.train(generate, algorithm='rmsprop')

This process will adjust the weights in the model so that the outputs of the model, given the inputs, will be closer and closer to the targets that we provide.

More Information

This concludes the quick start guide! Please read more information about creating models in theanets in Creating a Network, Training a Model, and Using a Model. Once you’re familiar with the basic concepts, the Reference section might also be useful.

The source code for theanets lives at http://github.com/lmjohns3/theanets. Please fork, explore, and send pull requests!

Finally, there is also a mailing list for project discussion and announcements. Subscribe online at https://groups.google.com/forum/#!forum/theanets.