RNNs

Use of RNNs

After being exposed to CNNs in the vision module, we will start working with recurrent neural networks, RNNs. Why do we need a different type of NN? Different NNs are good at learning different types of problems.

What if we wanted our NN to learn how to distinguish between a series of $0$ ’s and $1$ ’s that has an odd number of $1$ ’s from one that has an even number? We can try it with a dense neural network, but it will not be able to learn to classify the sequences. We can try tweaking the NN and how we train it, but much progress is not possible because the structure of the simple dense NN is not conducive for learning this classification.

Think about it this way: what if a human was given such a sequence and asked whether it was even or odd? Some would count the number of $1$ ’s. If the number is even, then the sequence is even. Others would likely keep a toggle switch while parsing through the sequence: one that flips each time a $1$ is encountered. The problem with the simple dense network is that it can’t remember. There is no storing of information about the input itself, so it can’t store the number of $1$ ’s or whether a sequence was even or odd until a given point. That’s where RNNs come in. RNNs are good for dealing with sequences because they have a memory system, which is referred to as the hidden state.

Before we can train a RNN to classify these sequences as even or odd, let’s break down the problem into a simpler subproblem that can be solved with a dense NN: $x o r$ , which stands for “exclusive or”. $x o r$ , denoted ^, is a logical operator, just like $a n d$ , denoted &&, and $o r$ , denoted ||. $a$ $x o r$ $b$ will be true, or $1$ , if exactly one of $a$ or $b$ is true/ $1$ .

xor problem

When the $a$ $x o r$ $b$ is plotted, it becomes clear that $x o r$ is not linearly separable. This means that there is not one line that we can draw on the graph that will separate the trues from the falses. However, we can take a look at $n a n d$ , “not and”, and $o r$ , which are linearly separable. $n a n d$ is again a logical operator. It can be thought of as being composed of two other logical operators: $n o t$ , denoted $!$ , and &&. $a$ $n a n d$ $b$ is equivalent to $!$ ( $a$ && $b$ ).

xor in terms of nand and or

When we take a closer look, we find that the values bound by $n a n d$ and $o r$ can give us those wanted from $x o r$ .

xor in terms of nand and or

$x o r$ , when expressed in terms of $n a n d$ and $o r$ , is linearly separable:

xor in terms of nand and or

In terms of a dense neural network, this means that if we have another step, which allows for interaction between the $n a n d$ and $o r$ , the $x o r$ problem can be learned. This corresponds to having two weight terms, which results in one for computing the $n a n d$ and one for computing the $o r$ . The third and final set of weights will combine the $n a n d$ and $o r$ by &&’ing them to produce $x o r$ . Think about it this way: a dense layer is linear because the matrix of its weights is a linear function, like all matrices. The non-linear activation function only weights the underlying linearity. This is why two steps (computing $n a n d$ and $o r$ , then combining them) are required.

explaining xor problem

Now that we have the $x o r$ problem solved, we can apply it to the sequence classifying. The toggle method from earlier is where $x o r$ comes in. A sequence is even, represented by $1$ , so far if the current digit is $1$ and the the sequence until now is odd. It is also even if the current digit is $0$ and the sequence until now is even. This can be represented as: evenness of sequence so far = current digit $x o r$ evenness of sequence until now.

Implementing a Simple RNN

Now to translate this to a RNN. The hidden state should hold evenness so far, and as we saw due to the non linearly separable nature of the $x o r$ , we will need to store both the previous $o r$ and $n a n d$ . This corresponds with the hidden state having a dimension of two. It will not always be so straightforward when it comes to deciding the dimension of a hidden state since most problems aren’t strictly linear like this one, and the precise contents the hidden state will store will not be known beforehand. In most cases, a rough ballpark for the dimension of the hidden state is what is known.

Our simple (aka “Vanilla”) RNN will look like this:

h_{t} = f_{h} (x_{t} W_{x h} + h_{t - 1} W_{h h} + b_{h}) y_{t} = f_{y} (h_{t} W_{h y} + b_{y})

$h_{t}$ is the hidden (or recurrent) state of the cell and $x_{t}$ is the sequence-element at step- $t$ , for $t = 0, 1, \dots, T - 1$ (with $T$ as the length of our sequence). $y_{T - 1}$ is the final output. The $W$ and $b$ parameters, the weights and biases respectively of the dense layer, are the learnable parameters of our model. These equations thus say that the new hidden state ( $h_{t}$ ) combines current input ( $x_{t}$ ) and the previous hidden state ( $h_{t - 1}$ ), then applies an activation function ( $f_{h}$ , e.g., $\tanh$ or $ReLU$ ). The output ( $y_{t}$ ) is then a function of the new hidden state (not necessarily applying the same activation function).

Graphically, the Vanilla RNN looks like this:

graphical representation of Vanilla RNN

Note that the RNN produces a result at each time step which signifies the evenness of the sequence so far. While we are only using the result from the last iteration because we only want to know the classification of the whole sequence, there are applications that use all or more of the outputs. For example in “sequence tagging”, the parts of speech of each word in a text can be labeled with the output from each iteration.

sequence tagging

Backpropagation

The way a RNN learns is with backpropagation through time, or BPTT. Think of this as unrolling the forward pass of a RNN for a number of time steps, or the length of the sequence with shared weights. The way backpropagation is programmed remains the same as before which is nice.

RNN BPTT rolled out

Nulling gradients ensures that they don’t accumulate over time and cause large changes in the weights that do not correspond to the current state of the model. Because of the sequential nature of RNNs (each step is dependent on the previous steps), backpropagation takes a lot of time and memory. This makes nulling gradients even more important!

Turing-Complete

RNNs are Turing-complete, which means that they can represent any program. In other words, RNNs can perform any computable function. This is a parallel to how CNNs were able to approximate or represent functions given some fixed number of inputs by the Universal Approximation Theorem. In stricter terms, for every algorithm, there is at least one finite RNN that can implement it. The RNN takes in and returns binary data. These RNNs have a fixed number of iterations and are structured like the simple RNN we saw earlier. These will have a piecewise linear approximation of a sigmoid as an activation function. The slope of the function will always be zero or undefined, and this corresponds to no learning. As a result, weights and biases have to be predetermined. This means that the structure of a RNN is conducive to representing any program, without having to learn to approximate it. As an exercise, we will manually determine the weights and biases for the RNN to solve the even-odd sequence classification from above.

Variations of RNNs

Lastly, there are some variations in how RNN cells are formulated. Some don’t apply an activation function to the output. Some first compute output as a function of the current state and input, and then update the current state to be this output. The structure of the RNN we looked at is from around the 1990s, and a lot of progress has been made since.

The Vanilla RNN was many-to-one in that it took in the input of a digit at each iteration and produced a one-digit end result signifying even/odd (we ignored the outputs of the previous iterations). We could implement a one-to-many RNN. An example of a use for one is taking in one image and outputting a variable length caption. Another option is the many-to-many, which takes in inputs during multiple iterations and outputs during multiple iterations as well. An example of this is the sequence-tagging from above. The RNN we used for the sequence classifying could be one as well, if we were interested in the evenness of the sequence at each iteration.

The key similarity between all RNNs is that output is ultimately a function of input and a hidden state which is dependent on previous inputs.