Noriko Tomuro 1 CSC 578 Neural Networks and Deep Learning PowerPoint Presentation, PPT

Noriko Tomuro 1 CSC 578 Neural Networks and Deep Learning PowerPoint Presentation, PPT - DocSlides

Fall 2018/19. 7. Recurrent Neural Networks. (Some figures adapted from . NNDL book. ). Recurrent Neural Networks. Noriko Tomuro. 2. Recurrent Neural Networks (RNNs). RNN Training. Loss Minimization. Bidirectional RNNs. ID: 720283

By natalia-silvester
Watch All docs

Likes : 2
Views : 4

Download this presentation

Click below link (As may be) to get this presentation.

Noriko Tomuro 1 CSC 578 Neural Networks and Deep Learning

Download Note - The PPT/PDF document "Noriko Tomuro 1 CSC 578 Neural Networks ..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.

Presentations text content in Noriko Tomuro 1 CSC 578 Neural Networks and Deep Learning

Noriko Tomuro
1
CSC 578Neural Networks and Deep Learning
Fall 2018/19
7. Recurrent Neural Networks
(Some figures adapted from
NNDL book
)
Recurrent Neural Networks
Noriko Tomuro
2
Recurrent Neural Networks (RNNs)
RNN Training
Loss Minimization
Bidirectional RNNs
Encoder-Decoder NNs
Deep RNNs
Recursive NNs
Long Short-Term Memory (LSTM)
LSTM Code Example
1 Recurrent Neural Networks
Machine Learning
, Tom Mitchell
3
Sequence models – beyond one-to-one problems.
1 Recurrent Neural Networks
Machine Learning
, Tom Mitchell
4
Recurrent Neural Networks (RNNs) use outputs of network units at time
t
as the input to other units at time
t+1
.
Because of the topologies, RNNs are often used in
sequential modeling
such as time series data.
Also the information brought from time
t
to
t+1
is essentially the context of the preceding input, and serves as the network’s internal memory.
Machine Learning
, Tom Mitchell
5
Sequence modeling
is to predict the next value Y
i
from the preceding values Y
1
..Y
i-1
(e.g. stock market price), or to predict an
output sequence
Y
1
..Y
n
for the given input sequence X1..Xn (e.g. part-of-speech tagging in NLP).There are many sequence models in Machine Learning, such as (Hidden) Markov Models, Maximum Entropy and Conditional Random Fields. In Neural Networks, sequence modeling can be depicted as:
Noriko Tomuro
6
Back to RNN. A basic RNNs are essentially equivalent to feed-forward networks since recurrence can be
unfolded
(in time).
Noriko Tomuro
7
Information from time t-1 could be from its hidden node(s) or output depending on the architecture.
Accordingly the activation function will be different.
Elman network Jordan network
(less powerful)
Note
: b and c are bias vectors. Also the neuron activation function could be something else besides tanh, such as
ReLU
.
2 RNN Training
http://www.wildml.com/2015/10/recurrent-neural-networks-tutorial-part-3-backpropagation-through-time-and-vanishing-gradients/
8
Each sequence produces an error as the
sum
of the deviations of all target signals
from the corresponding activations computed by the network.
To measure the error at each time
t
, most of the
loss functions
used in feed-forward neural networks can be used:
Negative likelihood:
Mean squared error (MSE)
Cross-Entropy:
Noriko Tomuro
9
There is also another network architecture for training, called ‘teacher forcing’.
Rather than the values computed at hidden or output nodes, the information from the previous node uses the
correct, target output
(from the training data).
0
Machine Learning
, Tom Mitchell
10
For loss minimization, common approaches are:
Gradient Descent
The standard method is
BackPropagation
Through Time (BPTT)
, which is a generalization of the BP algorithm for feed-forward networks.
Basically the error computed at
the end of the (input) sequence,
which is the sum of all errors in
the sequence, is propagated
backward through the ENTIRE
sequence, e.g. For
t
=3, where
Since the sequence could be long, oftentimes we clip the backward propagation by truncating the backpropagation to a few steps.
2.1 Loss Minimization
1
Machine Learning
, Tom Mitchell
11
2
DNN book
12
Then the gradient on the various parameters become:
However, gradient descent suffers from the same
vanishing gradient
problem as the feed-forward networks.
3
http://www.wildml.com/2015/10/recurrent-neural-networks-tutorial-part-3-backpropagation-through-time-and-vanishing-gradients/
13
4
Machine Learning
, Tom Mitchell
14
Global optimization methods
“Training the weights in a neural network can be modeled as a non-linear
global optimization problem
.
Arbitrary global optimization techniques may then be used to minimize this target function.
The most common global optimization method for training RNNs is
Genetic Algorithms
..” [
Wikipedia
]
5
Noriko Tomuro
15
3 Bidirectional RNNs
Bidirectional RNNs (BRNNs) combine an RNN that moves
forward
through time, beginning from the start of the sequence, with another RNN that moves
backward
through time, beginning from the end of the sequence.
By using two time directions, input information from the
past
and
future
of the current time frame can be used unlike standard RNN which requires the delays for including future information.
BRNNs can be trained using similar algorithms to RNNs, because the two directional neurons do not have any interactions.
6
Noriko Tomuro
16
4 Encoder-Decoder NNs
Generally speaking, encoder-Decoder networks learn the mapping from an input sequence to an output sequence.
With multilayer feed-forward networks, such networks are called ‘auto associators’:
Input and output could be the same (to learn the identity function -> compression) or different (e.g., classification with one-hot-vector output representation)
7
Noriko Tomuro
17
With recursive networks, an encoder-decoder architecture acts on sequence as the input/output unit, NOT a single unit/neuron. There is an RNN for encoding, and another RNN for decoding.
A hidden state connected from the end of the input layer essentially represents the context (variable C), or a semantic summary, of the input sequence, and it is connected to the decoder RNN.
8
Noriko Tomuro
18
5 Deep RNNs
RNNs can be made to deep networks in many ways. For example,
9
Noriko Tomuro
19
6 Recursive NNs
A
recursive
neural network is a kind of deep neural network created by applying
the same set of weights
recursively over a structured input.
“In the most simple architecture, nodes are combined into parents using a weight matrix that is shared across the whole network, and a non-linearity such as tanh.”
[
Wikipedia
]
0
Noriko Tomuro
20
If c
1
and c
2
are n-dimensional vector representation of nodes, their parent will also be an n-dimensional vector, calculated as
, where W is a
n
x 2
n
matrix.
Training
:
Typically, stochastic gradient descent (SGD) is used to train the network. The gradient is computed using
backpropagation through structure
(BPTS), a variant of backpropagation through time used for recurrent neural networks.
[
Wikipedia
]
1
Noriko Tomuro
21
7 Long Short-Term Memory (LSTM)
The idea for RNNs is to incorporate
dependencies
-- information from earlier in the input sequence, as the
context
or
memory
, in processing the current input.
Long Short-Term Memory (LSTM) networks are a special kind of RNN, capable of learning
long-term/distance
dependencies.
An LSTM network consists of
LSTM
units
. A common LSTM unit is composed of a context/state cell, an
input gate, an output gate and a forget gate. The cell remembers values over
arbitrary time intervals and the
three gates
regulate the flow of
information
into and out of the
cell.
[
Wikipedia
]
2
http://localhost:8888/notebooks/Temp-Heaton/t81_558_class10_lstm.ipynb
22
The Big Picture
:
LSTM maintains an internal state and produces an output. The following diagram shows an LSTM unit over three time slices: the current time slice (t), as well as the previous (t-1) and next (t+1) slice:
C is the context value. Both the output and context values are always fed to the next time slice.
3
The sigmoid layer outputs numbers between 0-1 determine how much
each component should be let through. Pink X gate is point-wise multiplication.
4
http://colah.github.io/posts/2015-08-Understanding-LSTMs/
24
Step-by-step walk through:
Forget gate controls the info coming from h
t-1
for the new input
x
t
(resulting in 0-1 by sigmoid), for all internal state cells (
i
’s) for time
t
.
Input gate applies sigmoid to control which values (
i
’s) to keep this time
t
. Then tanh is applied to create the draft of the new context C
t
.
5
Noriko Tomuro
25
(3) The old state C
t-1
is multiplied by f
t
, to forget the things in the previous context that we decided to forget earlier. Then we add
i
t
∗C~
t
. This is the new candidate values, scaled by how much we decided to update each state value.
(4) We also decide which information to output (by filtering through the output gate). Also the hidden value for this time t is set by the output value multiplied by the tanh of the new context.
6
RNN vs LSTM
7
http://colah.github.io/posts/2015-08-Understanding-LSTMs/
27
Gated Recurrent Unit (GRU)
8
GRUs also takes x
t
and h
t-1
as inputs. They perform some calculations and then pass along h
t
. What makes them different from LSTMs is that GRUs don't need the cell layer to pass values along. The calculations within each iteration insure that the h
t
values being passed along either retain a high amount of old information or are jump-started with a high amount of new information.
9
Noriko Tomuro
29
8 LSTM Code Example
0
Noriko Tomuro
30
The data has 256 input variables (a vector of size 256 for one input instance) and 1 output variable.
There are 128 (hidden) LSTM units for each time step/slice.
The task is binary classification.
Good explanation of ‘the number of hidden units’ in LSTM:
https://www.quora.com/What-is-the-relationship-between-timestep-and-number-hidden-unit-in-LSTM
1
Noriko Tomuro
31
The data has 1 input variable and 1 output variable.
4 (hidden) LSTM units are chosen (for each time step/slice).
The task is regression.
Another example:
https://machinelearningmastery.com/return-sequences-and-return-states-for-lstms-in-keras/