Lecture 12 Sequence to sequence models Alireza Akhavan Pour CLASSVISION Sequence to sequence model Introduction and concepts 2 شنبه ۱۰ آذر ۱۳۹۷ 3 شنبه ۱۰ آذر ۱۳۹۷ ID: 794148
Download The PPT/PDF document "1 شنبه، ۱۰ آذر ۱۳۹۷" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
1
شنبه، ۱۰ آذر ۱۳۹۷
Lecture 12: Sequence to sequence models
Alireza Akhavan Pour
CLASS.VISION
Slide2Sequence to sequence model: Introduction and concepts
2
شنبه، ۱۰ آذر ۱۳۹۷
Slide33
شنبه، ۱۰ آذر ۱۳۹۷
Slide44
شنبه، ۱۰ آذر ۱۳۹۷
Sequence to sequence modelJane visite l’Afrique en septembre
Jane is visiting Africa in September.
[Cho et al., 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation]
[Sutskever et al., 2014. Sequence to sequence learning with neural networks]
Slide55
شنبه، ۱۰ آذر ۱۳۹۷
Sequence to sequence modelJane visite l’Afrique en septembre
Jane is visiting Africa in September.
[Cho et al., 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation]
[Sutskever et al., 2014. Sequence to sequence learning with neural networks]
Encoder
Decoder
Slide66
شنبه، ۱۰ آذر ۱۳۹۷
A cat sitting on a chair
55
55
6
27
27
96
27
27
256
13
13
256
11
11
s = 4
3
3
s = 2
MAX-POOL
5
5
same
3
3
s = 2
MAX-POOL
13
13
384
3
3
same
3
3
=
13
13
384
13
13
256
6
6
256
3
3
3
3
s = 2
MAX-POOL
9216
Softmax
1000
4096
4096
[Mao et. al., 2014. Deep captioning with multimodal recurrent neural networks]
[
Vinyals
et. al., 2014. Show and tell
: Neural
image caption generator]
[
Karpathy
and Li, 2015. Deep visual-semantic alignments for generating image descriptions]
Image captioning
Slide77
شنبه، ۱۰ آذر ۱۳۹۷
Language model:
Machine translation as building a conditional language model
Slide88
شنبه، ۱۰ آذر ۱۳۹۷
Language model:
Machine translation:
Machine translation as building a conditional language model
وکتور 0
State
ی که
encoder
ایجاد کرده
Conditional language model
Slide99
شنبه، ۱۰ آذر ۱۳۹۷
Jane visite l’Afrique en septembre.
Jane is visiting Africa in September.
Jane is going to be visiting Africa in September.
In September, Jane will visit Africa.
Her African friend welcomed Jane in September.
Finding the most likely translation
English
French
Slide1010
شنبه، ۱۰ آذر ۱۳۹۷
Jane is visiting Africa in September.
Jane is going to be visiting Africa in September.
Why not a greedy search?
>
Slide11Beam
search
11شنبه، ۱۰ آذر ۱۳۹۷
Slide1212
شنبه، ۱۰ آذر ۱۳۹۷
a
in
jane
september
zulu
Step 1
Beam search algorithm
B = 3 (Beam width)
French
English
Slide1313
شنبه، ۱۰ آذر ۱۳۹۷
a
injaneseptemberzulu
Step 1
Step 2
Beam search algorithm
a
aaron
september
zulu
in
in
a
visiting
is
zulu
a
zulu
(B=3)
Slide1414
شنبه، ۱۰ آذر ۱۳۹۷
a
injaneseptemberzulu
Step 1
Step 2
Beam search algorithm
a
aaron
september
zulu
in
in
a
visiting
is
zulu
a
zulu
(B=3)
Slide1515
شنبه، ۱۰ آذر ۱۳۹۷
Beam search
in septemberjane isjane visits
september
in
jane visits
africa
in
september
. <EOS>
is
jane
visits
jane
برای هر کدام از این 3 خروجی، احتمال ها را نیز ذخیره کرده ایم
Slide16Refinements to beam search
16
شنبه، ۱۰ آذر ۱۳۹۷
Slide17Length normalization
17
شنبه، ۱۰ آذر ۱۳۹۷
|
) P(
|
,
وابسته به طول خروجی!
?
?
Slide18Beam search discussion
18
شنبه، ۱۰ آذر ۱۳۹۷Beam width B?
Unlike exact search algorithms like BFS (Breadth First Search) or DFS (Depth First Search), Beam Search runs faster but is not guaranteed to find exact maximum for.
you
might see in the production setting
B=10.
B=100, B=1000 are uncommon (sometimes used in research settings)
Large B:
Better result, slower
Small B:
worse result, faster
Slide19Error analysis on beam
search
19شنبه، ۱۰ آذر ۱۳۹۷
Slide20Example
20
شنبه، ۱۰ آذر ۱۳۹۷Jane visite l’Afrique en septembre.
Human: Jane visits Africa in September.Algorithm: Jane visited Africa last September.
RNN
Beam search
Jane
visits
Africa
…
Slide21Error analysis on beam search
21
شنبه، ۱۰ آذر ۱۳۹۷
Human: Jane visits Africa in September. Algorithm: Jane visited Africa last September.
Case 1:
Beam search chose
. But
attains higher
Conclusion: Beam search is at fault.
Case 2:
is a better translation than
But RNN predicted
Conclusion: RNN model is at fault.
(P(y
*
| X) > P(ŷ | X))
(P(y
*
| X) <= P(ŷ | X))
Slide22Error analysis process
22
شنبه، ۱۰ آذر ۱۳۹۷Jane visits Africa in September.
Jane visited Africa last September.
Human
Algorithm
At fault?
Figures out what faction of errors are “due to” beam search vs
. RNN
model
Slide2323
شنبه، ۱۰ آذر ۱۳۹۷
منابعhttps://www.coursera.org/specializations/deep-learning
https://towardsdatascience.com/sequence-to-sequence-model-introduction-and-concepts-44d9b41cd42d