Spring 2014 Class 12 Backpointer tables training with continuous speech 24 Mar 2014 1 Backpointers in Viterbi search In Viterbi search we retain a pointer to the best previous state at all trellis nodes ID: 426027
Download Presentation The PPT/PDF document "Design and Implementation of Speech Reco..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Design and Implementation of Speech Recognition Systems
Spring 2014Class 12: Backpointer tables; training with continuous speech24 Mar 2014
1Slide2
Backpointers in
Viterbi searchIn Viterbi search, we retain a pointer to the best previous state at all trellis nodesThis is performed even when we are recognizing continuous speechFrom Grammars etc.
2Slide3
3
Trellis with Complete Set of Backpointers
red
green
blueSlide4
4
Trellis with Complete Set of Backpointerst=1
red
green
blueSlide5
5
Trellis with Complete Set of Backpointerst=1
red
green
blueSlide6
6
Trellis with Complete Set of Backpointerst=1
red
green
blueSlide7
7
Trellis with Complete Set of Backpointerst=1
red
green
blueSlide8
8
Trellis with Complete Set of Backpointerst=1
t=2
red
green
blueSlide9
9
Trellis with Complete Set of Backpointerst=1
t=2
red
green
blueSlide10
10
Trellis with Complete Set of Backpointerst=1
t=2
red
green
blueSlide11
11
Trellis with Complete Set of Backpointerst=1
t=2
red
green
blueSlide12
12
Trellis with Complete Set of Backpointerst=1
t=2
t=3
red
green
blueSlide13
13
Trellis with Complete Set of Backpointerst=1
t=2
t=3
red
green
blueSlide14
14
Trellis with Complete Set of Backpointerst=1
t=2
t=3
red
green
blueSlide15
15
Trellis with Complete Set of Backpointerst=1
t=2
t=3
red
green
blueSlide16
16
Trellis with Complete Set of Backpointerst=1
t=2
t=3
red
green
blueSlide17
17
Trellis with Complete Set of Backpointerst=1
t=2
t=3
red
green
blueSlide18
18
Trellis with Complete Set of Backpointerst=1
t=2
t=3
t
=4
red
green
blueSlide19
19
Trellis with Complete Set of Backpointerst=1
t=2
t=3
t
=4
red
green
blueSlide20
20
Trellis with Complete Set of Backpointerst=1
t=2
t=3
t
=4
red
green
blueSlide21
21
Trellis with Complete Set of Backpointerst=1
t=2
t=3
t
=4
red
green
blueSlide22
22
Trellis with Complete Set of Backpointerst=1
t=2
t=3
t
=4
red
green
blueSlide23
23
Trellis with Complete Set of Backpointerst=1
t=2
t=3
t
=4
red
green
blueSlide24
24
Trellis with Complete Set of Backpointerst=1
t=2
t=3
t
=4
t
=5
red
green
blueSlide25
25
Trellis with Complete Set of Backpointerst=1
t=2
t=3
t
=4
t
=5
red
green
blueSlide26
26
Trellis with Complete Set of Backpointerst=1
t=2
t=3
t
=4
t
=5
red
green
blueSlide27
27
Trellis with Complete Set of Backpointerst=1
t=2
t=3
t
=4
t
=5
red
green
blueSlide28
28
Trellis with Complete Set of Backpointerst=1
t=2
t=3
t
=4
t
=5
red
green
blueSlide29
29
Trellis with Complete Set of Backpointerst=1
t=2
t=3
t
=4
t
=5
red
green
blueSlide30
30
Trellis with Complete Set of Backpointerst=1
t=2
t=3
t
=4
t
=5
t
=6
red
green
blueSlide31
31
Trellis with Complete Set of Backpointerst=1
t=2
t=3
t
=4
t
=5
t
=6
red
green
blueSlide32
32
Trellis with Complete Set of Backpointerst=1
t=2
t=3
t
=4
t
=5
t
=6
red
green
blueSlide33
33
Trellis with Complete Set of Backpointerst=1
t=2
t=3
t
=4
t
=5
t
=6Slide34
34
Trellis with Complete Set of Backpointerst=1
t=2
t=3
t
=4
t
=5
t
=6Slide35
35
Trellis with Complete Set of Backpointerst=1
t=2
t=3
t
=4
t
=5
t
=6Slide36
36
Trellis with Complete Set of Backpointerst=1
t=2
t=3
t
=4
t
=5
t
=6
t
=7Slide37
37
Trellis with Complete Set of Backpointerst=1
t=2
t=3
t
=4
t
=5
t
=6
t
=7Slide38
38
Trellis with Complete Set of Backpointerst=1
t=2
t=3
t
=4
t
=5
t
=6
t
=7Slide39
39
Trellis with Complete Set of Backpointerst=1
t=2
t=3
t
=4
t
=5
t
=6
t
=7Slide40
40
Trellis with Complete Set of Backpointerst=1
t=2
t=3
t
=4
t
=5
t
=6
t
=7Slide41
41
Trellis with Complete Set of Backpointerst=1
t=2
t=3
t
=4
t
=5
t
=6
t
=7Slide42
42
Trellis with Complete Set of Backpointerst=1
t=2
t=3
t
=4
t
=5
t
=6
t
=7Slide43
43
Trellis with Complete Set of Backpointerst=1
t=2
t=3
t
=4
t
=5
t
=6
t
=7
t
=8Slide44
44
Using BackpointersRetaining the complete set of back pointers can be very expensiveIn terms of memorySolution: Only retain back pointers to the entry into words
Which can be stored separately as a “backpointer table”Slide45
45
Trellis with Complete Set of Backpointers
1
red
green
blue
1, t=0, scr1,p=0,…Slide46
46
Trellis with Complete Set of Backpointerst=1
1
red
green
blue
1, t=0, scr1,p=0,…Slide47
47
Trellis with Complete Set of Backpointerst=1
1
red
green
blue
1, t=0, scr1,p=0,…Slide48
48
Trellis with Complete Set of Backpointerst=1
t=2
1
red
green
blue
1, t=0, scr1,p=0,…Slide49
49
Trellis with Complete Set of Backpointerst=1
t=2
1
red
green
blue
1, t=0, scr1,p=0,…Slide50
50
Trellis with Complete Set of Backpointerst=1
t=2
1
red
green
blue
1, t=0, scr1,p=0,…Slide51
51
Trellis with Complete Set of Backpointerst=1
t=2
t=3
1
red
green
blue
1, t=0, scr1,p=0,…Slide52
52
Trellis with Complete Set of Backpointerst=1
t=2
t=3
1
red
green
blue
1, t=0, scr1,p=0,…Slide53
53
Trellis with Complete Set of Backpointerst=1
t=2
t=3
1
red
green
blue
1, t=0, scr1,p=0,…Slide54
54
Trellis with Complete Set of Backpointerst=1
t=2
t=3
1
red
green
blue
1, t=0, scr1,p=0,…Slide55
55
Trellis with Complete Set of Backpointerst=1
t=2
t=3
1
red
green
blue
1, t=0, scr1,p=0,…Slide56
56
Trellis with Complete Set of Backpointerst=1
t=2
t=3
1
2
Retain backpointers (and add the to the table)
if deleting them will result in loss of word
history
red
green
blue
1, t=0, scr1,p=0,…
2, t=3, scr2,p=1,…Slide57
57
Trellis with Complete Set of Backpointerst=1
t=2
t=3
1
2
red
green
blue
1, t=0, scr1,p=0,…
2, t=3, scr2,p=1,…Slide58
58
Trellis with Complete Set of Backpointerst=1
t=2
t=3
1
2
red
green
blue
1, t=0, scr1,p=0,…
2, t=3, scr2,p=1,…Slide59
59
Trellis with Complete Set of Backpointerst=1
t=2
t=3
1
2
red
green
blue
1, t=0, scr1,p=0,…
2, t=3, scr2,p=1,…
t
=4Slide60
60
Trellis with Complete Set of Backpointerst=1
t=2
t=3
1
2
red
green
blue
1, t=0, scr1,p=0,…
2, t=3, scr2,p=1,…
t
=4Slide61
61
Trellis with Complete Set of Backpointerst=1
t=2
t=3
1
2
red
green
blue
1, t=0, scr1,p=0,…
2, t=3, scr2,p=1,…
t
=4Slide62
62
Trellis with Complete Set of Backpointerst=1
t=2
t=3
1
2
red
green
blue
1, t=0, scr1,p=0,…
2, t=3, scr2,p=1,…
t
=4Slide63
63
Trellis with Complete Set of Backpointerst=1
t=2
t=3
1
2
red
green
blue
1, t=0, scr1,p=0,…
2, t=3, scr2,p=1,…
t
=4Slide64
64
Trellis with Complete Set of Backpointerst=1
t=2
t=3
1
2
red
green
blue
1, t=0, scr1,p=0,…
2, t=3, scr2,p=1,…
t
=4Slide65
65
Trellis with Complete Set of Backpointerst=1
t=2
t=3
1
2
red
green
blue
t
=4
3
1, t=0, scr1,p=0,…
2, t=3, scr2,p=1,…
3, t=4, scr3,p=1,…
Retain backpointers (and add the to the table) if deleting them will result in loss of word
history
Backpointer
table entries also have information about word identity (indicated by color in the figure)Slide66
66
Trellis with Complete Set of Backpointerst=1
t=2
t=3
1
2
red
green
blue
t
=4
3
1, t=0, scr1,p=0,…
2, t=3, scr2,p=1,…
3, t=4, scr3,p=1,…Slide67
67
Trellis with Complete Set of Backpointerst=1
t=2
t=3
1
2
red
green
blue
t
=4
3
1, t=0, scr1,p=0,…
2, t=3, scr2,p=1,…
3, t=4, scr3,p=1,…Slide68
68
Trellis with Complete Set of Backpointerst=1
t=2
t=3
1
2
red
green
blue
t
=4
3
1, t=0, scr1,p=0,…
2, t=3, scr2,p=1,…
3, t=4, scr3,p=1,…Slide69
69
Trellis with Complete Set of Backpointerst=1
t=2
t=3
1
2
red
green
blue
t
=4
3
1, t=0, scr1,p=0,…
2, t=3, scr2,p=1,…
3, t=4, scr3,p=1,…Slide70
70
Trellis with Complete Set of Backpointerst=1
t=2
t=3
1
2
red
green
blue
t
=4
3
1, t=0, scr1,p=0,…
2, t=3, scr2,p=1,…
3, t=4, scr3,p=1,…Slide71
71
Trellis with Complete Set of Backpointerst=1
t=2
t=3
1
2
red
green
blue
t
=4
3
1, t=0, scr1,p=0,…
2, t=3, scr2,p=1,…
3, t=4, scr3,p=1,…Slide72
72
Trellis with Complete Set of Backpointerst=1
t=2
t=3
1
2
red
green
blue
t
=4
3
1, t=0, scr1,p=0,…
2, t=3, scr2,p=1,…
3, t=4, scr3,p=1,…
t
=5Slide73
73
Trellis with Complete Set of Backpointerst=1
t=2
t=3
1
2
red
green
blue
t
=4
3
1, t=0, scr1,p=0,…
2, t=3, scr2,p=1,…
3, t=4, scr3,p=1,…
t
=5Slide74
74
Trellis with Complete Set of Backpointerst=1
t=2
t=3
1
2
red
green
blue
t
=4
3
1, t=0, scr1,p=0,…
2, t=3, scr2,p=1,…
3, t=4, scr3,p=1,…
t
=5Slide75
75
Trellis with Complete Set of Backpointerst=1
t=2
t=3
1
2
t
=4
3
1, t=0, scr1,p=0,…
2, t=3, scr2,p=1,…
3, t=4, scr3,p=1,…
t
=5
4
Retain
backpointers
(and add
them
to the
table) if
deleting them will result in loss of
word history
4, t=5, scr4,p=1,…Slide76
76
Trellis with Complete Set of Backpointerst=1
t=2
t=3
1
2
t
=4
3
1, t=0, scr1,p=0,…
2, t=3, scr2,p=1,…
3, t=4, scr3,p=1,…
t
=5
4
4, t=5, scr4,p=1,…Slide77
77
Trellis with Complete Set of Backpointerst=1
t=2
t=3
1
2
t
=4
3
1, t=0, scr1,p=0,…
2, t=3, scr2,p=1,…
3, t=4, scr3,p=1,…
t
=5
4
4, t=5, scr4,p=1,…Slide78
78
Trellis with Complete Set of Backpointerst=1
t=2
t=3
1
2
t
=4
3
1, t=0, scr1,p=0,…
2, t=3, scr2,p=1,…
3, t=4, scr3,p=1,…
t
=5
4
4, t=5, scr4,p=1,…
t
=6Slide79
79
Trellis with Complete Set of Backpointerst=1
t=2
t=3
1
2
t
=4
3
1, t=0, scr1,p=0,…
2, t=3, scr2,p=1,…
3, t=4, scr3,p=1,…
t
=5
4
4, t=5, scr4,p=1,…
t
=6Slide80
80
Trellis with Complete Set of Backpointerst=1
t=2
t=3
1
2
t
=4
3
1, t=0, scr1,p=0,…
2, t=3, scr2,p=1,…
3, t=4, scr3,p=1,…
t
=5
4
4, t=5, scr4,p=1,…
t
=6
Note: This node will
not propagate further
(its time is up)Slide81
81
Trellis with Complete Set of Backpointerst=1
t=2
t=3
1
2
t
=4
3
1, t=0, scr1,p=0,…
2, t=3, scr2,p=1,…
3, t=4, scr3,p=1,…
t
=5
4
4, t=5, scr4,p=1,…
t
=6
Can delete the node and
its
backpointer
if
desiredSlide82
82
Trellis with Complete Set of Backpointerst=1
t=2
t=3
1
2
t
=4
3
1, t=0, scr1,p=0,…
2, t=3, scr2,p=1,…
3, t=4, scr3,p=1,…
t
=5
4
4, t=5, scr4,p=1,…
t
=6Slide83
83
Trellis with Complete Set of Backpointerst=1
t=2
t=3
1
2
t
=4
3
1, t=0, scr1,p=0,…
2, t=3, scr2,p=1,…
3, t=4, scr3,p=1,…
t
=5
4
4, t=5, scr4,p=1,…
t
=6Slide84
84
Trellis with Complete Set of Backpointerst=1
t=2
t=3
1
2
t
=4
3
1, t=0, scr1,p=0,…
2, t=3, scr2,p=1,…
3, t=4, scr3,p=1,…
t
=5
4
4, t=5, scr4,p=1,…
t
=6
5
5, t=6, scr5,p=1,…Slide85
85
Using BackpointersThe Backpointer table in the previous figure only retains sufficient history to obtain the best hypothesis
Sometimes we would like to retain additional information in the backpointer table that tells us what other words were considered (not pruned out) during recognition
e.g. when we want to create lattices for finding N-best hypotheses or to compute confidenceIn this case the
backpointer
table is expanded to include
all
trellis nodes at the final states of words
Additionally, all trellis nodes corresponding to non-emitting nodes may also be storedSlide86
86
Regular BP Tablet=1
t=2
t=3
1
red
green
blue
1, t=0, scr1,p=0,…Slide87
87
Regular Trellist=1
t=2
t=3
1
red
green
blue
1, t=0, scr1,p=0,…Slide88
88
Regular Trellist=1
t=2
t=3
1
2
red
green
blue
1, t=0, scr1,p=0,…
2, t=3, scr2,p=1,…Slide89
89
EXTENDED TRELLISt=1
t=2
t=3
1
red
green
blue
1, t=0, scr1,p=0,…Slide90
90
EXTENDED TRELLISt=1
t=2
t=3
1
2
red
green
blue
1, t=0, scr1,p=0,…
2, t=3, scr2,p=1,…
3
3,
t=3,
scr3,p=1
,…
FAILED
cross-word transitions too enter the
backpointer
tableSlide91
91
EXTENDED TRELLISt=1
t=2
t=3
1
2
red
green
blue
1, t=0, scr1,p=0,…
2, t=3, scr2,p=1,…
3
3,
t=3,
scr3,p=1
,…
FAILED
cross-word transitions too enter the
backpointer
table
BUT DO NOT PROPAGATE FURTHERSlide92
92
Using BackpointersEven with extended backpointer sets (including word ending and null-state entries), back pointer tables can be much smaller than retaining a full
backpointer matrix, one for each trellis node
Backpointers need not be stored as tables. They can, in fact be stored explicitly in tree format
The entries in the
backpointer
table may also contain information about the
forward
transition
E.g. word-ending lattice-node
backpointer
table entries may additionally point forward to the subsequent null-state
This information may be used to derive lattices directly from BP tablesSlide93
Decoding – finesse
The final computational and memory expense of decoding depends on the size of the language HMMWhen composed from HMMs of words (or phonemes, as we will see later), it can get quite largeParticularly from redundant non-emitting statesThe HMM must be minimizedTopic for another day..
93