1 Hitting The Right Paraphrases In Good Time
Description: 1 Hitting The Right Paraphrases In Good Time Stanley Kok Dept. of Comp. Sci. Eng. Univ. of Washington Seattle, USA Chris Brockett NLP Group Microsoft Research Redmond, USA Motivation Background Hitting Time Paraphraser Experiments Future
Related Topics
Download Presentation
"1 Hitting The Right Paraphrases In Good Time" is the property of its rightful owner. Permission is granted to download and print the materials on this website for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Presentation Transcript
slide1. 1 Hitting The Right Paraphrases In Good Time Stanley Kok
Dept. of Comp. Sci. & Eng.
Univ. of Washington
Seattle, USA Chris Brockett
NLP Group
Microsoft Research
Redmond, USA<br>
slide2. Motivation
Background
Hitting Time Paraphraser
Experiments
Future Work 2 Overview<br>
slide3. Motivation
Background
Hitting Time Paraphraser
Experiments
Future Work 3 Overview<br>
slide4. 4 What’s a paraphrase of… Paraphrase
System “is on good terms with” “is friendly with”
“is a friend of”
… Query expansion
Document summarization
Natural language generation
Question answering
etc. Applications<br>
slide5. 5 What’s a paraphrase of…<br>
slide6. 6 Bilingual Parallel Corpus …the cost dynamic is under control… …die kostenentwicklung unter kontrolle… …keep the cost in check… …die kosten unter kontrolle… … … Phrase Table<br>
slide7. BCB system [Bannard & Callison-Burch, ACL’05]
P(E2|E1) ¼C G P(E2|G) P(G|E1)
SBP system [Callison-Burch, EMNLP’08]
P(E2|E1) ¼C G P(E2|G,syn(E1)) p(G|E1, syn(E1)) 7 State of the Art<br>
slide8. 8 E1 E2 G1 F2 E3 E4 (in check) (under control) G2 G3 (unter kontrolle) F1 Graphical View<br>
slide9. 9 Graphical View Path lengths > 2 Add nodes to represent domain knowledge G1 F2 G2 G3 F1 E1 E2 E3 E4<br>
slide10. Motivation
Background
Hitting Time Paraphraser
Experiments
Future Work 10 Overview<br>
slide11. A A Random Walk Begin at node A
Randomly pick neighbor n E F D B C 11<br>
slide12. Random Walk Begin at node A
Randomly pick neighbor n
Move to node n E F D A 2 B C 12<br>
slide13. Random Walk Begin at node A
Randomly pick neighbor n
Move to node n
Repeat E F D A B 2 C 13<br>
slide14. Expected number of steps starting from node i before node j is visited for first time
Smaller hitting time → closer to start node i
Truncated Hitting Time [Sarkar & Moore, UAI’07]
Random walks are limited to T steps
Computed efficiently & with high probability by sampling random walks [Sarkar, Moore & Prakash ICML’08] 14 Hitting Time from node i to j<br>
slide15. Finding Truncated Hitting Time By Sampling E F D 1 B C A A T=5 15<br>
slide16. Finding Truncated Hitting Time By Sampling E F 4 A B C D A D T=5 16<br>
slide17. Finding Truncated Hitting Time By Sampling 5 F D A B C E A D E T=5 17<br>
slide18. Finding Truncated Hitting Time By Sampling E F 4 A B C D A D E D T=5 18<br>
slide19. Finding Truncated Hitting Time By Sampling E 6 D A B C F A D E D F T=5 19<br>
slide20. Finding Truncated Hitting Time By Sampling 5 F D A B C E A D E D F E T=5 20<br>
slide21. Finding Truncated Hitting Time By Sampling A D E D F E T=5 E F D A B C hAD=1 hAE=2 hAF=4 hAA=0 hAB=5 hAC=5 21<br>
slide22. Motivation
Background
Hitting Time Paraphraser
Experiments
Future Work 22 Overview<br>
slide23. 23 Hitting Time Paraphraser
(HTP) Phrase Tables English-German
English-French
German-French
etc. Phrase Paraphrases<br>
slide24. 24 Graph Construction<br>
slide25. 25 Graph Construction<br>
slide26. BFS from query phrase up to depth d or up to max. number n of nodes
d = 6, n = 50,000 26 … … … … … … … … … Graph Construction<br>
slide27. 27 Graph Construction … … … … … … … … …<br>
slide28. 28 Graph Construction … … … … … … … … … 0.6<br>
slide29. 29 Graph Construction … … … … … … … … … 0.5 0.5<br>
slide30. Run m truncated random walks to estimate truncated hitting time of each node
T = 10, m = 1,000,000
Prune nodes with hitting times = T Estimate Trunc. Hitting Times<br>
slide31. 31 Add Ngram Nodes “achieve the goal” “achieve the aim” “reach the objective”<br>
slide32. 32 Add “Syntax” Nodes “whose goal is” “the aim is” “the objective is” “what goal” start with article end with be start with
interrogatives<br>
slide33. 33 Add Not-Substring-Of Nodes “reach the” “reach the aim” “reach the objective” “objective” not-substring-of<br>
slide34. 34 Feature Nodes ngram nodes “syntax” nodes not-substring nodes phrase nodes<br>
slide35. Run m truncated random walks again
Rank paraphrases in increasing order of hitting times 35 Re-estimate Truncated Hitting Times<br>
slide36. Motivation
Background
Hitting Time Paraphraser
Experiments
Future Work 36 Overview<br>
slide37. Europarl dataset [Koehn, MT-Summit’05]
Use 6 of 11 languages: English, Danish, German, Spanish, Finnish, Dutch
About a million sentences per language
English−Foreign phrasal alignments by giza++ [Callison-Burch, EMNLP’08]
Foreign−Foreign phrasal alignments by MSR aligner 37 Data<br>
slide38. SBP system [Callison-Burch, EMNLP’08]
HTP with no feature node
HTP with bipartite graph 38 Comparison Systems<br>
slide39. NIST dataset
4 English translations per Chinese sentence
33,216 English translations
Randomly selected 100 English phrases
From 1-4grams in both NIST & Europarl datasets
Exclude stop words, numbers, phrases containing periods and commas 39 Evaluation Methodology<br>
slide40. For each phrase, randomly select a sentence from NIST dataset containing it
Substituted top 1 to 10 paraphrases for phrase 40 Methodology<br>
slide41. Manually evaluated resulting sentences
0: Clearly wrong; grammatically incorrect or does
not preserve meaning
1: Minor grammatical errors (e.g., subject-verb
disagreement; wrong tenses, etc.),
or meaning largely preserved but not completely
2: Totally correct; grammatically correct and
meaning is preserved
Correct: 1 and 2; Wrong: 0
Two evaluators; Kappa = 0.62 (substantial agree.) 41 Methodology<br>
slide42. 42 HTP vs. SBP<br>
slide43. 43 HTP vs. SBP 373
paraphrases per system<br>
slide44. 44 HTP vs. SBP 483
paraphrases 0.54<br>
slide45. 45 HTP vs. SBP 0.53 0.50<br>
slide46. 46 HTP vs. SBP 975
paraphrases 0.32 373
paraphrases 492
paraphrases 0.43 420 correct
paraphrases 145 correct
paraphrases<br>
slide47. 47 Timings<br>
slide48. Motivation
Background
Hitting Time Paraphraser
Experiments
Future Work 48 Overview<br>
slide49. Apply HTP to languages other than English
Evaluate HTP impact on applications
e.g., improve performance of resource-sparse machine translation systems
Add more features
etc. 49 Future Work<br>
slide50. HTP: a paraphrase system based on random walks
Good paraphrases have smaller hitting times
General graph
Path length > 2
Incorporate domain knowledge
HTP outperforms state-of-the-art 50 Conclusion<br>
Dept. of Comp. Sci. & Eng.
Univ. of Washington
Seattle, USA Chris Brockett
NLP Group
Microsoft Research
Redmond, USA<br>
slide2. Motivation
Background
Hitting Time Paraphraser
Experiments
Future Work 2 Overview<br>
slide3. Motivation
Background
Hitting Time Paraphraser
Experiments
Future Work 3 Overview<br>
slide4. 4 What’s a paraphrase of… Paraphrase
System “is on good terms with” “is friendly with”
“is a friend of”
… Query expansion
Document summarization
Natural language generation
Question answering
etc. Applications<br>
slide5. 5 What’s a paraphrase of…<br>
slide6. 6 Bilingual Parallel Corpus …the cost dynamic is under control… …die kostenentwicklung unter kontrolle… …keep the cost in check… …die kosten unter kontrolle… … … Phrase Table<br>
slide7. BCB system [Bannard & Callison-Burch, ACL’05]
P(E2|E1) ¼C G P(E2|G) P(G|E1)
SBP system [Callison-Burch, EMNLP’08]
P(E2|E1) ¼C G P(E2|G,syn(E1)) p(G|E1, syn(E1)) 7 State of the Art<br>
slide8. 8 E1 E2 G1 F2 E3 E4 (in check) (under control) G2 G3 (unter kontrolle) F1 Graphical View<br>
slide9. 9 Graphical View Path lengths > 2 Add nodes to represent domain knowledge G1 F2 G2 G3 F1 E1 E2 E3 E4<br>
slide10. Motivation
Background
Hitting Time Paraphraser
Experiments
Future Work 10 Overview<br>
slide11. A A Random Walk Begin at node A
Randomly pick neighbor n E F D B C 11<br>
slide12. Random Walk Begin at node A
Randomly pick neighbor n
Move to node n E F D A 2 B C 12<br>
slide13. Random Walk Begin at node A
Randomly pick neighbor n
Move to node n
Repeat E F D A B 2 C 13<br>
slide14. Expected number of steps starting from node i before node j is visited for first time
Smaller hitting time → closer to start node i
Truncated Hitting Time [Sarkar & Moore, UAI’07]
Random walks are limited to T steps
Computed efficiently & with high probability by sampling random walks [Sarkar, Moore & Prakash ICML’08] 14 Hitting Time from node i to j<br>
slide15. Finding Truncated Hitting Time By Sampling E F D 1 B C A A T=5 15<br>
slide16. Finding Truncated Hitting Time By Sampling E F 4 A B C D A D T=5 16<br>
slide17. Finding Truncated Hitting Time By Sampling 5 F D A B C E A D E T=5 17<br>
slide18. Finding Truncated Hitting Time By Sampling E F 4 A B C D A D E D T=5 18<br>
slide19. Finding Truncated Hitting Time By Sampling E 6 D A B C F A D E D F T=5 19<br>
slide20. Finding Truncated Hitting Time By Sampling 5 F D A B C E A D E D F E T=5 20<br>
slide21. Finding Truncated Hitting Time By Sampling A D E D F E T=5 E F D A B C hAD=1 hAE=2 hAF=4 hAA=0 hAB=5 hAC=5 21<br>
slide22. Motivation
Background
Hitting Time Paraphraser
Experiments
Future Work 22 Overview<br>
slide23. 23 Hitting Time Paraphraser
(HTP) Phrase Tables English-German
English-French
German-French
etc. Phrase Paraphrases<br>
slide24. 24 Graph Construction<br>
slide25. 25 Graph Construction<br>
slide26. BFS from query phrase up to depth d or up to max. number n of nodes
d = 6, n = 50,000 26 … … … … … … … … … Graph Construction<br>
slide27. 27 Graph Construction … … … … … … … … …<br>
slide28. 28 Graph Construction … … … … … … … … … 0.6<br>
slide29. 29 Graph Construction … … … … … … … … … 0.5 0.5<br>
slide30. Run m truncated random walks to estimate truncated hitting time of each node
T = 10, m = 1,000,000
Prune nodes with hitting times = T Estimate Trunc. Hitting Times<br>
slide31. 31 Add Ngram Nodes “achieve the goal” “achieve the aim” “reach the objective”<br>
slide32. 32 Add “Syntax” Nodes “whose goal is” “the aim is” “the objective is” “what goal” start with article end with be start with
interrogatives<br>
slide33. 33 Add Not-Substring-Of Nodes “reach the” “reach the aim” “reach the objective” “objective” not-substring-of<br>
slide34. 34 Feature Nodes ngram nodes “syntax” nodes not-substring nodes phrase nodes<br>
slide35. Run m truncated random walks again
Rank paraphrases in increasing order of hitting times 35 Re-estimate Truncated Hitting Times<br>
slide36. Motivation
Background
Hitting Time Paraphraser
Experiments
Future Work 36 Overview<br>
slide37. Europarl dataset [Koehn, MT-Summit’05]
Use 6 of 11 languages: English, Danish, German, Spanish, Finnish, Dutch
About a million sentences per language
English−Foreign phrasal alignments by giza++ [Callison-Burch, EMNLP’08]
Foreign−Foreign phrasal alignments by MSR aligner 37 Data<br>
slide38. SBP system [Callison-Burch, EMNLP’08]
HTP with no feature node
HTP with bipartite graph 38 Comparison Systems<br>
slide39. NIST dataset
4 English translations per Chinese sentence
33,216 English translations
Randomly selected 100 English phrases
From 1-4grams in both NIST & Europarl datasets
Exclude stop words, numbers, phrases containing periods and commas 39 Evaluation Methodology<br>
slide40. For each phrase, randomly select a sentence from NIST dataset containing it
Substituted top 1 to 10 paraphrases for phrase 40 Methodology<br>
slide41. Manually evaluated resulting sentences
0: Clearly wrong; grammatically incorrect or does
not preserve meaning
1: Minor grammatical errors (e.g., subject-verb
disagreement; wrong tenses, etc.),
or meaning largely preserved but not completely
2: Totally correct; grammatically correct and
meaning is preserved
Correct: 1 and 2; Wrong: 0
Two evaluators; Kappa = 0.62 (substantial agree.) 41 Methodology<br>
slide42. 42 HTP vs. SBP<br>
slide43. 43 HTP vs. SBP 373
paraphrases per system<br>
slide44. 44 HTP vs. SBP 483
paraphrases 0.54<br>
slide45. 45 HTP vs. SBP 0.53 0.50<br>
slide46. 46 HTP vs. SBP 975
paraphrases 0.32 373
paraphrases 492
paraphrases 0.43 420 correct
paraphrases 145 correct
paraphrases<br>
slide47. 47 Timings<br>
slide48. Motivation
Background
Hitting Time Paraphraser
Experiments
Future Work 48 Overview<br>
slide49. Apply HTP to languages other than English
Evaluate HTP impact on applications
e.g., improve performance of resource-sparse machine translation systems
Add more features
etc. 49 Future Work<br>
slide50. HTP: a paraphrase system based on random walks
Good paraphrases have smaller hitting times
General graph
Path length > 2
Incorporate domain knowledge
HTP outperforms state-of-the-art 50 Conclusion<br>