Parallel Algorithms: Theory and Practice CS260 –
Description: Parallel Algorithms: Theory and Practice CS260 Lecture 3 Yihan Sun Some of the slides are from MIT 6.712, 6.886 and CMU 15-853. Last Lecture Two scan algorithms Divide-and-conquer Reduce the problem size Computational models PRAM
Related Topics
Download Presentation
"Parallel Algorithms: Theory and Practice CS260 –" is the property of its rightful owner. Permission is granted to download and print the materials on this website for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Presentation Transcript
slide1. Parallel Algorithms: Theory and Practice CS260 – Lecture 3
Yihan Sun * Some of the slides are from MIT 6.712, 6.886 and CMU 15-853.<br>
slide2. Last Lecture Two scan algorithms
Divide-and-conquer
Reduce the problem size
Computational models
PRAM
Fork-join: N-way vs. binary
Atomic primitives
Implement parallel algorithms using C++
I’ll ask for your feedback (about) every two lectures, you can be prepared for that 2<br>
slide3. In this lecture Matrix multiplication
Solve recurrences using Master Theorem
Some applications that can be solved in parallel 3<br>
slide4. Homework 1 The deadline has been extended to 27th Jan – 5 more days
Code submitted to iLearn
If you want to use the late days, let me know
After this lecture, you’ll be able to solve all the problems in Homework 1 4 Course Website: https://www.cs.ucr.edu/~yihans/teaching/palgo.html
Under my UCR homepage<br>
slide5. Matrix Multiplication 5<br>
slide6. Matrix Multiplication Consider standard iterative matrix-multiplication algorithm X Y Z := for i = 1 to N do
for j = 1 to N do
for k = 1 to N do
Z[i][j] += X[i][k] * Y[k][j]<br>
slide7. Matrix Multiplication 7 for i = 1 to N do
for j = 1 to N do
for k = 1 to N do
Z[i][j] += X[i][k] * Y[k][j] par_for i = 1 to N do
par_for j = 1 to N do
for k = 1 to N do
Z[i][j] += X[i][k] * Y[k][j] par_for i = 1 to N do
par_for j = 1 to N do
par_for k = 1 to N do
temp[k] = X[i][k] * Y[k][j];
Z[i][j] = parallel_reduce(temp, N);<br>
slide8. Recursive Matrix Multiplication X11 Y11 Z11 := Z12 Z21 Z22 X12 X21 X22 Y21 Y12 Y22 Compute 8 submatrix products recursively
Z11 := X11Y11 + X12Y21
Z12 := X11Y12 + X12Y22
Z21 := X21Y11 + X22Y21
Z22 := X21Y12 + X22Y21<br>
slide9. Matrix Multiplication 9<br>
slide10. How to solve a recurrence in general? 10<br>
slide11. Solving recurrences – Master Theorem 11<br>
slide12. 12 … … … …<br>
slide13. 13 … …<br>
slide14. 14 … …<br>
slide15. 15 …<br>
slide16. 16 … … … You can always analyze a specific algorithm using the tree …<br>
slide17. Master Theorem 17<br>
slide18. Matrix multiplication 18<br>
slide19. Master Theorem Quiz 19<br>
slide20. Use recurrence to compute work and depth 20 S1;
S2; In parallel:
S1;
S2; Case 1:
S = execute S2 after finishing S1 Case 2:
S = execute S2 and S1 in parallel<br>
slide21. Flatten algorithm 21<br>
slide22. Flatten Algorithm 22<br>
slide23. Flatten algorithm 23 Flatten(A, n) {
parallel_for (i = 0 to n) S[i] = |A[i]|;
offset = scan_exclusive(S, n);
parallel_for (i = 0 to n) {
off = offset[i];
parallel_for (j = 0 to S[i])
B[off+j]=A[i][j];
}
return B;} Element b2 is at location 3+2=5 Element e1 is at location 13+1=14<br>
slide24. Constructing a Tableau 24<br>
slide25. Edit Distance: Dynamic Programming Given two strings A and B, the edit distance of A and B is the least number of edits (inserts and deletes) to modify A to B
S[i][j] is the edit distance of A[1..i] and B[1..j]
S[i][j] = min of
S[i-1][j-1] if A[i]==B[j]: convert A[1..i-1] to B[1..j-1] and keep A[i] since A[i]=B[j]
S[i-1][j]+1: convert A[1..i-1] to B[1..j] and delete A[i]
S[i][j-1]+1: convert A[1..i] to B[1..j-1] and insert B[j] at the end atcacac acac Case 1: just convert atcaca to aca (if A[i]==B[j])
S[i][j]=S[i-1][j-1] A[1..i]= B[1..j]= Case 2: convert atcaca to acac, and delete the last “c” in A
S[i][j]=S[i-1][j]+1 atcacac acac A[1..i]= B[1..j]= atcacac acac A[1..i]= B[1..j]= Case 3: convert atcacac to aca, and insert a “c” at the end of A
S[i][j]=S[i][j-1]+1 Take the minimum of them acab atcacac b<br>
slide26. Edit Distance: Dynamic Programming S[i][j] = min of
S[i-1][j-1] if A[i]==B[j]
S[i-1][j]+1
S[i][j-1]+1 A B insert delete common Note: can be filled in any order as long as the cells to the left and above are filled.
Can follow path back through matrix to construct edits.<br>
slide27. Page27 Edit Distance: Dynamic Programming Sequential code:
for i = 1 to n
M[i,1] = i;
for j = 1 to m
M[1,j] = j;
for i = 2 to n
for j = 2 to m
if (A[i] == B[j])
M[i,j] = M[i-1,j-1];
else
M[i,j] = 1 + min(M[i-1,j],M[i,j-1]); 15-853<br>
slide28. Constructing a Tableau 28 Dynamic Programming
Longest common subsequence
Edit distance
……<br>
slide29. Constructing a Tableau 29<br>
slide30. Constructing a Tableau 30<br>
slide31. Constructing a Tableau: 4-way divide-and-conquer 31<br>
slide32. Constructing a Tableau 32<br>
slide33. Constructing a Tableau 33<br>
slide34. Constructing a Tableau: 9-way divide-and-conquer 34<br>
slide35. Filtering/packing 35<br>
slide36. Parallel filtering / packing 36<br>
slide37. Parallel filtering / packing 37<br>
Yihan Sun * Some of the slides are from MIT 6.712, 6.886 and CMU 15-853.<br>
slide2. Last Lecture Two scan algorithms
Divide-and-conquer
Reduce the problem size
Computational models
PRAM
Fork-join: N-way vs. binary
Atomic primitives
Implement parallel algorithms using C++
I’ll ask for your feedback (about) every two lectures, you can be prepared for that 2<br>
slide3. In this lecture Matrix multiplication
Solve recurrences using Master Theorem
Some applications that can be solved in parallel 3<br>
slide4. Homework 1 The deadline has been extended to 27th Jan – 5 more days
Code submitted to iLearn
If you want to use the late days, let me know
After this lecture, you’ll be able to solve all the problems in Homework 1 4 Course Website: https://www.cs.ucr.edu/~yihans/teaching/palgo.html
Under my UCR homepage<br>
slide5. Matrix Multiplication 5<br>
slide6. Matrix Multiplication Consider standard iterative matrix-multiplication algorithm X Y Z := for i = 1 to N do
for j = 1 to N do
for k = 1 to N do
Z[i][j] += X[i][k] * Y[k][j]<br>
slide7. Matrix Multiplication 7 for i = 1 to N do
for j = 1 to N do
for k = 1 to N do
Z[i][j] += X[i][k] * Y[k][j] par_for i = 1 to N do
par_for j = 1 to N do
for k = 1 to N do
Z[i][j] += X[i][k] * Y[k][j] par_for i = 1 to N do
par_for j = 1 to N do
par_for k = 1 to N do
temp[k] = X[i][k] * Y[k][j];
Z[i][j] = parallel_reduce(temp, N);<br>
slide8. Recursive Matrix Multiplication X11 Y11 Z11 := Z12 Z21 Z22 X12 X21 X22 Y21 Y12 Y22 Compute 8 submatrix products recursively
Z11 := X11Y11 + X12Y21
Z12 := X11Y12 + X12Y22
Z21 := X21Y11 + X22Y21
Z22 := X21Y12 + X22Y21<br>
slide9. Matrix Multiplication 9<br>
slide10. How to solve a recurrence in general? 10<br>
slide11. Solving recurrences – Master Theorem 11<br>
slide12. 12 … … … …<br>
slide13. 13 … …<br>
slide14. 14 … …<br>
slide15. 15 …<br>
slide16. 16 … … … You can always analyze a specific algorithm using the tree …<br>
slide17. Master Theorem 17<br>
slide18. Matrix multiplication 18<br>
slide19. Master Theorem Quiz 19<br>
slide20. Use recurrence to compute work and depth 20 S1;
S2; In parallel:
S1;
S2; Case 1:
S = execute S2 after finishing S1 Case 2:
S = execute S2 and S1 in parallel<br>
slide21. Flatten algorithm 21<br>
slide22. Flatten Algorithm 22<br>
slide23. Flatten algorithm 23 Flatten(A, n) {
parallel_for (i = 0 to n) S[i] = |A[i]|;
offset = scan_exclusive(S, n);
parallel_for (i = 0 to n) {
off = offset[i];
parallel_for (j = 0 to S[i])
B[off+j]=A[i][j];
}
return B;} Element b2 is at location 3+2=5 Element e1 is at location 13+1=14<br>
slide24. Constructing a Tableau 24<br>
slide25. Edit Distance: Dynamic Programming Given two strings A and B, the edit distance of A and B is the least number of edits (inserts and deletes) to modify A to B
S[i][j] is the edit distance of A[1..i] and B[1..j]
S[i][j] = min of
S[i-1][j-1] if A[i]==B[j]: convert A[1..i-1] to B[1..j-1] and keep A[i] since A[i]=B[j]
S[i-1][j]+1: convert A[1..i-1] to B[1..j] and delete A[i]
S[i][j-1]+1: convert A[1..i] to B[1..j-1] and insert B[j] at the end atcacac acac Case 1: just convert atcaca to aca (if A[i]==B[j])
S[i][j]=S[i-1][j-1] A[1..i]= B[1..j]= Case 2: convert atcaca to acac, and delete the last “c” in A
S[i][j]=S[i-1][j]+1 atcacac acac A[1..i]= B[1..j]= atcacac acac A[1..i]= B[1..j]= Case 3: convert atcacac to aca, and insert a “c” at the end of A
S[i][j]=S[i][j-1]+1 Take the minimum of them acab atcacac b<br>
slide26. Edit Distance: Dynamic Programming S[i][j] = min of
S[i-1][j-1] if A[i]==B[j]
S[i-1][j]+1
S[i][j-1]+1 A B insert delete common Note: can be filled in any order as long as the cells to the left and above are filled.
Can follow path back through matrix to construct edits.<br>
slide27. Page27 Edit Distance: Dynamic Programming Sequential code:
for i = 1 to n
M[i,1] = i;
for j = 1 to m
M[1,j] = j;
for i = 2 to n
for j = 2 to m
if (A[i] == B[j])
M[i,j] = M[i-1,j-1];
else
M[i,j] = 1 + min(M[i-1,j],M[i,j-1]); 15-853<br>
slide28. Constructing a Tableau 28 Dynamic Programming
Longest common subsequence
Edit distance
……<br>
slide29. Constructing a Tableau 29<br>
slide30. Constructing a Tableau 30<br>
slide31. Constructing a Tableau: 4-way divide-and-conquer 31<br>
slide32. Constructing a Tableau 32<br>
slide33. Constructing a Tableau 33<br>
slide34. Constructing a Tableau: 9-way divide-and-conquer 34<br>
slide35. Filtering/packing 35<br>
slide36. Parallel filtering / packing 36<br>
slide37. Parallel filtering / packing 37<br>