Ramin Zabih Some slides from K Wayne Lecture 1 Dijkstras algorithm Administrivia Web site is httpsgithubcomcornelltechCS5112F18 As usual this is pretty much all you need to know ID: 757380
Download Presentation The PPT/PDF document "CS5112: Algorithms and Data Structures f..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
CS5112: Algorithms and Data Structures for Applications
Ramin ZabihSome slides from: K. Wayne
Lecture 1: Dijkstra’s algorithmSlide2
Administrivia
Web site is: https://github.com/cornelltech/CS5112-F18As usual, this is pretty much all you need to knowCourse staff:Instructors: Ramin Zabih (rdz@cs.cornell.edu) & Greg Zecchini (
gez3@cornell.edu)TA: Richard Bowen (rsb349@cornell.edu), TBAConsultants/graders: Iris Zhang (
wz337@cornell.edu
), TBASlide3
Basic information
CS5112 work is constant but not very time intensiveHomework every 2 weeks or so, quiz every week1 prelim 10/25 and final 12/4, both in-class closed bookOpen book doesn’t actually help in my experience
Greg will teach the 5 evening clinics 6:30pm-8pmGreg and Ramin will lecture, with a few guest lecturesWe are working on getting consultants to help students who don’t have a lot of programming experienceSlide4
Academic integrity
Each student is expected to abide by the Cornell University Code of Academic Integrityhttps://theuniversityfaculty.cornell.edu/academic-integrity/ Any work submitted by a student in this course for academic credit will be the student's own work
Exception: you do the homework assignments in groups of twoWe take this seriously. Students have been expelled from Cornell for violations. Copying code is easy to catch.Slide5
TodayClinic this evening (here!)
HW1 out tonight, due in 2 weeksBased primarily on today’s lecture and clinicFirst quiz a week from todayPlacement exam available tonight, due in 24 hoursIf you were a CS major, possibly worth your timeSlide6
Clinics schedule
Evening sessions to review some of the fundamentals underlying much of this course's contentAttendance is mandatorySchedule (dates final, topics tentative):
8/23 Graphs and Graph Algorithms8/30 Hashing and Related Data Structures9/6 Sorting and Searching9/20 Development tools (UNIX commands, Github
, etc.)
10/4 Cloud DevelopmentSlide7
Course theme: algorithms and applications
Algorithms are the key tool in CS, but without applications it’s hard to appreciate their importanceWe will focus on 3 key application areas:CryptocurrencyAI (artificial intelligence)
AR/VR (augmented/virtual reality)BUT this is not a course about those applicationsApplication of algorithms is often not obvious!Slide8
Lecture Outline
The shortest path problemDijkstra’s algorithmApplications: image editing and pirate grammarSlide9
Two very common approaches in CS
Given a problem where you are searching for a solution:Try everything (exhaustive search)Do what seems best at the moment, repeatedly (greedy algorithms)Exhaustive search (almost) never works on serious problems
Greedy algorithms are widely usedCurrently famous example: SGD for neural networksNote: there are other approaches we will coverSuch as dynamic programmingSlide10
The shortest path problem
General version: given a graph with edge weights, a starting node
and a target , find shortest path from
to
Claim: this problem is impossible to solve!
Slide11
Obvious application of shortest paths: airfare
Nodes are cities, edges are direct flights, weights are airfareWhat is the cheapest way to get from LGA to Ithaca?Presumably you can charter a planeSlide12
Fixing the problem definition
Suppose that there is a flight from Boise to El Paso, and back again, that the airline pays you $1 to fly aroundFurther, suppose that you can get to Boise (or El Paso)You can make an arbitrary amount of money by just flying back and forth!This is a cycle in the graph whose sum of weights is negativeEasy solution: require positive edge weightsOr maybe detect negative cycles?Slide13
Not so obvious applications
Making fake photographsSpeech recognition/predicting stock prices by DTWPirate grammar!Modeling a Cornell student (at end of class)Slide14
Making fake photographs
Sneak preview: will cover this in the AR/VR section of CS5112How do we create images like this:
Given an image, how do you cut out an object from it?You don’t want to manually select the pixelsSlide15
Intelligent scissors
Idea: shortest pathsE.N. Mortensen and W.A. Barrett, Interactive Segmentation with Intelligent Scissors, SIGGRAPH 1995Adobe calls this the “Magnetic Lasso”Video
hereMore details in November!Slide16
Dynamic Time Warping (DTW)Slide17
Rules of Pirate grammar
Pirates always start their sentences with “Barkeep!”90% of the time they next say “More” (i.e., they order)10% of the time they next say “Yer
a” (i.e., they insult)If they say “More”, they next say:60% “Of your best”40% “Of the same”Lots more rules, discovered by experts in pirate linguistics
Question: what sentence is a pirate most likely to say? Slide18
Pirate grammar as a graph
“Barkeep!”
“More”
“
Yer
a”
“of your best”
“of the same”
“scurvy”
“grog!”
“grub.”
“chicken.”
“wimpy”
.9
.1
.6
.4Slide19
Simplified pirate grammar
“Barkeep!”
“More”
“
Yer
a”
“of your best grog!”
“of the same wimpy grub.”
“scurvy chicken.”
.9
.1
.6
.4
1Slide20
How to make this into shortest paths?
On the surface this is not at all obviousWhich is why this is worth thinking about carefully
What we actually need to determine is the probability of any individual sentenceExample: “Barkeep! More of your best grog!” = .9 * .6 = .45So we look at all paths from the root to a leaf nodeEach edge has a probability
Multiply these together and find the max
This looks like “find the path where the product of the edges is maximized”, not “find the shortest path from
to
”
Slide21
Easy part: Add a fake source and sink
Red links have probability 1Now we need to find the “highest product path” from
to
“Barkeep!”
“More”
“
Yer
a”
“of your best grog!”
“of the same wimpy grub.”
“scurvy chicken.”
.9
.1
.6
.4
1
Slide22
Algebra to the rescue
We want to maximize the product of edge probabilitiesWhich are numbers between 0 and 1Instead we need to minimize the sum of edge weights
We know that log is monotonic, and
Maximize the product of edge probabilities = maximize the sum of log probabilities
Which are negative:
Maximizing anything is the same as minimizing its negative
Slide23
Algebra in action
“Barkeep!”
“More”
“
Yer
a”
“of your best grog!”
“of the same wimpy grub.”
“scurvy chicken.”
.9
.1
.6
.4
1
Slide24
Key property of shortest paths
Suppose the shortest path from to
goes via
I.e.,
Otherwise, we would take that “shortcut” instead, and create an even shorter path
Considering
paths, only need shortest
path
Don’t need to try everything!
shortest s-v path Slide25
Shortest paths by increasing budgets
Here is the basic idea, which we will simply speed upWhere can you fly from LGA on a $1 budget? Does that get you to Ithaca?If it does, you are doneIf not, add $1 to your budget and do it againYou can think of this as expanding a ball around
until you eventually get to
Though we are doing this on a graph
Slide26
Example
For $1 can get to
For $2 can also get to
Gray area shows budget at $2
At $3 we can also get to
via
Key concepts:
Explored nodes:
Fringe:
1
2
4
2
1
2
2
3
3Slide27
Key concepts
Explored nodes: we know the cheapest way to get thereShown as inside the gray zoneFringe nodes: adjacent to an explored nodeWhen we increase the budget we add a fringe node into the set of explored nodesThis is pretty inefficient, hold that thoughtKeep on doing this until
(i.e. Ithaca) is in the explored nodes
Slide28
Budget approach is crazy
Suppose the cheapest flight from LGA is $500In our example, imagine increasing by $.01So we consider $2.01, $2.02, …But we know that nothing will happen until we increase our budget to $3Why not just do this directly?Slide29
Dijkstra’s algorithm
We maintain an explored set with an invariant:
For each
hold the
shortest
path from
to
, write this as
Both the distance and the actual path, see HW1
Easiest to just think about the distance
Add an unexplored node
to
But, which one to choose?
On the fringe of
, so we add just one edge
Slide30
Choice of edge for a fringe node
The fringe node
can be adjacent to several nodes in
If we choose to add
, pick the right node in
to connect it to
s
v
u1
d(u1)
S
w
1
u2
w
2
d(u2)
d(u1) +
w
1
versus
d(u2) +
w
2Slide31
Choice of fringe node
If we pick to add to
, we will connect it to the in
that minimizes
+ the length of the
edge
Call this shortest path length
(v)
Think of this as “cheapest way to add
to
”
But can we pick an arbitrary
to add?
Can prove that this would break our invariant about
!
Pick
with smallest
(v)
, then add it to
with
=
(v)
Slide32
Shortest path example
Cost of path s-2-3-5-t
= 9 + 23 + 2 + 16 = 48.
s
3
t
2
6
7
4
5
23
18
2
9
14
15
5
30
20
44
16
11
6
19
6Slide33
HW1 algorithm
Start with
all other nodes in
, else
(i.e. upper bound)
Pick
on fringe of
that minimizes
I.e., the
with a neighbor in
that is cheapest to add to
On recursive call, we will have
is now in
, and no longer in
Done when we pick target
Computes more than shortest
path!
Slide34
Dijkstra's Shortest Path Algorithm
Find shortest path from s to t.Blue edges: shortest path to a node within S.Green edges: what we would add for each fringe vertex.
s
3
t
2
6
7
4
5
24
18
2
9
14
15
5
30
20
44
16
11
6
19
6Slide35
s
3
t
2
6
7
4
5
24
18
2
9
14
15
5
30
20
44
16
11
6
19
6
0
S = { s }
Q = { 2, 3, 4, 5, 6, 7, t }Slide36
s
3
t
2
6
7
4
5
24
18
2
9
14
15
5
30
20
44
16
11
6
19
6
15
9
14
0
(7)
S = { s }
Q = { 2, 3, 4, 5, 6, 7, t }
(2)
X
X
XSlide37
s
3
t
2
6
7
4
5
24
18
2
9
14
15
5
30
20
44
16
11
6
19
6
15
9
14
0
S = { s }
Q = { 2, 3, 4, 5, 6, 7, t }
X
X
X
minSlide38
s
3
t
2
6
7
4
5
24
18
2
9
14
15
5
30
20
44
16
11
6
19
6
15
9
14
0
S = { s, 2 }
Q = { 3, 4, 5, 6, 7, t }
X
X
XSlide39
s
3
t
2
6
7
4
5
24
18
2
9
14
15
5
30
20
44
16
11
6
19
6
15
9
14
0
S = { s, 2 }
Q = { 3, 4, 5, 6, 7, t }
X
X
X
(3)
X
33Slide40
s
3
t
2
6
7
4
5
24
18
2
9
14
15
5
30
20
44
16
11
6
19
6
15
9
14
0
S = { s, 2 }
Q = { 3, 4, 5, 6, 7, t }
X
X
X
X
33
minSlide41
s
3
t
2
6
7
4
5
24
18
2
9
14
15
5
30
20
44
16
11
6
19
6
15
9
14
0
S = { s, 2, 6 }
Q = { 3, 4, 5, 7, t }
X
X
X
X
33
44
X
X
32Slide42
s
3
t
2
6
7
4
5
24
18
2
9
14
15
5
30
20
44
16
11
6
19
6
15
9
14
0
S = { s, 2, 6 }
Q = { 3, 4, 5, 7, t }
X
X
X
X
33
44
X
X
32
minSlide43
s
3
t
2
6
7
4
5
18
2
9
14
15
5
30
20
44
16
11
6
19
6
15
9
14
0
S = { s, 2, 6, 7 }
Q = { 3, 4, 5, t }
X
X
X
44
X
35
X
59
X
24
X
33
X
32Slide44
s
3
t
2
6
7
4
5
18
2
9
14
15
5
30
20
44
16
11
6
19
6
15
9
14
0
S = { s, 2, 6, 7 }
Q = { 3, 4, 5, t }
X
X
X
44
X
35
X
59
X
24
X
33
X
32
minSlide45
s
3
t
2
6
7
4
5
24
18
2
9
14
15
5
30
20
44
16
11
6
19
6
15
9
14
0
S = { s, 2, 3, 6, 7 }
Q = { 4, 5, t }
X
X
X
44
X
35
X
59
X
X
51
X
34
X
33
X
32Slide46
s
3
t
2
6
7
4
5
18
2
9
14
15
5
30
20
44
16
11
6
19
6
15
9
14
0
S = { s, 2, 3, 6, 7 }
Q = { 4, 5, t }
X
X
X
44
X
35
X
59
X
X
51
X
34
min
X
33
X
32
24Slide47
s
3
t
2
6
7
4
5
18
2
9
14
15
5
30
20
44
16
11
6
19
6
15
9
14
0
S = { s, 2, 3, 5, 6, 7 }
Q = { 4, t }
X
X
X
44
X
35
X
59
X
X
51
X
34
24
X
50
X
45
X
33
X
32Slide48
s
3
t
2
6
7
4
5
18
2
9
14
15
5
30
20
44
16
11
6
19
6
15
9
14
0
S = { s, 2, 3, 5, 6, 7 }
Q = { 4, t }
X
X
X
44
X
35
X
59
X
X
51
X
34
24
X
50
X
45
min
X
33
X
32Slide49
s
3
t
2
6
7
4
5
18
2
9
14
15
5
30
20
44
16
11
6
19
6
15
9
14
0
S = { s, 2, 3, 4, 5, 6, 7 }
Q = { t }
X
X
X
44
X
35
X
59
X
X
51
X
34
24
X
50
X
45
X
33
X
32Slide50
s
3
t
2
6
7
4
5
18
2
9
14
15
5
30
20
44
16
11
6
19
6
15
9
14
0
S = { s, 2, 3, 4, 5, 6, 7 }
Q = { t }
X
X
X
44
X
35
X
59
X
X
51
X
34
X
50
X
45
min
X
33
X
32
24Slide51
s
3
t
2
6
7
4
5
24
18
2
9
14
15
5
30
20
44
16
11
6
19
6
15
9
14
0
S = { s, 2, 3, 4, 5, 6, 7, t }
Q = { }
X
X
X
44
X
35
X
59
X
X
51
X
34
X
50
X
45
X
33
X
32Slide52
s
3
t
2
6
7
4
5
24
18
2
9
14
15
5
30
20
44
16
11
6
19
6
15
9
14
0
S = { s, 2, 3, 4, 5, 6, 7, t }
Q = { }
X
X
X
44
X
35
X
59
X
X
51
X
34
X
50
X
45
X
33
X
32Slide53
Implementation notes
There are many ways to speed this up in practiceGraph representationsYou will explore 2 of these in HW1Naïve Dijkstra with nodes and
edges is
We need to remove from
the smallest node
with smallest value of
Priority queue implements remove-min in
This makes Dijkstra run in
time
Slide54
Another class of examples
Let’s model student behavior over time (hourly basis)Students have 3 possible states:Awake (A)Sleeping (S)Doing CS5112 homework (H)If you know their state at time
you know the probability of their other states at time
Example: A goes to A (.5), S (.49), H(.01)
Slide55
Trellis graph
We want to find the most likely 12 hour day for a studentAt every time there are 3 nodes, for A/S/HThere are edges with transition probabilities
Just like pirate grammar!So a day is a 12-node path through the graphThis is closely related to a “Hidden Markov Model”Widely used! Famous examples include speech, handwriting, computer vision, bioinformatics, etc.
Slide56
Example
Important note: with
states and time there are
nodes in the graph and
edges
So running time of naïve Dijkstra is
Can reduce this to
with dynamic programming (Viterbi)
.5
.49
.01