Christopher H Lin Mausam Daniel S Weld University of Washington AAAI2012 Toronto ON Canada 1 2 3 Translate A B 4 Translate A B HumanTr A B Choose HumanTr A B 5 Translate A ID: 644554
Download Presentation The PPT/PDF document "Dynamically Switching between Synergisti..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Dynamically Switching between Synergistic Workflows for Crowdsourcing
Christopher H. Lin, Mausam, Daniel S. WeldUniversity of WashingtonAAAI-2012, Toronto, ON, Canada
1Slide2
2Slide3
3
Translate ABSlide4
4
Translate AB
HumanTr
A
B
Choose
HumanTr
A
BSlide5
5
Translate AB
MachineTr
A
B
ProofRdSlide6
6
Translate AB
Translate
A
C
Translate
C
BSlide7
7
www.crowdsourcing.com
Is this a face?
Yes
NoSlide8
8
Congressman Tony Coelho landed a million - dollar job on Wall Street
after
resigning over a controversial junk - bond investment last
summer.
NER TaggingSlide9
9
Congressman Tony Coelho landed a million - dollar job on Wall Street
after
resigning over a controversial junk - bond investment last
summer.
NER Tagging
locationSlide10
10
www.crowdsourcing.com
location
film
Congressman
Tony Coelho
landed
a million - dollar job on
Wall Street
after
resigning over a controversial junk - bond investment last
summer. Slide11
11
www.crowdsourcing.com
Wall Street
http://en.wikipedia.org/wiki/Wall_Street
Wall Street
is the financial district of New York City
,
named after and centered on the eight-block-long street running from Broadway to South
Street
.
…
Wall Street (1987 film)
http://en.wikipedia.org/wiki/Wall_Street _(1987_film)
Wall Street
is a 1987 American drama film released by 20th Century Fox. It was directed by Oliver Stone and stars Michael Douglas, Charlie Sheen, and Daryl
Hannah…
Congressman
Tony Coelho
landed
a million - dollar job on
Wall Street
after
resigning over a controversial junk - bond investment last
summer. Slide12
12
7
0%
90%
A/B Testing AccuracySlide13
13
7
0%
90%
A/B Testing AccuracySlide14
14
www.crowdsourcing.com
us_county
l
ocation
citytown
location
Washington State
Washington D.CSlide15
15
www.crowdsourcing.com
us_county
l
ocation
citytown
location
Washington State
Washington D.CSlide16
16
www.crowdsourcing.com
us_county
l
ocation
citytown
location
Washington State
Washington D.CSlide17
17
N
Create job using best workflow
[Dai
et al,
2010]Slide18
18
N
Create job using best workflow
[Dai
et al,
2010]
suboptimalSlide19
19
N
Create job using best workflowSlide20
20
www.crowdsourcing.com
us_county
l
ocation
citytown
location
Washington State
Washington D.CSlide21
21
www.crowdsourcing.com
Washington
http://en.wikipedia.org/wiki/Washington
Washington, D.C., formally the District of Columbia and commonly referred to as Washington, “the District,” or simply D.C., is the capital of the United States….
Washington(state)
http://en.wikipedia.org/wiki/Washington_(state)
Washington () is a state in the Pacific Northwest region of the United States located
north of Oregon, west of Idaho, and south of the Canadian province of British Columbia, on the coast of the Pacific Ocean….
Washington State
Washington D.CSlide22
22
www.crowdsourcing.com
Washington
http://en.wikipedia.org/wiki/Washington
Washington, D.C., formally the District of Columbia and commonly referred to as Washington, “the District,” or simply D.C., is the capital of the United States….
Washington(state)
http://en.wikipedia.org/wiki/Washington_(state)
Washington () is a state in the Pacific Northwest region of the United States located
north of Oregon, west of Idaho, and south of the Canadian province of British Columbia, on the coast of the Pacific Ocean….
Washington State
Washington D.CSlide23
23
www.crowdsourcing.com
Washington
http://en.wikipedia.org/wiki/Washington
Washington, D.C., formally the District of Columbia and commonly referred to as Washington, “the District,” or simply D.C., is the capital of the United States….
Washington(state)
http://en.wikipedia.org/wiki/Washington_(state)
Washington () is a state in the Pacific Northwest region of the United States located
north of Oregon, west of Idaho, and south of the Canadian province of British Columbia, on the coast of the Pacific Ocean….
Washington State
Washington D.CSlide24
24
N
Create job using best workflow
[Dai
et al,
2010]
BeforeSlide25
25
AgentHuntSlide26
26
AgentHunt
DynamicallySlide27
27
POMDP
States
Actions
Transition Function
Observation ProbabilitiesSlide28
28
POMDP
States
Actions
Transition Function
Observation ProbabilitiesSlide29
29Slide30
30
2 + 2 = ?
www.crowdsourcing.comSlide31
31
2 + 2 = ?
www.crowdsourcing.com
EASY
d
= 0Slide32
32
www.crowdsourcing.com
?
=Slide33
33
www.crowdsourcing.com
?
=
HARD
d
= 1Slide34
34
Correct Answer
Workflow DifficultiesSlide35
35
POMDP
States
Actions
Transition Function
Observation ProbabilitiesSlide36
36
K+1 actionsSlide37
37
POMDP
States
Actions
Transition Function
Observation ProbabilitiesSlide38
38
POMDP
States
Actions
Transition Function
Observation ProbabilitiesSlide39
39Slide40
Difficulty
40Slide41
Worker Ability
41
DifficultySlide42
Source:Wikipedia
42Slide43
[Figure courtesy
Peng
Dai]
43
4.0
2.0
1.0
0.5
0.25Slide44
44Slide45
Learn New Parameters
45Slide46
[Dai et al,
2010]
46
AgentHunt
beats
SingleBestWorkflowSlide47
[Dai et al,
2010]
47
AgentHunt
beats
SingleBestWorkflowSlide48
AgentHunt
Single
Best
Workflow
Average
Accuracy (%)
92.45
85.85
AverageNet Utility
-13.36
-18.35
48
AgentHunt
beats
SingleBestWorkflowSlide49
Contributions
A probabilistic model for multiple workflowsA decision-theoretic agent that dynamically switches between workflowsReal experiments that show you should not throw away your workflows
49Slide50
INTERESTED?
50
http://cs.washington.edu/homes/chrislinSlide51
Repeated Labeling
Romney et al, 1986; Sheng et al, 2008; Raykar et al, 2010; Whitehill et al, 2009
;
Dai et al, 2010, 2011;
Kamar
et al, 2012;
Parameswaran et al, 2010;
Karger et al, 2011; Snow et al, 2008;
Complicated WorkflowsLittle et al, 2009; Bernstein et al, 2010; Liem et al, 2011; Noronha et al, 2011; Lasecki et al, 2011;
Kulkarni et al, 2012;
AI Control of Workflows
Dai et al, 2010,2011;
Shahaf
et al, 2010; Weld et al, 2011;
Kamar
et al, 2012;
51Slide52
52
How many people are there in this military base?
www.crowdsourcing.com
Source:WikipediaSlide53
Christopher H. Lin, Mausam
, and Daniel S. Weld. “Crowdsourcing Control: Moving Beyond Multiple Choice.”
In UAI 2012
53