Assembly Line for the Information Age HumanComputer Cooperation for LargeScale Product Classification Jianfu Chen Computer Science Department Stony Brook University Machines Transform ID: 167661
Download Presentation The PPT/PDF document "The “" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
The “Assembly Line” for the Information Age
Human-Computer Cooperation for Large-Scale Product Classification
Jianfu
Chen
Computer
Science Department,
Stony
Brook UniversitySlide2
Machines Transform Human HistorySlide3
People have always been seeking the optimal way of integrating machine and human labor.Slide4
20th Century
Ford Assembly Line
Integrates Machine and Human Labor EfficientlySlide5
21st Century – Information Age
“Mass Production” of InformationSlide6
We want to find the optimal ways to integrate machine and human intelligence.NOT
all products could be produced fully automatically by machinesassembly line integrated machine & human laborNOT all information can be produced fully automatically by computersWe want to find optimal ways to integrate machine and human intelligence
What’s the “
Assembly Line
” for the Information Age?Slide7
A Case Study: Large scale product classification
Kindle Fire HD 8.9" 4G LTE Wireless8.9" HD Display, Dolby Audio, Dual-Band Dual-Antenna Wi-Fi, 4G LTE, 32GB or 64GB
Goal:
optimally integrate computer and human effort
Achieve a lower unit cost for product classification
More precisely, optimize the
accuracy-cost
tradeoffSlide8
An “Assembly Line” for
Human Computer Cooperation
3Com V.35 cable
V.35
cable ( DTE ) - DB-50 (M) - M/34 (V.35) (M) - 10 ft
26121609
A list of
K
candidate classes
System
Accuracy
Machine Accuracy
Human Accuracy
X
Cost
is Human labor cost, i.e., the salary paid to workers, which is proportional to the
working time
spent.
=Slide9
A quick glance at Accuracy-Cost Relation
Assume K determines the Accuracy and Cost.System Accuracy
Machine Accuracy
increases
as K increases
Human
Accuracy
decreases
as K increases
Cost increases as K increases
Slide10
A quick glance at Accuracy-Cost RelationSlide11
There is an optimal cost that gives the highest accuracy.Slide12
Towards a more realistic analysis of accuracy-cost relationship
With the above “assembly line” model, human accuracy and working time are influenced by a set of factorsKTask difficultyExpertiseI am familiar with
office supplies
, but not familiar with
nuts and bolts.
Cognitive characteristics
Careful, smart, quick
Independent of the taskSlide13
Use a probabilistic graphical model to capture the cognitive process of human classification
A probabilistic graphical model shows how the above different factors interact with each other, and influence the accuracy and cost.Specifically, we use Bayesian Network, which characterizes the causal relationships of different factors.Slide14
Use a Bayesian Network to predict accuracy and cost
Slide15
Not only visually intuitive, but also formal
Slide16
Inference and learningwith this Bayesian Network, we predict
the accuracy and cost by
Training data
A set of examples with
known class
labels
We let each human worker work on multiple examples, record the correctness and the working time
EM
algorithm
learns
the parameters and the hidden variables
Slide17
usage of the modelPredict the accuracy-cost tradeoffGiven certain
budget, what’s the highest accuracy we can achieve?To achieve certain accuracy, what’s the lowest expected
cost
?
How to charge customers?
Optimally assign the workers to the tasksSlide18
Related Workstime and motion studyScientific management (Taylorism)
CrowdsourcingAmazon Mechanical Turklearning worker expertise and accuracyItem Response TheoryPsychometricsIQ test, GRE, GMATSlide19
ConclusionIn information age, we need a new “assembly line” to integrate human and machine intelligence.
We try to model human accuracy and working time by considering the interactions of a set of relevant factors, using a probabilistic graphical model
.
We use the model to
predict the
accuracy-cost
tradeoff
, decide how to charge customers, and optimally assign tasks to human workers.Slide20
Thank you!