CrowdWorker Reliability in Image Quality Assessment Hybrid HumanMachine Computing 2017 Vlad Hosu and Dietmar Saupe Image quality assessment Domainexperts are more reliable How do we find them in the crowd ID: 799538
Download The PPT/PDF document "Induced Degradations for" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Induced Degradations for Crowd-Worker Reliability in Image Quality Assessment
Hybrid Human-Machine Computing 2017
Vlad Hosu and Dietmar Saupe
Slide2Image quality assessment
Slide3Domain-experts are more reliable How do we find them, in the crowd?
Slide4Proficiency
testsGeneral crowdExperts
Slide5Proficiency questions
main task,
not tested
Slide6High quality (no
degradation)
motion blurcolor fringingspatial jitter
under-exposure
g
rain / noise
JPEG artifacts
over-sharpening
over-saturation
artificial degradations
(imply a change in quality)
Slide7How well do crowd experts perform on an IQA task?
Slide8Freelance
professional
photographersCrowd workers
“ground truth”
Slide9Experiments
0.83
0.860.90
Direct testing
No testing
Proficiency testing
rank order correlations, crowd vs experts
Slide10How well does proficiency testing work?
10 freelancers
~0.9110 freelancers
≅
0.90
50 crowd-workers
proficienc
y tested
Slide11Conclusionstest on objective correlates of a subjective
quantitydegradations for technical image qualityharder to setup, has less downsides
Slide12Thank you!
Questions?