With People Jaime Teevan Microsoft Research jteevan Microsoft Kevyn CollinsThompson Susan Dumais Eric Horvitz Adam Kalai Ece Kamar Dan Liebling Merrie Morris Ryen White ID: 560232
Download Presentation The PPT/PDF document "Slow Search" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Slow SearchWith People
Jaime Teevan, Microsoft Research, @jteevanMicrosoft: Kevyn Collins-Thompson, Susan Dumais, Eric Horvitz, Adam Kalai, Ece Kamar, Dan Liebling, Merrie Morris, Ryen WhiteCollaborators: Michael Bernstein, Jin-Woo Jeong, Yubin Kim, Walter Lasecki, Rob Miller, Peter Organisciak, Katrina PanovichSlide2
Slow MovementsSlide3
Speed Focus in Search ReasonableSlide4
Not All Searches Need to Be Fast
Long-term tasksLong search sessionsMulti-session searchesSocial searchQuestion askingTechnologically limitedMobile devicesLimited connectivitySearch from spaceSlide5
Making Use of Additional TimeSlide6
Crowdsourcing
Using human computation to improve searchSlide7
Replace Components with People
Search processUnderstand queryRetrieve Understand resultsMachines are good at operating at scalePeople are good at understanding
w
ith Kim, Collins-ThompsonSlide8
Understand Query: Query Expansion
Original query: hubble telescope achievementsAutomatically identify expansion terms:space, star, astronomy, galaxy, solar, astro, earth, astronomerBest expansion terms cover multiple aspects of the queryAsk crowd to relate expansion terms to a query termIdentify best expansion terms:a
stronomer
,
astronomy
,
star
space
star
astronomy
galaxy
solar
astro
earth
astronomer
hubble
1
1
2
1
0
0
0
1
telescope
1
2
200001achievements00000001
Slide9
Understand Results: Filtering
Remove irrelevant results from listAsk crowd workers to vote on relevanceExample: hubble telescope achievementsSlide10
People Are Not Good Components
Test corporaDifficult Web queriesTREC Web Track queriesQuery expansion generally ineffectiveQuery filteringImproves quality slightlyImproves robustnessNot worth the time and costNeed to use people in new waysSlide11
Understand Query: Identify Entities
Search engines do poorly with long, complex queries
Query:
Italian
restaurant in
Squirrel Hill
or
Greenfield with
a gluten-free menu and a fairly sophisticated
atmosphere
Crowd workers identify important attributes
Given list of potential attributes
Option add new attributes
Example: cuisine, location, special diet, atmosphere
Crowd workers match attributes to query
Attributes used to issue a structured search
w
ith Kim, Collins-ThompsonSlide12
Understand Results: Tabulate
Crowd workers used to tabulate search resultsGiven a query, result, attribute and valueDoes the result meet the attribute?Slide13
People Can Provide Rich Input
Test corpus: Complex restaurant queries to YelpQuery understanding improves resultsParticularly for ambiguous or unconventional attributesStrong preference for the tabulated resultsPeople who liked traditional results valued familiarityPeople asked for additional columns (e.g., star rating)Slide14
Create Answers from Search Results
Understand queryUse log analysis to expand query to related queriesAsk crowd if the query has an answerRetrieve: Identify a page with the answer via log analysisUnderstand results: Extract, format, and edit an answer
w
ith Bernstein, Dumais, Liebling, HorvitzSlide15
Create Answers to Social Queries
Understand query: Use crowd to identify questionsRetrieve: Crowd generates a responseUnderstand results: Vote on answers from crowd, friends
w
ith
Jeong
, Morris, LieblingSlide16
Pros & Cons of THe Crowd
Opportunities and challenges of crowdsourcing searchSlide17
Personalization with the Crowd
?
w
ith Organisciak, Kalai, Dumais, MillerSlide18
Matching Workers versus Guessing
Matching workersRequires many workers to find a good matchEasy for workersData reusableGuessingRequires fewer workersFun for workersHard to capture complex preferences
Rand.
Match
Guess
Salt shakers
1.64
1.43
1.07
Food (Boston)
1.51
1.19
1.38
Food (Seattle)
1.68
1.26
1.28
(RMSE for 5 workers)Slide19Slide20
Extraction and Manipulation Threats
with Lasecki, KamarSlide21
Information Extraction
Target task: Text recognitionAttack taskComplete target taskReturn answer from target:
1234 5678 9123 4567
1234 5678 9123 4567
62.1%
32.8%Slide22
gun (36%), fun (26%), sun (12%)
Task ManipulationTarget task: Text recognitionAttack taskEnter “sun” as the answer for the attack task
sun (75%)
s
un (28%)Slide23
Friendsourcing
Using friends as a resource during the search processSlide24
Searching versus AskingSlide25
Searching versus Asking
Friends respond quickly58% of questions answered by the end of searchAlmost all answered by the end of the daySome answers confirmed search findingsBut many provided new informationInformation not available onlineInformation not actively soughtSocial content
w
ith Morris, PanovichSlide26
Shaping the Replies from Friends
Should I watch E.T.?Slide27
Shaping the Replies from Friends
Larger networks provide better repliesFaster replies in the morning, more in the eveningQuestion phrasing importantInclude question markTarget the question at a group (even at anyone)Be brief (although context changes nature of replies)Early replies shape future repliesOpportunity for friends and algorithms to collaborate to find the best content
w
ith Morris, PanovichSlide28
SummarySlide29
Further Reading in Slow Search
Slow searchTeevan, J., Collins-Thompson, K., White, R., Dumais, S.T. & Kim, Y. Slow Search: Information Retrieval without Time Constraints. HCIR 2013.Teevan, J., Collins-Thompson, K., White, R. & Dumais, S.T. Slow Search. CACM 2014 (to appear).CrowdsourcingJeong, J.W., Morris, M.R., Teevan, J. & Liebling, D. A Crowd-Powered Socially Embedded Search Engine. ICWSM 2013.Bernstein, M., Teevan, J., Dumais, S.T., Libeling, D. & Horvitz, E. Direct Answers for Search Queries in the Long Tail. CHI 2012.
Pros and cons of the crowd
Lasecki, W., Teevan, J., & Kamar, E.
Information Extraction and Manipulation Threats in Crowd-Powered Systems
. CSCW 2014
.
Organisciak, P., Teevan, J., Dumais, S.T., Miller, R.C. & Kalai, A.T.
Personalized Human Computation
. HCOMP 2013.
Friendsourcing
M.R. Morris, J. Teevan & K. Panovich.
A Comparison of Information Seeking Using Search Engines and Social Networks. ICWSM 2010
.
J
. Teevan, M.R. Morris & K. Panovich.
Factors Affecting Response Quantity, Quality and Speed in Questions Asked via Online Social Networks
. ICWSM 2011.Slide30
Questions?
Slow Search with PeopleJaime Teevan, Microsoft Research, @jteevan