Evidence Signals and Tasks Vishwa Vinay Microsoft Research Cambridge Introduction Signals Explicit Vs Implicit Evidence Of what From where Used how Tasks Ranking Evaluation amp many more things search ID: 240174
Download Presentation The PPT/PDF document "Click" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Click Evidence Signals and Tasks
Vishwa Vinay
Microsoft Research, CambridgeSlide2
Introduction
Signals
Explicit
Vs
Implicit
Evidence
Of what?
From where?
Used how?
Tasks
Ranking, Evaluation & many more things searchSlide3
Clicks as Input Task
=
Relevance Ranking
Feature
in relevance ranking function
Signal
select
URL,
count
(*)
as
DocFeature
from
Historical_Clicks
group
by
URL
select
Query
,
URL,
count
(*)
as
QueryDocFeature
from
Historical_Clicks
group
by
Query
,
URLSlide4
Clicks as Input Feature
in relevance ranking function
Static feature (
popularity
)
Dynamic
feature (for
this
query-doc pair)
“Query Expansion using Associated Queries”, Billerbeck et al, CIKM 2003
“
Improving Web Search Ranking by Incorporating User Behaviour”,
Agichtein
et al, SIGIR
2006
‘Document Expansion’
Signal
bleeds to
similar
queriesSlide5
Clicks as Output
Task
=
Relevance Ranking
Result Page = Ranked list of documents
Ranked list = Documents sorted based on
Score
Score = Probability that this result will be
clicked
Signal
Did my prediction agree with the user’s action?
“Web-Scale Bayesian Click-through rate Prediction for Sponsored Search Advertising in Microsoft’s Bing Search Engine”, Graepel et al, ICML 2010Slide6
Clicks as Output Calibration: Merging results from different sources (comparable
scores
)
“Adaptation of Offline Vertical Selection Predictions in the Presence of User Feedback”, Diaz et al, SIGIR 2009
Onsite Adaptation of ranking function
“A Decision Theoretic Framework for Ranking using Implicit Feedback”,
Zoeter
et al, SIGIR 2008Slide7
Clicks for Training
Task
=
Learning a ranking function
Signal
Query=“Search Solutions 2010”
Absolute: Relevant={Doc1, Doc3},
NotRelevant
={Doc2}
Preferences: {Doc2
Doc1}, {Doc2
Doc3}
Rank 1
Doc1
http://irsg.bcs.org/SearchSolutions/2010/sse2010.phpRank 2Doc2http://irsg.bcs.org/SearchSolutions/2009/sse2009.phpRank 3Doc3http://isquared.wordpress.com/2010/10/10/search-solutions-2010-titles-and-abstracts/Slide8
Clicks for Training
Preferences from Query-> {URL, Click} events
Rank bias & Lock-in
Randomisation & Exploration
“Accurately Interpreting
Clickthrough
Data as Implicit Feedback”,
Joachims
et al, SIGIR 2005
Preference Observations into Relevance Labels
“Generating Labels from Clicks”,
Agrawal
et al, WSDM 2010Slide9
Clicks for Evaluation
Task
=
Evaluating a ranking function
Signal
Engagement and Usage metrics
Query=“Search Solutions 2010”
Controlled experiments for A/B Testing
Rank
Old Ranker
New (and Improved?)
1
http://irsg.bcs.org/SearchSolutions/2009/sse2009.php
http://irsg.bcs.org/SearchSolutions/2010/sse2010.php
2http://www.online-information.co.uk/online2010/trails/search-solutions.htmlhttp://isquared.wordpress.com/2010/10/10/search-solutions-2010-titles-and-abstracts/3http://irsg.bcs.org/SearchSolutions/2010/sse2010.phphttp://irsg.bcs.org/SearchSolutions/2009/sse2009.phpSlide10
Clicks for Evaluation
Disentangling relevance from other effects
“An experimental comparison of click position-bias models”,
Craswell
et al, WSDM 2008
Label-free evaluation of retrieval systems (‘Interleaving’)
“How Does
Clickthrough
Data Reflect Retrieval Quality?”, Radlinski et al, CIKM 2008Slide11
Personalisation with Clicks
Task
=
Separate out Individual preferences from aggregates
Signal
: {
User
, Query, URL, Click} tuples
Query=“Search Solutions 2010”
Rank
URL
Tony
Vinay
1
http://irsg.bcs.org/SearchSolutions/2010/sse2010.php
2http://isquared.wordpress.com/2010/10/10/search-solutions-2010-titles-and-abstracts/3http://irsg.bcs.org/SearchSolutions/2009/sse2009.phpSlide12
Personalisation with Clicks
Click event as a
rating
“Matchbox: Large Scale Bayesian Recommendations”, Stern et al, WWW 2009
Sparsity
- collapse
using user
groups (
groupisation
)
“Discovering and Using Groups to Improve Personalized Search”,
Teevan et al, WSDM 2009 - collapse using doc structureSlide13
Miscellaneous
Using co-clicking for query suggestions
“Random Walks on the Click Graph”,
Craswell
et al, SIGIR 2007
User behaviour models for
Ranked lists:
“Click chain model in Web Search”,
Guo
et al, WWW 2009
Whole page:
“Inferring Search
Behaviors
Using Partially Observable Markov Model”, Wang et al, WSDM 2010User activity away from the result page
“BrowseRank: Letting Web Users Vote for Page Importance”, Liu et al, SIGIR 2008Slide14
Additional Thoughts
Impressions & Examinations
Raw click counts versus normalised ratios
All clicks are not created equal
- Skip
Click
LastClick
OnlyClick
Query=“Search Solutions 2010”
Page / Rank
URL
Impression
Examination1 / 1http://irsg.bcs.org/SearchSolutions/2010/sse2010.php111 / 2
http://www.online-information.co.uk/online2010/trails/search-solutions.html
111 / 3
http://isquared.wordpress.com/2010/10/10/search-solutions-2010-titles-and-abstracts/
1
1
1
/ 4
http://irsg.bcs.org/SearchSolutions/2009/sse2009.php
1
0?
…
…
…
…
1 / 10
http://somesite.org/irrelevant.htm
1
0
2 / 1
http://someothersite.org/alsoirrelevant.htm
00Slide15
Clicks and Enterprise Search
Relying on the click signal
Machine learning and non-click features
Performance Out-Of-the-Box
Shipping a shrink-wrapped product
The self-aware adapting system
Good OOB
Gets better with use
Knows when things go wrongSlide16
Thank youvvinay@microsoft.com