Implicit User Feedback Hongning
5K - views

Implicit User Feedback Hongning

Similar presentations


Download Presentation

Implicit User Feedback Hongning




Download Presentation - The PPT/PDF document "Implicit User Feedback Hongning" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.



Presentation on theme: "Implicit User Feedback Hongning"— Presentation transcript:

Slide1

Implicit User Feedback

Hongning

Wang

CS@UVa

Slide2

Explicit relevance feedback

2

Updated

query

Feedback

Judgments:

d

1

+

d

2

-

d

3 +

d

k -...

Query

User

judgment

Retrieval

Engine

Document

collection

Results:

d

1

3.5

d

2 2.4…dk 0.5...

CS@UVa

CS 4501: Information Retrieval

Slide3

Relevance feedback in real systems

Google used to provide such functions

Vulnerable to spammers

RelevantNonrelevant

CS@UVa

CS 4501: Information Retrieval

3

Slide4

How about using clicks

Clicked document as relevant, non-clicked as non-relevant

Cheap, largely available

CS@UVaCS 4501: Information Retrieval4

Slide5

Recap: feedback as model interpolation

Query Q

Document D

Results

Feedback Docs

F={d

1

, d

2

, …, d

n

}

Generative model

=0

No feedback

=1

Full feedback

Q

:

Rocchio

feedback

in vector space model?

A: Very similar, but with different interpretations.

Key: estimate the feedback model

CS@UVa

CS4501: Information Retrieval

5

Slide6

Recap: how to estimate

F

?

the 0.2

a 0.1

we 0.01to 0.02…flight 0.0001company 0.00005 …

Known

Background

p(

w|C)

accident =?

regulation =? passenger=?

rules =?

Unknown

query topic

p(w|

F

)=?

“airport security”

=0.7

=0.3

Feedback

Doc(s)

Suppose,

we know

the identity of each

word

ML

Estimator

fixed

; but we don’t...

CS@UVa

CS4501: Information Retrieval

6

Slide7

Recap: Expectation Maximization algorithm

Identity (“hidden”) variable:

z

i

{1 (background), 0(topic)}

thepaperpresentsatext

miningalgorithmthepaper...

z

i

1

1

1100010...

Suppose the parameters are all known, what’s a reasonable guess of

z

i

?

- depends on

 (why?)

- depends on p(w|C) and p(w|F) (how?)

E-step

M-step

Why in

Rocchio

we did not distinguish a word’s identity?

CS@UVa

CS4501: Information Retrieval7

Slide8

Is click reliable?

Why do we click on the returned document?

Title/snippet looks attractive

We haven’t read the full text content of the documentIt was ranked higherBelief bias towards rankingWe know it is the answer!CS@UVaCS 4501: Information Retrieval8

Slide9

Is click reliable?

Why do not we click on the returned document?

Title/snippet has already provided the answer

Instant answers, knowledge graphExtra effort of scrolling down the result pageThe expected loss is larger than skipping the documentWe did not see it….Can we trust click as relevance feedback?CS@UVaCS 4501: Information Retrieval

9

Slide10

Accurately Interpreting

Clickthrough

Data as Implicit Feedback

[Joachims SIGIR’05]Eye tracking, click and manual relevance judgment to answerDo users scan the results from top to bottom?How many abstracts do they read before clicking?How does their behavior change, if search results are artificially manipulated?CS@UVaCS 4501: Information Retrieval10

Slide11

Which links do users view and click?

Positional bias

First 5 results are visible without scrolling

Fixations: a spatially stable gaze lasting for approximately 200-300 ms, indicating visual attentionCS@UVaCS 4501: Information Retrieval

11

Slide12

Do users scan links from top to bottom?

View the top two results within the second or third fixation

Need scroll down to view these results

CS@UVa

CS 4501: Information Retrieval

12

Slide13

Which links do users evaluate before clicking?

The lower the click in the ranking, the more abstracts are viewed before the click

CS@UVa

CS 4501: Information Retrieval13

Slide14

Does relevance influence user decisions?

Controlled relevance quality

Reverse the ranking from search engine

Users’ reactionsScan significantly more abstracts than beforeLess likely to click on the first resultAverage clicked rank position drops from 2.66 to 4.03Average clicks per query drops from 0.8 to 0.64CS@UVaCS 4501: Information Retrieval14

Slide15

Are clicks absolute relevance judgments?

Position bias

Focus on position one and two, equally likely to be viewed

CS@UVaCS 4501: Information Retrieval15

Slide16

Are clicks relative relevance judgments?

Clicks as

pairwise

preference statementsGiven a ranked list and user clicksClick > Skip AboveLast Click > Skip AboveClick > Earlier ClickLast Click > Skip PreviousClick > Skip Next(1)

(2)

(3)

CS@UVa

CS 4501: Information Retrieval

16

Slide17

Clicks as pairwise preference

statements

Accuracy against manual relevance judgment over abstract

CS@UVaCS 4501: Information Retrieval17

Slide18

How accurately do clicks correspond to explicit judgment of a document?

Accuracy against manual relevance judgment

CS@UVa

CS 4501: Information Retrieval18

Slide19

What do we get from this user study?

Clicks are influenced by the relevance of results

Biased by the trust over rank positions

Clicks as relative preference statement is more accurateSeveral heuristics to generate the preference pairsCS@UVaCS 4501: Information Retrieval19

Slide20

How to utilize such preference pairs?

Pairwise learning to rank algorithms

Will be covered later

CS@UVaCS 4501: Information Retrieval20

Slide21

Recap: Accurately

Interpreting

Clickthrough

Data as Implicit FeedbackEye tracking, click and manual relevance judgment to answerDo users scan the results from top to bottom?How many abstracts do they read before clicking?How does their behavior change, if search results are artificially manipulated?CS@UVaCS 4501: Information Retrieval21

Slide22

Recap: which

links do users view and click?

Positional bias

First 5 results are visible without scrollingFixations: a spatially stable gaze lasting for approximately 200-300 ms, indicating visual attentionCS@UVaCS 4501: Information Retrieval

22

Slide23

Recap: are

clicks relative relevance judgments?

Clicks as

pairwise preference statementsGiven a ranked list and user clicksClick > Skip AboveLast Click > Skip AboveClick > Earlier ClickLast Click > Skip PreviousClick > Skip Next(1)

(2)

(3)

CS@UVa

CS 4501: Information Retrieval

23

Slide24

Recap: clicks as pairwise preference

statements

Accuracy against manual relevance judgment over abstract

CS@UVaCS 4501: Information Retrieval24

Slide25

An eye tracking study of the effect of target rank on web

search

[Guan CHI’07]

Break down of users’ click accuracyNavigational searchCS@UVaCS 4501: Information Retrieval25First result

Slide26

An eye tracking study of the effect of target rank on web

search

[Guan CHI’07]

Break down of users’ click accuracyInformational searchCS@UVaCS 4501: Information Retrieval26First result

Slide27

Users failed to recognize the target because they did not read it!

Navigational search

CS@UVa

CS 4501: Information Retrieval27

Slide28

Users did not click because they did not read the results!

Informational search

CS@UVa

CS 4501: Information Retrieval28

Slide29

Predicting clicks

: e

stimating

the click-through rate for new ads [Richardson WWW’07]To maximize ad revenue

Position-bias is also true in online ads

Observed low CTR is not just because of ads’ quality, but also their display positions! Cost per click: basic business model in search engines

Estimated click-through rate

CS@UVa

CS 4501: Information Retrieval

29

Slide30

Combat position-bias by explicitly modeling it

Being clicked is related to its quality and position

 

 

Calibrated CTR for ads ranking

Discounting factor

Logistic regression by features of the ad

CS@UVa

CS 4501: Information Retrieval

30

Slide31

Parameter estimation

Discounting factor

Approximation: positions being clicked must be seen already

Calibrated CTRMaximum likelihood for with historic clicks

 

CS@UVa

CS 4501: Information Retrieval

31

Slide32

Calibrated CTR is more accurate for new ads

Simple counting of CTR

Unfortunately, their evaluation criterion is still

based on biased clicks in testing setCS@UVaCS 4501: Information Retrieval

32

Slide33

Click models

Decompose relevance-driven clicks from position-driven clicks

Examine: user reads the displayed result

Click: user clicks on the displayed resultAtomic unit: (query, doc)(q,d1)

(q,d

4

)(q,d3)(q,d2)

Prob.

Pos.

Click probability

CS@UVa

33

Examine probability

Relevance quality

CS 4501: Information Retrieval

Slide34

Cascade Model [

Craswell

et al. WSDM’08]

Sequential browsing assumptionAt each position decides whether to move on

Assuming

Only one click is allowed on each search result page

 

Kind

of “Click > Skip Above”?

CS@UVaCS 4501: Information Retrieval

34

Slide35

User Browsing Model

[

Dupret

et al. SIGIR’08]Examination depends on distance to the last click

From absolute discount to relative discount

CS@UVa

35Attractiveness, determined by query and URL

Examination, determined by position and distance to last click

EM for parameter estimation

Kind

of “Click

> Skip

Next” + “Click > Skip Above”?CS 4501: Information Retrieval

Slide36

More accurate prediction of clicks

Perplexity – randomness of prediction

Cascade model

Browsing modelCS@UVaCS 4501: Information Retrieval

36

Slide37

Dynamic Bayesian Model

[

Chapelle

et al. WWW’09]A cascade modelRelevance quality: Perceived relevance

User’s satisfaction

Examination chain

CS@UVa

37

Intrinsic relevance

CS 4501: Information Retrieval

Slide38

Accuracy in predicting CTR

CS@UVa

CS 4501: Information Retrieval

38

Slide39

Revisit User Click Behaviors

Match my query?

Redundant doc?

Shall I move on?

CS@UVa

39

CS 4501: Information Retrieval

Slide40

Content-Aware Click Modeling

[Wang et al. WWW’12]

Encode dependency within user browsing behaviors via descriptive features

Relevance quality of a document: e.g., ranking features

Chance to further examine the result documents:

e.g., position, # clicks, distance to last click

Chance to click on an examined and relevant document: e.g., clicked/skipped content similarityCS@UVa40

CS 4501: Information Retrieval

Slide41

Quality of relevance modeling

Estimated relevance for ranking

CS@UVa

41

CS 4501: Information Retrieval

Slide42

Understanding user behaviors

Analyzing factors affecting user clicks

CS@UVa

42

CS 4501: Information Retrieval

Slide43

What you should know

Clicks as implicit relevance feedback

Positional

biasHeuristics for generating pairwise preferencesAssumptions and modeling approaches for click modelsCS@UVaCS 4501: Information Retrieval43

Slide44

CS@UVa

CS 4501: Information Retrieval

44

Slide45