ContextBased Information Retrieval Michael Kandefer and Stuart C Shapiro University at Buffalo Department of Computer Science and Engineering Center for Multisource Information Fusion Center for Cognitive Science ID: 381141
Download Presentation The PPT/PDF document "An F-Measure for" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
An F-Measure for Context-Based Information Retrieval
Michael Kandefer and Stuart C. ShapiroUniversity at BuffaloDepartment of Computer Science and EngineeringCenter for Multisource Information FusionCenter for Cognitive Science{mwk3,shapiro}@cse.buffalo.eduSlide2
IntroductionCommonsense 2009
One of the major long-term goals of AI is to endow computers with common senseOne challenge is the accumulation of large amounts of knowledge about our everyday worldManaging a large-scale knowledge store is necessary Slide3
Introduction
Building commonsense reasoners requires access to large amounts of informationDeductive reasoners suffer performance issues when working with large KBsOptimal solution:Use only that information that is needed for reasoningConsidered the relevant informationNot practical
Can take as long as reasoningSlide4
Introduction
Solution: Context-based Information Retrieval Use context to help establish information that is likely to be relevantThe environment and other constraintsNot the KR sense of contextHeuristic that sacrifices precision for rapid retrieval Useful for many applications:
HCI Devices, Embodied acting agentsProblem: Which CBIR techniques are better?How do you measure CBIR output?Slide5
CBIR Process
Input (I)
CBIR Process
Reasoning Engine
Retrieved Propositions
Background Knowledge Sources (
BKS
)
Query (
Q
)Slide6
F-Measure
Retrieved Propositions(RET)
RelevantPropositions(REL)
Recall (r)
Precision (p)
F-Measure
vs.Slide7
Establishing Relevant Propositions
Relevant propositions are only those needed for performing the required reasoningEstablish what’s really relevantCan be generated manuallyNot practical for large KBsAutomatic procedures are desirableRun prior to use of CBIR procedureRuntime is not a huge issue
Two will be discussedRelevance-theoreticDistance from the OptimalSlide8
CBIR
Input (I)
CBIR Process
Reasoning Engine
Retrieved Propositions
Background Knowledge Sources (
BKS
)
Query (
Q
)Slide9
Relevant Proposition Tagging
Input (I)
Retrieved Propositions
Background Knowledge Sources (BKS)
Query (
Q
)
Relevant Propositions
RPT
vs.Slide10
Relevance-Theoretic Sperber
and Wilson’s Relevancy TheoryModel of utterance interpretationReceives an input utterance and determines how relevant it is to an agent’s beliefsCan be used for other cognitive processesProposed for measuring relevance in IREstablishing the set of relevant propositionsSlide11
S & W Relevance
After {I Q} is inserted into BKS , a
proposition p BKS is relevant
if it causes a positive cognitive effect¬p
{
I
Q
}
p
helps
strengthens
some
q
{
I
Q}, orp contributes to a contextual implication:{{
I Q}
BKS} non-trivially derives using p some proposition
q {I Q}
alone does not non-trivially derive q, andBKS
alone does not non-trivially derive qp strengthens q
if: q was already derived in {I Q
} and BKS can
non-trivially
derive
q
using
p
i.e.,
q
is independently derived
Non-trivial
derivations are not easy to formalize
No formalization provided by S & W
Consider propositions used in forward chaining as
non-trivialSlide12
Example
BKS
A1 : ∀(x, y)(Blunt(x) ∧ Conical(x) ∧ Drawer(y) ∧ ConnectedByTip(x, y) → Handle(x)).
A2 : ∀(x)(Handle(x) →
CanBePulled
(x)).
A3 : Blunt(h1).
A4 : Conical(h1).
A5 : ∀(x, y)(Rope(x) ∧ Light(y) ∧ Connected(x, y) →
CanBePulled
(x)
A6 : ∀(x, y)(Blunt(x) ∧ Conical(y) ∧
ConnectedByBase
(x, y) → ¬Handle(x)
A7 : ∀(x)(Drawer(x) →
ContainsItems
(x)).
{I
Q}:
{Drawer(d1) ∧
ConnectedByTip(h1, d1) ∧ CanBePulled(h1)}.
rel: {A1, A2, A3, A4, A7}Slide13
Example
CBIR Name|REL||RET|
|RET REL|
RecallPrecision F-Measure
CBIR1
5
4
4
0.8
1.00
0.899
CBIR2
5
5
4
0.8
0.80
0.800
CBIR3
5
540.8
0.800.800BKS
575
1.00.710.830
rel:
{A1, A2, A3, A4, A7}
CBIR
Result Name
CBIR Retrieved
Propositions
CBIR1
{A1,A2,A3,A4}
CBIR2
{A1,A2,A3,A4,A6}
CBIR3
{A2,A3,A4,A5,A7}Slide14
Distance from the Optimal
Using I, Q, and BKS and some reasoner capable of maintaining origin setsOrigin setsProduct of relevance logic/ATMSThe propositions required for deriving some proposition Procedure:
Generate the origin set required for deriving QUse the
origin set as the relevant propositionsCompare CBIR results to the optimal solutionSlide15
Finding the Optimal Solution
Given: Q, BKS, and I2. Load the BKS into a reasoner.3. Add I to the
BKS.4. Query the reasoner on Q.5. Examine the origin set for
Q, , defined as: {A - I| A
{
BKS
I
}
A
├
Q
¬∃A’((A’ A)
A’ ├ Q) } 6. Select the sets in that have the minimal cardinality. This new set of origin sets will be denoted with
min( ) Slide16
Example
BKS
A1 : ∀(x, y)(Blunt(x) ∧ Conical(x) ∧ Drawer(y) ∧ ConnectedByTip(x, y) → Handle(x)).
A2 : ∀(x)(Handle(x) →
CanBePulled
(x)).
A3 : Blunt(h1).
A4 : Conical(h1).
A5 : ∀(x, y)(Rope(x) ∧ Light(y) ∧ Connected(x, y) →
CanBePulled
(x)
A6 : ∀(x, y)(Blunt(x) ∧ Conical(y) ∧
ConnectedByBase
(x, y) → ¬Handle(x)
A7 : ∀(x)(Drawer(x) →
ContainsItems
(x)).
I:
{Drawer(d1) ∧
ConnectedByTip(h1, d1)}
rel:
{A1, A2, A3, A4}
Q: {CanBePulled(h1)}Slide17
Example
CBIR Name|REL||RET|
|RET REL|
RecallPrecision F-Measure
CBIR1
4
4
4
1.0
1.00
1.0
CBIR2
4
5
4
1.0
0.80
0.889
CBIR3
4
530.75
0.600.667BKS
474
1.00.570.726
rel:
{A1, A2, A3, A4}
CBIR
Result Name
CBIR Retrieved
Propositions
CBIR1
{A1,A2,A3,A4}
CBIR2
{A1,A2,A3,A4,A6}
CBIR3
{A2,A3,A4,A5,A7}Slide18
Relevance-theoretic vs. Distance from the Optimal
SimilaritiesRules of inference used to create relevant proposition setDifferencesDistance of the optimal generates relevant proposition sets that precisely match the original definitionRelevance-theoretic values CBIR outputs with multiple paths of inference to a
solutionRelevance-theoretic requires a formalization of the non-trivial
derivation conceptSlide19
Conclusions and Future Work
ConclusionsRelevance-theoretic approach is less successful at measuring some CBIR results than the distance from the optimalUsesComparing different CBIR algorithmsImproving CBIR ProceduresMany CBIR procedures have various parameters that can be modified to change their performanceFuture Work
Use the theoretical discussion to help construct comparisons of CBIR results