Remembering to forget: a competence-preserving case deletion policy for case-based reasoning - PowerPoint Presentation

Slide1

Remembering to forget: a competence-preserving case deletion policy for case-based reasoning systems.

Barry Smyth, Mark

Keane

Context of the paperUtility problem

When cost associated with searching for relevant knowledge outweighs the benefit of applying this knowledge.

Deletion strategy

Ensuring that stored knowledge is genuinely useful.

Traditional strategies exists

Will keep performance in check.

May cause problems for CBR system competence.

Goal of the presented workTo examine the deletion strategy in the context of case-based reasoning systems.

Outline a suite of algorithms for constructing and efficiently maintaining a competence model at runtime.

To introduce two new deletion policies

Footprint deletion and footprint-utility deletion.

Related workQualitative Results Concerning the Utility of Explanation-Based Learning

The utility problem. Certain “harmful” knowledge may actually degrade system performance.

The Utility Problem in Case-Based Reasoning

In CBR systems a special case of the utility problem arises, called the swamping problem.

Information Filtering. Selection Mechanisms

in Learning Systems

Strategies for dealing with the utility problem in terms of information filters applied at different stages in the problem solving process.

Methodology Coverage

and

reachability of cases.

Coverages is set of target problems that it can be used to solve.

Reachability is the set of cases that can be used to provide a solution for the target

Four basic classes of cases

Pivotal, auxiliary, spanning and support.

The Footprint Deletion Policy

Combining footprint deletion and utility deletion.

Experiment 1Initial case-base of 50 cases and a set of 160 target problems.

Firstly presented to the system with no learning taking place

To test the basic competence of the case-base

The result was a benchmark competence of 100% for the given case-base and problem set

Secondly presented to the system with learning allowed and case base restricted to 75 cases

The swamping limit

Experiment 1Four studies carried out

Same target problems, varying deletion policy.

Learning proceeded unhindered until the case base size reached the swamping limit

Subsequent learning only allowed to take place after a case had been removed from the case-base.

Experiment 1 - Results

The two competence-preserving policies managed to maintain overall competence at its benchmark level of 100%.

Random deletion (RD) and utility deletion (UD) resulted in significant drops in competence as important cases were deleted over the course of the experiment.

Almost immediately on reaching the swamping limit competence begins to drop as crucial cases are removed.

Overall competence had fallen to 80% and 86% of its former level for the UD and RD policies respectively.

Experiment 2Necessary to ensure that if a competence contributing case must be deleted, that this case

i

s carefully selected to minimize competence loss.

Re-running previous experiment a number of times with a different swamping limit imposed each time.

From 5 cases to 95 cases.

Overall competence noted and graphed after each re-run.

Experiment 2 Results

The two footprint policies are optimizing competence.

The traditional policies are not.

The curves of the former rise to the maximum benchmark level more rapidly than the curves of the latter.

As more and more cases are allowed into the case-base, the footprint polices seek out those with largest comp.—contribution

In contrast UD and RD based methods have no understanding of competence.

Experiment 2- Results

FD and FUD results can be explained in terms of the distribution of the four different cases bases.

To achieve benchmark competence the case-base must contain at least 27 cases

FD and FUD reach about 90% of the benchmark competence once the swamping limit is just over 25 cases.

RD and UD only reach about 60%

Evaluation

Traditional deletion policies are effective in controlling the swamping problem

from

a performance perspective.

They may lead to

c

ompetence degradation in many CBR systems.

Proposed a solution that uses a model of case competence to guide the learning and deletion of cases.

Suite of algorithms and and two new deletion policies.

Preliminary experimental results are promising

Competence modelling approach may also be used during the initial case acquisition stage of system development

Often undesirable to store every available case in the initial case-base.