Pseudo-Relevance Feedback For Multimedia Retrieval - PowerPoint Presentation

Slide1

Pseudo-Relevance Feedback For Multimedia Retrieval

By Rong Yan, Alexander G. and Rong Jin

Mwangi

S.

Kariuki

2008-11629Slide2

QuizWhat’s Negative Pseudo-Relevance feedback in multimedia retrieval?Slide3

IntroductionAs a result of high demand of content based access to video information.

Content based implies that searching can be done not only through manually indexed terms-directly evaluate if the video content(image and the audio) is similar to the query.Need to allow users to query and retrieve based on the audio information and the imagery of the video, Content-based video retrieval (CBVR) or Content based multimedia retrievalUsing pattern recognition technique.NPRF retrieves images/items which are not similar to the query or relevant information.Slide4

CBVR rely on pre-defined generic similarity which determines the distance between the two images. Limitations include:-visual features representation limited to capturing fairly low-level physical features(color, texture or shape).

Different query scenarios require different similarity metrics to model the distribution of examples. E.g. Sky and water(sea)Slide5

Standard relevance system iteratively asks the user more training examples as relevant or non-relevant for the learning algorithms.After a interactive relevance feedback, the system must then re-build a new classifier.

The top-ranked example from generic similarity metric doesn’t always make the correct result due to poor performance of the current visual information retrieval in applications.e.g.-cars shape Slide6

The Information Digital Video Library System.

Focuses on information extraction from video.Involves the integration of speech, image and natural language.After retrieving the metadata, the system enables full content search and retrieval of the spoken language and visual documents.Slide7

Informedia interface provides multiple levels of abstractions including:-

Visual Icons with relevance measureShort titles or headlinesTopic identification of storiesFilmstrip(storyboard) viewsTranscriptDynamic mapsActive video skimsFace detections and recognitionImage retrieval Slide8

Relevance and Pseudo-Relevance Feedback in Information Retrieval

Main retrieval techniquePseudo-Relevance Feedback is an automatic retrieval approach without any user intervention.Starting with a small no. of positive examples and no negative examples, then extract the strong negative to train the classifier.Transductive learning and co-training are two of paradigms to utilize the information of unlabeled data.Co-training is used to the multimedia retrieval since redundant information is available from different modalities.Slide9

Pseudo-Relevance Feedback

Define the query-text description plus audio, image or video.Video retrieval algorithm retrieves a set of relevant video shots from given data collections.Taking target(T)and query(Q) the retrieval algorithm should provide permutation of the video shots t(i) in T which is sorted by their similarity to the user queries q(i) in Q.The difference between two video segments is measured through a similarity metric between their feature vectors.Then the video collected are separated into two parts of each query positive examples (T+) and Negative examples T(-).Precision and recall are performance measures for retrieval systems, But we use mean average precision since we want the rank.Precision after every retrieved relevant shot is computed and these precisions are averaged. The average precision of this average precision gives the mean average precision.The main idea in PRF approach is to automatically feed back the data which are identified based on generic similarity metricSlide10

Analysis

We can define the positive distance d+ as the distance between the positive data T+ and the queries.The negative distance d- is defined also.The distance d+ and d- will converge towards a gaussian distribution when the no. of examples goes to infinity.Therefore the probability density function(pdf) p(x) for both distance are in form of,Slide11

Which sometimes is also called the error function

er f(x). Slide12

Statistical Model for average precision

Let p(t) be the probability density of T for the data distribution, p(+)t (positive) and P(-)t(Negative) distributions.Slide13

Probabilistic Output and combination

Fusion, combining the base metric and PRF metric.Reduce the prediction variance and offer more stable results.Linearly normalize the scores to a certain interval e.g.[-1,1] As a result all scores(-,+) are gaussian distributed, then we can obtain the probability by applying bayes rule. Parametric sigmoid model to fit the posterior directly Slide14

Base Similarity Metric

Algorithm used to generate the base retrieval scores.Expressed as follows;Can handle multiple examples in arbitrary metric spaces.Retrieval algorithm is assigned a score for each video frame, while the basic unit is a video shot(Multiple frames)-choose the maximal retrieval score of a frame within a video shot’s retrieval score.Slide15
Slide16

Sampling strategy

No. of feedback training examples will be sampled as the input to a learning algorithm(+ e.g.)Subset of the e.g. that are dissimilar to the queries will be considered as (- e.g.).Slide17

Classification Algorithm

SVMs are known to yield good generalization performance compared to other classification algorithms. The decision function is of the form;Slide18

Query Slide19

ResultsSlide20

ConclusionImproved information retrieval, negative pseudo-relevance feedback.

Using learning algorithm for classification-very successful.Multimedia query e.g. provide the (+) training e.g. for machine learning theory(-) training e.g. are obtained from the initial simple Euclidian similarity metric.SVM classifier that learns to weight the discriminating features-improves retrieval performance.NPRF shows the ability to separate Gaussian distribution of the (-) and (+) image reducing the variances.Slide21

Answer 3rd slide.Questions?

Thanks