Jorge Carrillo de Albornoz Laura Plaza Pablo Gervás Alberto Díaz Universidad Complutense de Madrid NIL Natural Interaction based on Language 1 Jorge Carrillo de Albornoz ECIR 2011 Motivation ID: 223884
Download Presentation The PPT/PDF document "A Joint Model of Feature Mining and Sent..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
A Joint Model of Feature Mining and Sentiment Analysis for Product Review Rating
Jorge Carrillo de AlbornozLaura PlazaPablo Gervás Alberto Díaz
Universidad Complutense de MadridNIL (Natural Interaction based on Language)
1
Jorge Carrillo de Albornoz - ECIR 2011Slide2
Motivation
Product review forums have become commonplaceReviews are of great interestCompanies use them to exploit their marketing-mixIndividuals are interested in others’ opinions when purchasing a productManual analysis is unfeasible
Typical NLP tasks:Subjective detectionPolarity recognitionRating inference, etc.Jorge Carrillo de Albornoz - ECIR 20112Slide3
Motivation
Traditional approaches:Term frequencies, POS, etc.Polar expressionsThey do not take into account:The product features on which the opinions are expressedThe relations between them
Jorge Carrillo de Albornoz - ECIR 20113Slide4
Hypothesis
Humans have a conceptual model of what is relevant regarding a certain productThis model influences the polarity and strength of their opinionsIt is necessary to combine feature mining and sentiment analysis strategies toAutomatically extract the important featuresQuantify the strength of the opinions about such features
Jorge Carrillo de Albornoz - ECIR 20114Slide5
The HotelReview Corpus
25 reviews from 60 different hotels (1500 reviews)Each review:The cityThe reviewer nationalityThe date
The reviewer categoryA score in 0-10 ranking the opinionA free-text describing, separately, what the reviewer liked and disliked5Jorge Carrillo de Albornoz - ECIR 2011Slide6
The HotelReview Corpus
No relation between the score and the text describing the user opinion:
Two annotatorsExcellent, Good, Fair, Poor and Very poorGood, Fair and PoorAfter removing conflicting judgments =1000 reviewsJorge Carrillo de Albornoz - ECIR 20116Good location. Nice roof restaurant - (I have stayed in the baglioni more than 5 times before). Maybe reshaping/redecorating the lobby.Noisy due to road traffic. The room was extremely small. Parking awkward. Shower screen was broken and there was no bulb in the bedside light.Slide7
The HotelReview Corpus
Jorge Carrillo de Albornoz - ECIR 20117
Download: http://nil.fdi.ucm.es/index.php?q=node/456Slide8
Automatic Product Review Rating
Step I: Detecting Salient Product FeaturesIdentifying the features that are relevant to consumersStep II:
Extracting the User OpinionExtracting from the review the opinions expressed on such featuresStep III: Quantifying the User OpinionsPredicting the polarity of the sentences associated to each featureStep IV: Predicting the Rating of a ReviewTranslating the product review into a Vector of Feature Intensities (VFI)Jorge Carrillo de Albornoz - ECIR 20118Slide9
Step I: Detecting Salient Product Features
Objective: Identifying the product features that are relevant to consumers Given a set of reviews R={r
1, r2, …, rn}:The set of reviews is represented as a graphVertices = conceptsEdges = is a + semantic similarity relationsThe concepts are ranked according to its salience and a degree-based clustering algorithm is executedThe result is a number of clusters where each cluster represent a product featureJorge Carrillo de Albornoz - ECIR 20119Slide10
Step II: Extracting the User Opinion on Each Product Feature
Objective: Locating in the review all textual mentions related to each product featureMapping the reviews to WordNet concepts
Associating the sentences to feature clusters:Most Common Feature (MCF): more WordNet concepts in commonAll Common Features (ACF): every feature with some concept in commonMost Salient Feature (MSF): the sentence is associated to the highest score featureJorge Carrillo de Albornoz - ECIR 201110Slide11
Step III: Quantifying the User Opinions
Objective: Quantifying the opinion expressed by the reviewer on the different product featuresClassifying the sentences of each review into positive or negativeAny polarity classification system may be usedOur system:
Concepts rather than termsEmotional categoriesNegations and quantifiersJorge Carrillo de Albornoz - ECIR 201111Slide12
Step IV: Predicting the Rating of a Review
Objective: Aggregate all previous information to provide an overall rating for the reviewMapping the product review to a VFIA VFI is a vector of N+1 values representing the detected features and the
other featureTwo strategies for assigning values to the VFI:Binary Polarity (BP): the position in the VFI of the feature assigned to each sentence is increased or decreased in one according to the polarity of the sentenceProbability of Polarity (PP): the feature position is increased or decreased with the probability calculated by the classifierJorge Carrillo de Albornoz - ECIR 201112[-1.0, 0.0, 0.0, 0.0, …,-1.0, 0.0, 0.0,1.0, 0.0, 1.0, 0.0, …., 1.0]Slide13
Experimental Setup
HotelReview corpus: 1000 reviewsDifferent feature sets:Feature set 1: 50 reviews 24 feature clusters and 114 concepts
Feature set 2: 1000 reviews 18 feature clusters and 330 conceptsFeature set 3: 1500 reviews 18 feature clusters and 353 conceptsBaselines:Carrillo de Albornoz et al. (2010)Pang et al. (2002)Jorge Carrillo de Albornoz - ECIR 201113Slide14
Experiment 1
Objectives: To examine the effect of the product feature setTo determine the best heuristic for sentence-to-feature assignment (
Most Common Feature, All Common Features and Most Salient Feature)Task: Three classes classification (Poor, Fair and Good)We use the Binary Polarity strategy for assigning values to the VFI vectorJorge Carrillo de Albornoz - ECIR 201114Slide15
Experiment 1 - Results
Method
Feature set 1Feature set 2Feature set 3MCFACFMSFMCFACFMSFMCFACFMSFLogistic 69.8 67.7
69.8 70.4
67.4
70.8 69.1 67.4
70
LibSVM
69
67.1
69.2
69
67.8
69.2
68.8
67.7
69
FT
66.8
64.2
66.8
66.3
65.2
68.6
68.4
65.8
68.4
Jorge Carrillo de Albornoz - ECIR 2011
15
Average accuracies for different classifiers, using different feature sets
and
sentence-to-feature assignment strategiesSlide16
Experiment 1 - Discussion
Feature set 2 reports the best results for all classifiersAccuracy differs little across different feature sets and increasing the number of reviews used for extracting the features does not always improve accuracyThis is due to the fact that users are concerned about a small set of features which are also quite consistent among users
Jorge Carrillo de Albornoz - ECIR 201116Slide17
Experiment 1 - Discussion
The Most Salient Feature (MSF) heuristic for sentence-to-feature assignment produces the best outcomeThe Most Common Feature (MCF) heuristic reports very close results
But the All Common Features (ACF) one behaves significantly worseIt seems that only the main feature in each sentence provides useful information for the taskJorge Carrillo de Albornoz - ECIR 201117Slide18
Experiment 1I
Objectives:To check if the Probability of Polarity strategy produces better results than the Binary Polarity strategy
Test the system in a 5-classes prediction taskTasks: Three classes classification (Poor, Fair and Good)Five classes classification (Very Poor, Poor, Fair, Good and Excellent)We use the Feature set 2 and the MSF strategy for these experimentsJorge Carrillo de Albornoz - ECIR 201118Slide19
Experiment 1I - Results
Jorge Carrillo de Albornoz - ECIR 201119
Method3-classes5-classesLogistic71.7 46.9LibSVM69.4 45.3FT 66.9 43.7
Carrillo de Albornoz et al. [9]66.7
43.2
Pang
et al. [4]
54.2
33.5
Average accuracies for different classifiers in the 3-classes and 5-classes prediction task.Slide20
Experiment 1I - Discussion
The Probability of Polarity behaves significantly better than the Binary Polarity strategyIt allows to captures the degree of negativity/positivity of a sentence, not only its polarity
It is clearly not the same to say The bedcover was a bit dirty than The bedcover was terribly dirtyJorge Carrillo de Albornoz - ECIR 201120Slide21
Experiment 1I - Discussion
The results in the 5-classes prediction task are considerably lower than in the 3-classes taskThis was expected:The task is more difficultThe borderline between
Poor-Very poor and Good-Excellent instances is fuzzyOur system significantly outperforms both baselines in all tasksJorge Carrillo de Albornoz - ECIR 201121Slide22
Conclusions and Future Work
The system performs significantly better than previous approachesThe product features have different impact on the user opinionUsers are concerned about a relatively small set of product featuresThe salient features can be easily obtained from a relatively small set of product reviews and without previous knowledge
Differences between the various Weka classifiers are not markedJorge Carrillo de Albornoz - ECIR 201122Slide23
Conclusions and Future Work
Error propagation of the sentence polarity classifierError assigning sentences to featuresNot enough information:Dirty. Stinky. Unfriendly. NoisyCo-reference problem:Anyway, everybody else was nice
To evaluate the system over other domainsTo translate the system to other languageJorge Carrillo de Albornoz - ECIR 201123Slide24
Thank you!
Any question?Jorge Carrillo de Albornoz - ECIR 2011
24