Instructor: - PowerPoint Presentation

Slide1

Instructor: Smaranda MuresanColumbia Universitysmara@ccls.columbia.edu

Sentiment LexiconsSlide2

AnnouncementsClass setup on Courseworks too. Class website linked to Courseworks (“Class website” tab)TA’s (

Arpit Gupta) office hoursMonday 4:15-5:15pm in TA room in MuddTA’s email:

ta.cmsm@gmail.comSlide3

Class TodayWord level sentiment analysis (Sentiment Lexicons)Discussion of the two papersIntroduction to Sentiment Analysis beyond words

(sentence level, text level) (to facilitate discussion of articles next week)Slide4

What is sentiment analysis?

Attempts to identify the sentiment/opinion

that a person may hold towards an

object/person/topic

etc

It is a finer grain analysis compared to subjectivity analysis

Sentiment Analysis

Subjectivity analysis

Positive

SubjectiveNegative NeutralObjective

This film should be brilliant. It sounds like a great plot, the actors are first grade, and the supporting cast is good as well, and Stallone is attempting to deliver a good performance. However, it can

’

t hold

up.Slide5

Why sentiment analysis?Movie: is

this review positive or negative?

Products

: what do people think about the new iPhone?

Public sentiment

: how is consumer confidence? Is despair increasing?

Politics

: what do people think about this candidate or issue?

Prediction

: predict election outcomes or market trends from sentiment5Slide6

Goal of today’s lectureGain insights into how sentiment is expressed lexicallyBegin developing resources that are useful in higher level classification (phrase level, sentence level, document level)Explore different philosophies on how to build such large scale sentiment lexiconsSlide7

What are we classifying

gross

(

gross,adj

)

(

gross,noun

)

(

gross,verb)gross outGROSS!!!The soup was gross – 1 starThe horror movie was gross – 5 starsSlide8

WordsAdjectives

positive:

honest important mature large patient

Ron Paul is the only

honest

man in Washington.

Kitchell’s writing is unbelievably

mature

and is only likely to get better. To humour me my patient father agrees yet again to my choice of film Slide9

WordsAdjectives

negative: harmful hypocritical inefficient insecure

It

was a

macabre

and

hypocritical

circus. Why are they being so inefficient ? Slide from Janyce WiebeSlide10

Other parts of speechVerbspositive

: praise, love

negative

:

blame, criticize

Nouns

positive

:

pleasure, enjoyment

negative: pain, criticismSlide11

Hand Annotated/Compiled Lexicons

WordNet-based approaches

Distributional Approaches

How to build sentiment lexiconsSlide12

General Inquirer (GI)Harvard General Inquirer Database (Stone, 1966)Total of 11,788 terms

http://www.wjh.harvard.edu/~inquirer/spreadsheet_guide.htmhttp://www.wjh.harvard.edu/~inquirer/homecat.htm

Positive (1915 words)

vs

Negative (2291 words)

(rest of 7582 could be consider Neutral)

Strong

vs

WeakActive vs PassiveOverstated versus Understated

Pleasure, Pain, Virtue, ViceMotivation, Cognitive Orientation, etcSlide13

WordNet (Miller, 1995; Fellbaum, 1998)

Semantic Lexical resource http://wordnetweb.princeton.edu/perl/webwn

www.globalwordnet.org

(multilingual)

Synsets

(denote different senses of a word)Slide14

http://www-3.unipv.it/wnop

/

Micro-

WNOp

(

Cerini

et al 1997)1105

Wordnet Sysnsets related to opinion topic(initial words were selected from the GI)Slide15

Micro-WNOp

(Carrenini et al 1997)

Micro-

WNOp

statistics reduced to the 702

sysnsets

when everyone agreed

ISSUES with Hand built Lexicons such as GI, Micro-

WNOp

???Slide16

Hand Annotated/Compiled Lexicons

WordNet-based approaches

Distributional Approaches

How to build sentiment lexiconsSlide17

Simple sense/sentiment propagationHypothesis: Sentiment is constant throughout regions of lexically related items. Thus,

sentiment properties of

hand-built seed-sets

will be

preserved as we follow

WordNet

relations out

from them

.

SentiWordNet (Esuli and Sebastiani, 2006)Approx 1.7 Million wordsUsing WordNet and Machine Learning (Classifiers).Each synset is assigned three scoresPositiveNegativeObjectiveSlide18

Values in 3 dimension sum to 1.

Ex:

P=0.75, N=0, O=0.25Slide19

Building SentiWordNetLp, Ln, Lo are the three seed sets

Iteratively

expand the seed sets through K steps

Train

the classifier for the expanded setsSlide20

Lp

Ln

also-see

antonymy

Expansion of seed sets

The sets at the end of kth step are called Tr(k,p) and Tr(k,n)

Tr(k,o) is the set that is not present in Tr(k,p) and Tr(k,n)Slide21

Committee of classifiersTrain a committee of classifiers of different types and different K-values for the given dataObservations:Low values of K give high precision and low recallAccuracy in determining positivity or negativity, however, remains almost constantSlide22

Useful Sentiment Tutorialhttp://sentiment.christopherpotts.net/Has code related to WordNet

propagation methods (used in SentiWordNet) Many other pointers!Issues with the

WordNet

based propagation lexicons? Slide23

Other Sentiment LexiconsSlide24

MPQA Subjectivity Cues LexiconHome page: http://www.cs.pitt.edu/mpqa/

subj_lexicon.html6885 words from 8221 lemmas2718 positive4912 negativeEach word annotated for intensity (strong, weak)

GNU GPL

24

Theresa

Wilson,

Janyce

Wiebe

, and Paul Hoffmann (2005). Recognizing Contextual Polarity in Phrase-Level Sentiment Analysis. Proc. of HLT-EMNLP-2005.Riloff and Wiebe (2003). Learning extraction patterns for subjective expressions. EMNLP-2003.Slide25

Bing Liu Opinion LexiconBing Liu's Page on Opinion Mininghttp://www.cs.uic.edu/~liub/FBS/opinion-lexicon-English.rar

6786 words2006 positive4783 negative

25

Minqing

Hu and Bing Liu.

Mining

and Summarizing Customer

Reviews. ACM SIGKDD-2004.Slide26

Disagreements between polarity lexicons

Opinion Lexicon

General

Inquirer

SentiWordNet

MPQA

33/5402

(0.6%)

49/2867 (2%)1127/4214 (27%)Opinion Lexicon32/2411 (1%)1004/3994 (25%)General Inquirer520/2306 (23%)SentiWordNet26Christopher Potts, Sentiment Tutorial, 2011 Slide27

Hand Annotated/Compiled Lexicons

WordNet-based approaches

Distributional Approaches

2 papers for discussion today

How to build sentiment lexiconsSlide28

Presenter: Smaranda Muresan

Predicting the semantic orientation of adjectives Hatzivassiloglou

&

McKeown

1997Slide29

GoalPredicting polarity of adjectives from a large corpusTest the hypothesis: the morphosyntactic

properties of coordination provide reliable information about adjectival oppositions and lexical polaritiesSlide30

Adjectives conjoined by “

and” have same polarityFair and legitimate, corrupt

and

brutal

*fair

and

brutal, *corrupt

and

legitimateAdjectives conjoined by “but

” do notfair but brutalSlide31

ApproachExtract conjunctions of adjectives from a large corpus, along with relevant morphological relationsUse a log-linear regression model to predict orientation of two different adjectives Use a clustering algorithm to separate the adjectives into two subsets of different orientation

Use average frequencies in each group to assign the label (group with highest frequency is labeled positive)Slide32

Seed dataLabel seed set of 1336 adjectives

(all >20 in 21 million word Wall Street Journal corpus)657 positive

adequate central clever

famous intelligent

remarkable

reputed sensitive

slender

thriving…

679 negativecontagious drunken ignorant lanky listless primitive strident troublesome unresolved unsuspecting…

Further validation: ask 4 human judges to label a subset of 500 adjectives: 96.97% average inter-judge agreement32Slide33

Validating the HypothesisRun a parser on 21 million words dataset to get 15,048 conjunction tokens involving 9,296 pairs of distinct adjective pairs.Each conjunction was classified into :

1) conjunction used (and, or, but ,…)2) type of modification (attributive, predicative) 3) number modified noun (singular or plural)Considered conjunction where both members were in the seed set

(e.g.

clever and sensitive

)

Count percentage of conjunction in each category with adjectives of same or different orientationSlide34

Validating Hypothesis

For almost all the cases p-values are low. Hence the statistics are significant. ‘and’ usually joins adjectives of same orientation‘

but’ is opposite and joins adjectives of different orientationSlide35

Link Prediction

classy

nice

helpful

fair

brutal

irrational

corrupt

Baseline: always use same orientation – 77.84%

the “but” rule

morphological rules (adequate

-inadequate

)

Better idea: supervised learning using

log-linear

regressionSlide36

Result of PredictionLog Linear Regression models performs slightly better than baselineSlide37

Clustering for partitioning the graph into two groups

Log Linear model generates a dissimilarity score between two adjective between 0 and 1

37

classy

nice

helpful

fair

brutal

irrational

corruptSlide38

Labeling the clustersTwo key insights about pairs of words of opposite orientations:

- semantically unmarked member has positive orientation (e.g honest (unmarked)

vs

dishonest (marked))

- semantically unmarked

member is the most frequent

38

classynicehelpfulfairbrutalirrationalcorrupt

+

-Slide39

Output polarity lexiconPositivebold decisive disturbing generous good honest important large mature patient peaceful positive proud sound stimulating straightforward strange talented vigorous witty…Negative

ambiguous cautious cynical evasive harmful hypocritical inefficient insecure irrational irresponsible minor outspoken pleasant reckless risky selfish tedious

unsupported vulnerable

wasteful…

39Slide40

Output polarity lexiconPositivebold decisive disturbing

generous good honest important large mature patient peaceful positive proud sound stimulating straightforward strange talented

vigorous witty…

Negative

ambiguous

cautious

cynical

evasive harmful hypocritical inefficient insecure irrational irresponsible minor outspoken

pleasant reckless risky selfish tedious unsupported vulnerable wasteful…40Slide41

Evaluating Clustering of AdjectivesTried to account for graph connectivityUsed the adjectives from seed set (A) and links given by conjunction and morphological rulesSeparate in training/testing using a parameter

αhigher α creates subset of A such that more adjectives are connected to each other.Slide42

Clustering ResultsHighest accuracy obtained when highest number of links were present.

Ratio of group frequency correctly identified the positive subgroupSlide43

Graph Connectivity and Performance Parameter P measures how well each link is predicted independently – PrecisionParameter k – average number of links for each adjective:Goal: even if P is low, given enough data (high k) a high performance for group prediction is achieved Slide44

ResultsSlide45

Discussion pointsWhat do you see the major contribution of this paper? - Helps to highlight in a quantitative way the relationship between sentiment and particular words and constructions (coordination)- useful linguistic insight

- corpus best method (thus avoiding limitation of human built resources such as WordNet)

- Can be extended to nouns and verbs.

Classic paper, cited 1127 timesSlide46

Discussion pointsDoes it have all the information for anyone to be able to replicate the results?How is the dissimilarity value computed? (multiple values are delivered for an adjective pair in different environments)

What are the limitations of the approach?Method is limited by human cleverness in coming up with useful constructionsSlide47

Velikovich et alSlide48

Class TodayWord level sentiment analysis (Sentiment Lexicons)Discussion of the two papersIntroduction to Sentiment Analysis beyond words (phrase level, text level)

(to facilitate discussion of articles next week)Slide49

What is sentiment analysis?

Attempts to identify the sentiment/opinion/attitude

that a person may hold towards an

object/person/topic

etcSlide50

ComponentsHolder (source) of attitude

Target (aspect) of attitudeType

of attitude

From a set of types

Like, love, hate, value, desire,

etc.

Or (more commonly) simple weighted

polarity

: positive, negative, neutral, together with strength

Text containing the attitudeSentence or entire document50This film should be brilliant. It sounds like a great plot, the actors are first grade, and the supporting cast is good as well, and Stallone is attempting to deliver a good performance. However, it can’t hold up.Slide51

Sentiment AnalysisSimplest task:Is the attitude of this text positive or negative?More complex:Rank the attitude of this text from 1 to 5

Advanced:Detect the target, source, or complex attitude typesSlide52

Sentiment AnalysisSimplest task:Is the attitude of this text positive or negative?More complex:Rank the attitude of this text from 1 to 5

Advanced:Detect the target, source, or complex attitude typesSlide53

Sentiment Analysis

A Baseline AlgorithmSlide54

Sentiment Classification in Movie ReviewsPolarity detection:Is an IMDB movie review positive or negative?

Data: Polarity Data 2.0: http://www.cs.cornell.edu/people/pabo/movie-review-data

Bo Pang, Lillian Lee, and

Shivakumar

Vaithyanathan

. 2002. Thumbs up? Sentiment Classification using Machine Learning Techniques. EMNLP-2002,

79—86.

Bo Pang and Lillian Lee. 2004. A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts. ACL, 271-278Slide55

Text Classification: definitionThe classifier (test phase): Input

: a document d (e.g., a movie review)Output

: a predicted class

c

from some fixed set of labels

c

1

,...,

cK

(e,g,pos, neg)The learner (training phase):Input: a set of m hand-labeled documents (d1,c1),....,(dm,cm)Output: a learned classifier f:d  cSlide56

IMDB data in the Pang and Lee databasewhen _star wars_ came out some twenty years ago , the image of traveling throughout the stars has become a commonplace image . […]

when han solo goes light speed , the stars change to bright lines , going towards the viewer in lines that converge at an invisible point . cool .

_

october

sky_ offers a much simpler image–that of a single white dot , traveling horizontally across the night sky . [. . . ]

“ snake eyes ” is the most aggravating kind of movie : the kind that shows so much potential then becomes unbelievably disappointing .

it’s not just because this is a

brian

depalma film , and since he’s a great director and one who’s films are always greeted with at least some fanfare . and it’s not even because this was a film starring nicolas cage and since he gives a brauvara performance , this film is hardly worth his talents . ✓✗Slide57

Baseline Algorithm (adapted from Pang and Lee)TokenizationFeature ExtractionClassification

using different classifiersNaïve BayesMaxEntSupport Vector Machines (SVM)Slide58

Sentiment Tokenization IssuesDeal with HTML and XML markupTwitter mark-up (names, hash tags)Capitalization (preserve for words in all caps)

Phone numbers, datesEmoticonsUseful code:Christopher Potts sentiment tokenizer

Brendan O’Connor twitter tokenizer

58Slide59

Extracting Features for Sentiment ClassificationHow to handle negationI

didn’t like this movie vs

I really like this movie

Which words to use?

Only adjectives

All

words

All words turns out to work better, at least on this data

59Slide60

NegationAdd NOT_ to every word between negation and following punctuation:

didn’t like this movie , but Ididn’t

NOT_like

NOT_this

NOT_movie

but I

Das

, Sanjiv and Mike Chen. 2001. Yahoo! for Amazon: Extracting market sentiment from stock message boards. In Proceedings of the Asia Pacific Finance Association Annual Conference (APFA).Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up? Sentiment Classification using Machine Learning Techniques. EMNLP-2002, 79—86.Slide61

Classification methodsNaïve BayesMaxEntSVMSlide62

Evaluating ClassificationEvaluation must be done on test data that are independent of the training data usually a disjoint set of instances

Classification accuracy: c

/

n

where

n

is the total number of test instances and

c

is the number of test instances correctly classified by the system.

Adequate if one class per documentResults can vary based on sampling error due to different training and test sets.Average results over multiple training and test sets (splits of the overall data) for the best results.Slide from Chris ManningSlide63

Cross-ValidationBreak up data into 10 folds

(Equal positive and negative inside each fold?)For each foldChoose the fold as a temporary test setTrain on 9 folds, compute performance on the test fold

Report

average

performance of the 10

runsSlide64

Other issues in ClassificationMaxEnt and SVM tend to do better than Naïve Bayes

64Slide65

Problems: What makes reviews hard to classify?Subtlety:

Perfume review in Perfumes: the Guide:

“If you are reading this because it is your darling fragrance, please wear it at home exclusively, and tape the windows shut.

”

65Slide66

Thwarted Expectationsand Ordering Effects“This film should be

brilliant. It sounds like a great plot, the actors are first grade, and the supporting cast is

good

as well, and Stallone is attempting to deliver a good performance. However, it

can’t hold up

.

”

Well as usual Keanu Reeves is nothing special, but surprisingly, the

very talented Laurence Fishbourne

is not so good either, I was surprised.66Slide67

Due Next ClassReadings Chapter 4 from Pang and Lee “Opinion Mining and Sentiment Analysis” book2 papers for discussions

A short data analysis assignment Description on Courseworks under AssignmentsGoal is to get a better understanding of data and the problems discussed in class

Grade: Excellent/Good/Insufficient

Due before class. No late submissionsSlide68

Next classDiscussion of 2 papers (50 minutes)25 minutes per paper

Prepare a 15 min presentations and lead discussion for 10 minutes5 min breakMore in depth lecture on sentiment analysis & open questions (can lead to ideas for projects)30 minutes

Introduction to Emotion/Mood

(

25 minutes)Slide69

AnnouncementsThe assignments of paper for discussions will be done by Saturday, Feb 1, 5pm. TA office hours 4:15-5:15pm Mondays in the TA room in

MuddTA email: ta.cmsm@gmail.com

Email TA if you’d like a tutorial on Text Classification and existing toolkitsSlide70

AnnouncementsGrading policy slightly updated to include data analysis assignments10% data analysis assignments (3 assignments, grading Excellent/Good/Insufficient). No late submissions! See class website or details30% discussion of papers

60% project10% literature review part5% class presentation45% final paper and project