Abhijit Mishra 1 Aditya Joshi 123 Pushpak Bhattacharyya 1 1 IIT Bombay India 2 Monash University Australia 3 IITBMonash Research Academy At 5th Workshop on Computational Approaches to Subjectivity Sentiment amp Social Media Analysis ACL 2014 Baltimore ID: 414430
Download Presentation The PPT/PDF document "A cognitive study of subjectivity extrac..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
A cognitive study of subjectivity extraction in sentiment annotation
Abhijit Mishra1, Aditya Joshi1,2,3, Pushpak Bhattacharyya11 IIT Bombay, India 2 Monash University, Australia3IITB-Monash Research Academy
At 5th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis, ACL 2014, Baltimore Slide2
Subjectivity Extraction
Goal: To identify subjective portions of textSlide3
Motivation
Strong AI suggests that a machine must be perform sentiment analysis in a manner and accuracy similar to human beingsDo humans perform subjective extraction as well?A “cognitive study” of subjectivity extraction in sentiment annotationSlide4
Outline
Sentiment Oscillations & Subjectivity ExtractionExperiment SetupAnticipation & HomingConclusion & Future WorkSlide5
Sentiment Oscillations & subjectivity extraction
Subjective documents may be:Humans perform subjectivity extraction either as a result of “anticipation” or as “homing”.Which of the two methods are adopted depends on the linear/oscillating nature of the subjective document.Linear:The story was captivating. The actors did a great job. I absolutely loved the movie!Oscillating:The story was captivating. Only if they had better actors. But then I enjoyed the movie, on the whole.Slide6
Experiment Setup (1/2)
A human annotator reads a document and predicts its sentimentA Tobii T120 eye-tracker records eye movements while he/she reads the document* No time restriction, no user input required: to minimize errors.Slide7
Experiment Setup (2/2)
Dataset3 Movie reviews in English from imdbOne linear, one oscillating, one between the two extremes (D0, D1, D2 respectively)Three documents? Really?!To eliminate predictabilityTo reduce errors due to fatigue12 human annotators (P0, .. P11 respectively)Slide8
Observations: Anticipation (1/2)
In case of linear subjective documents, an annotator reads some sentences and begins to skip sentences. Slide9
Observations: Anticipation (2/2)
DocumentLengthAverage number of non-unique sentences read by participantsD01021D1933.83D21350.42Slide10
Observations: Homing (1/3)
In case of oscillating subjective documents, an annotator (a) first reads all sentences, (b) revisits some sentences againSlide11
Observations: Homing (2/3)
Considerable overlap between sentences that are read in the second passAll of them are subjective.ParticipantTFD-SEPTFDTFC-SEP57.3821P73.1511P9
51.94
10
26
P11
116.6
16
56
Reading statistics for D1
TFD: Total fixation duration for subjective extract;
PTFD: Proportion of total fixation
duration = (TFD)/(Total duration);
TFC-SE:
Total fixation count for subjective extractSlide12
Observations: Homing (3/3)
Homing at a sub-sentence levelSarcasmMultiple regressions around the sarcasm portion for participant P1, document D1Participant P1 does not correctly detect the sentiment of the documentThwartingSlide13
Conclusion & Future Work
Based on how sentiment changes through a document, humans may perform subjectivity extraction as a result of anticipation or homingApplications:Pricing models for crowd-sourced annotationSentiment classifiers that incorporate “sentiment runlengths”Slide14
References
WikiSent : Weakly Supervised Sentiment Analysis Through Extractive Summarization With Wikipedia, Subhabrata Mukherjee and Pushpak Bhattacharyya, ECML PKDD 2012A sentimental education: sentiment analysis using subjectivity summarization based on minimum cuts, Bo Pang, Lillian Lee, ACL 2004