Baddam Richard Huang Ananya Malik Kai McKeever CS 6471 Spring 2022 A Quick Summary of the Papers Paper A computational approach to politeness with application to social factors Condolence and Empathy in Online Communities ID: 1003754
Download Presentation The PPT/PDF document "Language Civility 2 Seema" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
1. Language Civility 2Seema Baddam, Richard Huang, Ananya Malik, Kai McKeeverCS 6471, Spring 2022
2. A Quick Summary of the PapersPaper"A computational approach to politeness with application to social factors""Condolence and Empathy in Online Communities"Year published20132020Main topicPolitenessDistress & CondolenceContextWikipedia, Stack ExchangeRedditML problemBinary classificationBinary classification, Topic modeling, RegressionMain featuresPoliteness strategies, Relationship with social powerRecognition, Structure, Trends, Empathy, Quality
3. Presentation OutlinePoliteness PaperDiscussionDetailsStrengths and WeaknessesCondolences and Empathy PaperDiscussionDetailsStrengths and WeaknessesPut it all togetherOnline information seekingChoices the authors madeOnline/offline implications
4. Discuss Politenesshttps://padlet.com/cs6471/discussDo you agree with the idea that politeness can be considered a "payment of respect"? (Kaplan 1999)
5. A computational approach to politeness with application to social factorsDanescu-Niculescu-Mizil, Cristian, Moritz Sudhof, Dan Jurafsky, Jure Leskovec, and Christopher Potts. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 250-259. 2013.
6. Politeness, noun.behavior that is respectful and considerate of other peopleshowing or characterized by correct social usagemarked by an appearance of consideration, tact, deference, or courtesyPoliteness TheoryDatasetPoliteness StrategiesPredicting PolitenessSocial FactorsStrengths & Weaknesses
7. Politeness Theory: FaceFirst proposed by Brown and Levinson (1987)Builds off the idea of a faceFace: "The public self-image that every member wants to claim for himself"Positive Face: desire to be accepted and respectedNegative Face: desire to not be imposed onPoliteness TheoryDatasetPoliteness StrategiesPredicting PolitenessSocial FactorsStrengths & Weaknesses
8. Definition of Positive and Negative PolitenessPoliteness is a face-saving actPositive Politeness: A strategy to increase acceptance and respectNegative Politeness: A strategy to decrease imposition on othersPoliteness TheoryDatasetPoliteness StrategiesPredicting PolitenessSocial FactorsStrengths & Weaknesses
9. Positive or Negative Politeness?Politeness TheoryDatasetPoliteness StrategiesPredicting PolitenessSocial FactorsStrengths & WeaknessesI’m sorry it’s a lot to ask, but can you lend me a thousand dollars?You look so sad. Is there anything I can do?
10. Positive or Negative Politeness?Politeness TheoryDatasetPoliteness StrategiesPredicting PolitenessSocial FactorsStrengths & WeaknessesI’m sorry it’s a lot to ask, but can you lend me a thousand dollars?You look so sad. Is there anything I can do?
11. Politeness and PowerWhat do we mean by power? Broadly includes “social status, authority, and autonomy.” (Brown and Levinson, 1987)Politeness Theory predicts a negative correlation between the position and politeness of the speakerPoliteness TheoryDatasetPoliteness StrategiesPredicting PolitenessSocial FactorsStrengths & Weaknesses
12. The Dataset: Politeness Annotated RequestsRequests in online communitiesRequests give rise to more negative politeness strategies Wikipedia and Stack Exchange# Requests# AnnotatedWikipedia35,6614,353Stack Exchange373,5196,604Total409,18010,957Politeness TheoryDatasetPoliteness StrategiesPredicting PolitenessSocial FactorsStrengths & Weaknesses
13. The Dataset: Politeness Annotated RequestsAnnotations using Amazon Mechanical TurksEach request labeled by 5 different annotatorsPoliteness score: normalized and averaged scoreInter-annotator pairwise correlation compared for reliabilityPoliteness TheoryDatasetPoliteness StrategiesPredicting PolitenessSocial FactorsStrengths & Weaknesses
14. The Dataset: Politeness Annotated RequestsBinary perception:Annotated requests broken into four groups by quartileFull agreement more common in first and fourth quartileChose to use top politeness quartile as referenced in later discussionsPoliteness TheoryDatasetPoliteness StrategiesPredicting PolitenessSocial FactorsStrengths & Weaknesses
15. Politeness Strategies From Wikipedia requestsExtracted using Stanford Dependency Parser (de Marneffe et al., 2006)Politeness Scorepositive value indicate politenessTop Politeness Quartile MembershipGives a more robust and intuitive measure of politenessPoliteness TheoryDatasetPoliteness StrategiesPredicting PolitenessSocial FactorsStrengths & Weaknesses
16. StrategyPoliteness ScoreTop Quartile Membership1. Gratitude0.8778%2. Deference0.7870%3. Greeting0.4345%4. Positive Lexicon0.1232%5. Negative Lexicon-0.1322%6. Apologizing0.3653%7. Please0.4957%8. Please start-0.3022%9. Indirect (btw)0.6358%10. Direct Question-0.4315%Positive Politeness StrategiesNegative Politeness StrategiesStrategyPoliteness ScoreTop Quartile Membership11. Direct start-0.439%12. Counterfactual modal0.4752%13. Indicative modal0.0927%14. 1st person start0.1229%15. 1st person pl.0.0827%16. 1st person0.0828%17. 2nd person0.0530%18. 2nd person start-0.3017%19. Hedges0.1428%20. Factuality-0.3813%Not PoliteMore PolitePoliteness TheoryDatasetPoliteness Strategies
17. 9. Indirect (btw)15. 1st person pl.“By the way, where did you find . . .”Could we find a less complex name . . .“By the way, could we find a less complex name?”Is this sentence more polite?7. Please8. Please Start“Could you please say more. . .”“Please do not remove warnings . . .”Which phrase is more polite?Politeness TheoryDatasetPoliteness StrategiesPredicting PolitenessSocial FactorsStrengths & Weaknesses
18. Training dataset and ExperimentsBinary labels: Polite = Top politeness quartileImpolite = Bottom politeness quartileWikipediaWikipediaTraining/Validation setTest setStack ExchangeStack ExchangeWikipediaStack ExchangeStack ExchangeWikipediaCDID4 different experiments:2 In-Domain2 Cross-DomainPoliteness TheoryDatasetPoliteness StrategiesPredicting PolitenessSocial FactorsStrengths & Weaknesses
19. The Training PhaseTwo classifiers: Bag-of-Words (BOW) : SVM + unigram features Ling. : SVM + linguistic features + unigram featuresMain goal of the training:Evaluate the robustness and domain-independence of the politeness strategies.Politeness TheoryDatasetPoliteness StrategiesPredicting PolitenessSocial FactorsStrengths & Weaknesses
20. Relationship between Politeness and PowerRequest to become admin:Potential social status outcomePoliteness before and after the election results:Change in social statusPoliteness TheoryDatasetPoliteness StrategiesPredicting PolitenessSocial FactorsStrengths & Weaknesses
21. Observations:Status ↑ => Politeness ↓Status ↓ => Politeness ↑Relationship between Politeness and PowerPoliteness TheoryDatasetPoliteness StrategiesPredicting PolitenessSocial FactorsStrengths & Weaknesses
22. Other social factors on politenessPreliminary results:- Wikipedians from the U.S. Midwest are the most polite- Female Wikipedians are generally more polite- Programming language communities on Stack Exchange vary significantly by politenessPoliteness TheoryDatasetPoliteness StrategiesPredicting PolitenessSocial FactorsStrengths & Weaknesses
23. StrengthsMachine Learning application to Politeness theorySupported power and politeness relationship hypothesisCreated unique Request corpus annotated for politenessPoliteness TheoryDatasetPoliteness StrategiesPredicting PolitenessSocial FactorsStrengths & Weaknesses
24. WeaknessesCross cultural criticism of politeness theoryEnglish datasetWestern cultureAmazon Mechanical Turk (AMT) for annotationsPoliteness TheoryDatasetPoliteness StrategiesPredicting PolitenessSocial FactorsStrengths & Weaknesses
25. Condolence and Empathy in Online CommunitiesZhou, Naitian, and David Jurgens.In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 609-626. 2020.
26. More definitions, gleaned from paperDistress: expression of unhappiness or painCondolence: expression of emotional support, often in response to distress
27. Paper topic walkthroughRecognizing Distress/CondolenceCondolence BehaviorCondolence StructureEmpathyQuality of CondolenceStrengths & Weaknesses
28. I lost my jobI lost my catI broke my hand todayI am sorry for your lossSending my thoughts and prayersI am sorry about your situationDISTRESSCondolenceRecognizing Distress/CondolenceCondolence BehaviorCondolence StructureEmpathyQuality of CondolenceStrengths & Weaknesses
29. Discuss Distress/Condolencehttps://padlet.com/cs6471/discussHow does the concept of condolence relate to politeness?
30. How did the paper recognize distress and condolence? DistressCondolenceReddit 2017 DatasetSVM Based Classifier BERT Based ClassifierReddit 2016 DatasetSVM Based Classifier BERT Based ClassifierRecognizing Distress/CondolenceCondolence BehaviorCondolence StructureEmpathyQuality of CondolenceStrengths & Weaknesses
31. Metrics achieved Recognizing Distress/CondolenceCondolence BehaviorCondolence StructureEmpathyQuality of CondolenceStrengths & Weaknesses
32. Is condolence directly proportional to distress? Recognizing Distress/CondolenceCondolence BehaviorCondolence StructureEmpathyQuality of CondolenceStrengths & Weaknesses
33. When do we receive condolence the most? Stoneman-Douglas SchoolShootingActor Stefan StefanssondiesRecognizing Distress/CondolenceCondolence BehaviorCondolence StructureEmpathyQuality of CondolenceSlide derived from Zhou & Jurgens’s EMNLP 2020 presentation deck:https://slideslive.com/38939143/condolences-and-empathy-in-online-communities
34. Is every reply a condolence? Recognizing Distress/CondolenceCondolence BehaviorCondolence StructureEmpathyQuality of Condolence
35. Is every reply a condolence? Is this Condolence? DistressRecognizing Distress/CondolenceCondolence BehaviorCondolence StructureEmpathyQuality of Condolence
36. Is every reply a condolence? Recognizing Distress/CondolenceCondolence BehaviorCondolence StructureEmpathyQuality of Condolence
37. Condolence structure20-topic LDA for theme identification—measuring probability of topicTrends observed from this analysisShorter condolences = more sympathyMiddle-length condolences = more personal experiencesLonger condolences = more advice and reframingRecognizing Distress/CondolenceCondolence BehaviorCondolence StructureEmpathyQuality of CondolenceStrengths & Weaknesses
38. Condolence structureSympathy as bookends / “sympathy sandwich”Recognizing Distress/CondolenceCondolence BehaviorCondolence StructureEmpathyQuality of CondolenceStrengths & WeaknessesImage from Zhou & Jurgens’s project webpage:https://blablablab.si.umich.edu/projects/condolence/
39. What is Empathy?Varying definitionsMost computational approaches: mirroring emotional stateEmpathetic Concern vs Personal Distress (Batson et al., 1987)Zhou & Jurgens use definition based on appraisal theory“Observer appraises a person’s situation in the same way as the person experiencing the distress”Recognizing Distress/CondolenceCondolence BehaviorCondolence StructureEmpathyQuality of CondolenceStrengths & WeaknessesBatson et al. (1987) as cited in Sven Buechel’s EMNLP talk: https://vimeo.com/306137544
40. How Zhou & Jurgens model empathyRegression models: Random Forest and RoBERTaCompared two treatments: Target Comments + Observer TextObserver Text OnlyTarget = Person expressing distressObserver = Person offering condolenceRecognizing Distress/CondolenceCondolence BehaviorCondolence StructureEmpathyQuality of CondolenceStrengths & Weaknesses
41. How Zhou & Jurgens model empathyBeing able to align inputs between target and observer benefits performance of both models Recognizing Distress/CondolenceCondolence BehaviorCondolence StructureEmpathyQuality of CondolenceStrengths & Weaknesses
42. Condolence Effectiveness on RedditSampling positive and negative examples of condolences“Your comment made my day”Logistic RegressionWhether response of gratitude is receivedHelpful/Unhelpful/Control FeaturesRecognizing Distress/CondolenceCondolence BehaviorCondolence StructureEmpathyQuality of CondolenceStrengths & Weaknesses
43. Condolence Effectiveness on RedditGood condolencesLexical alignmentPerson-centered responses*Buechel et al. empathy*Compassionate empathyPoor condolencesReligionSympathyAdviceRecognizing Distress/CondolenceCondolence BehaviorCondolence StructureEmpathyQuality of CondolenceStrengths & Weaknesses* Less likely to receive gratitude
44. Reddit’s context“We speculate that people may turn to Reddit for lighter, less-personal forms of support in times of distress, whereas the more compassion-like empathy of Buechel et al. (2018) is helpful when more personal responses are not licensed by the relative anonymity of the platform.” (emphasis ours)To what extent do you agree?Recognizing Distress/CondolenceCondolence BehaviorCondolence StructureEmpathyQuality of CondolenceStrengths & Weaknesses
45. StrengthsCreated a large dataset from scratchConnected methods/results to social psychologyCongruent with existing theorySuggestions for enhancing support in online communitiesRecognizing Distress/CondolenceCondolence BehaviorCondolence StructureEmpathyQuality of CondolenceStrengths & Weaknesses
46. WeaknessesHeuristics to generate datasetHow well do seed phrases recognize/assess condolence?Assumption of common statements in condolence statementsGeneralizabilityAcross cultures, languages, regionsRecognizing Distress/CondolenceCondolence BehaviorCondolence StructureEmpathyQuality of CondolenceStrengths & Weaknesses
47. Putting it together
48. Why do people ask questions online?Is it empathy? Zhou & Jurgens: distressed indiv didn’t find empathetic comments helpfulIs it information-seeking?Concrete suggestions?Domain-specific expectations (Wikipedia vs StackOverflow vs Reddit)
49. Modeling social constructsZhou and Jurgens: “a fixed set of phrases to identify thanks in replies”Danescu-Niculescu-Mizil et al.: politeness as average of five scores of crowdworker annotatorsWhat alternatives are available?How might we assess those choices in light of alternatives?What other choices are worth discussing?
50. Do these findings matter offline?Influence of Reddit/Wikipedia/StackOverflow moderatorsWhat is lost in online speech?Factors ofAnonymityContextToneSpontaneity
51. More Discussion QuestionsIf you were to run the politeness study again, would you use crowdworkers for annotators?What sorts of safeguards should be considered?How might you communicate the ethics that were followed?Do you find it difficult to console others?Do you agree with the explanation this difficulty is “due to the emotional and mental effort…”?How might Zhou & Jurgen’s results generalize to other platforms?Facebook? Instagram? … LinkedIn? 😬