Recommended Banget Nih

!
!
Computational
! Analysis of
Affect and Emotion in Language
Saif M. Mohammad
saif.mohammad@nrc-cnrc.gc.ca
Cecilia Ovesdotter Alm

coagla@rit.edu
Tutorial at EMNLP 2015, Lisboa, Portugal
(Mohammad!and!Alm,!2015)!
Sentiment
and
emotions
PhD in
Computer Postdoc, Senior
Crowd Research
annotations Science, Univ. of
Univ. of Maryland Officer
Toronto
Socia media
posts
NRC, Canada
!
Survey!on!Aﬀect!Analysis:!
(Mohammad,!2015b)!
!
Affect and
subjectivity
AS in Lang &
Speech Assistant
Linguistic Processing Professor
PhD in
sensing Linguistics GS in Comp. CLaSP
Science & Co-Director
Health and
wellness; Engineering
education
UIUC RIT, USA

(Mohammad!and!Alm,!2015)! 2!
Aims
•  Convey why you should care about analysis of affect and

affective conditions, experiences, and activities, and
introduce prospects and challenges in this area
•  Provide a broad background on terminology, conceptual

frameworks, resources, and linguistic considerations
(characteristics, data, annotation, etc.)
•  Give an overview of affect processing which spans both

the lexical level and the level of longer units, and discuss
how visualizing outcomes may enhance analysis
•  Exemplify the area’s relevance in interesting application

domains, followed by wrap-up on future directions
•  Springboard a conversation as we head into EMNLP
Tutorial roadmap
Introduction
Emotive Language Use
Linguistic Data
BREAK! Computational Modeling (Part 1)

30!min!
Computational Modeling (Part 2)
30 min
BREAK Visualizing Computational Outcomes
Survey of Applications
Future Directions and Wrap-up
Introduction © Doug Savage.
Topics:
•  Opportunities for language as a cognitive sensor of affect and emotion
•  NLP tasks and applications involving affect and emotion
•  Concepts and key terminology
•  Challenges to automatic affect detection, characterization, and generation
Affect extends beyond thumbs up/down
(Mohammad!and!Alm,!2015)! 6! (Darwin)
I now have my foot in the door of the custom cake
Your thoughts: decorating business. I start in customer service as a
cashier/barista, work my way through frosting, and then
Which emotional either into wedding, birthday, or sculpted cakes! I have
tone is conveyed? been unemployed for 3 months now and this is huge. It
means I can start saving money again, paying my bills
and loans, and all the while doing something I love!
I create these goals for myself, such as working out or finishing projects and until I finish the
goals that I have set out for myself, I can't finish anything else. I can't go out, I can't do
anything because all I'm thinking about are the unfinished goals.
It seems like I invested too much in it to drop it and I get trapped in this mental prison.
But when I try to work on my projects I just sit there lethargically doing nothing. […]
Relationships with people, such as a significant other, tend to be one of the most interesting
things humans do. There is a lot of variability when dealing with people, so there's a lot
more that tends to keep our interests. Perhaps finding a significant other isn't something that
you should quit on just yet?
Unfortunately, it's an unrealistic expectation to believe that different milestones will
automatically lead to a better life. You can only expect to get enjoyment from life when you
make enjoyment a priority. You can't passively wander through life and hope things will pick
up, you have to seek out what you want from life and strive for it.
Affect-related topics have evolved into a priority in CL
Language serves social and interpersonal functions. Affective meaning is

key for human interaction and a prominent characteristic of language use.
This extends beyond opinions vs. factual or polarity distinctions into multiple
phenomena: emotion, mood, personality, attitude, certainty, credibility,
volition, veracity, friendliness, etc.
Besides recognition, characterization, or generation of affect states, this

involves analysis of affect-related conditions, experiences, and activities.
Consider for instance:
•  Communication-oriented conditions (ASD)
•  Job stress/satisfaction and attention
•  Creative expression in literature and music
•  Risk/protective factors in mental and cognitive health
•  Reasoning dimensions such as decision style and confidence
•  Emotive topics such as domestic violence
Affective meaning
•  Early recognized as important (Bühler, Jakobson, Lyons, Halliday) yet

quite neglected in language science and computational linguistics
•  Affective computing field (Picard and beyond)
•  Steadily growing interest in affective computational semantics
•  Beyond ‘propositional’ meaning which tends to disregard emotive

semantics (truth-conditional semantics: Katz, Fodor, Montague, Lyons,
Russell, Frege)
(1) Bill Clinton was President of the United States.

(2) Bill Clinton never lies.
(3) Bill Clinton loves his wife.
(4) Hillary’s husband is very intelligent/a fool.
Affect analysis for human-centered computing
•  HCC as a theoretical/functional paradigm acknowledges…
•  the central role of humans in computing

•  that humans and machines work better in tandem
•  and language is an essential component of the human condition
•  Linguistic sensing:
•  Language as sensor of the body area network for personalized computing

•  Capture and measure the linguistic signal response of a language user
•  Language data analysis for non-linguistic aims
•  Linguistic data tend to…
•  link to a human individual and a group of individuals

•  be temporally unfolding (evolving states)
•  reflect language users’ overt physical states and latent cognitive states
Linguistic sensing of affective states or affect-related behaviors and experiences

can be used for exploring HCC problems
– such as in healthcare, educational, political, or artistic domains.
Take, for instance, expressive text-to-speech
Consider a person who reads stories. Storytelling

is a complex performance act, and emotional
expressivity is a critical part of good storytelling.
(Mohammad!and!Alm,!2015)! 11! (Alm, 2009)

!
(Alm-Arvius, 1998)
Making sense of affective meaning for TTS is a hard problem, as any utterance
could potentially be rendered with affective tone. Systems should understand
when it is sensible to use a given affect and how to deal with affect ambiguity.
Long list of applied motivations
•  Therapeutic education
•  Customer service and branding
•  Tutoring systems
•  iCALL or edutainment
•  Affective interfaces and dialog systems
•  Medical support systems
•  Public health exploration
We will return to some of these application domains later on.

Concepts: affect states and affect attributes
•  Emotion Distinct
temporal
•  Mood granularity Semantic
•  Personality E<M<P granularity
Target/
•  Attitude source/ Intensity
trigger
Perspective
(self/ Affect Core vs.
periphery
other)
Linguistic Fake vs.

unit real
Modality
Challenges to automatic affect detection,
characterization, and generation
•  Emotional gist of an utterance is not simply the sum of

emotional associations of its component words.
•  Negation and modals impact affect of the text, without

themselves having strong sentiment associations.
•  Emotions are often not explicitly stated.

Another Monday, and another week working my tail off.
Conveys a sense of frustration without overt markers.
•  Prosodic information often absent in text.
•  Different degrees of affect depending on sense and context.

Mary hugged her daughter before going to work. emotional
The pipeline hugged the state border. rather unemotional
•  Difficult to interpret creative uses of language such as sarcasm,

irony, humor, and metaphor/figurative language
•  Some texts, such as social media or literary texts, can be rife
in nonstandard language:
•  misspellings parlament
•  creatively spelled words happeee
•  hashtagged words or similar phenomena #loveumom
•  abbreviations lmao
•  Most machine learning algorithms for affect analysis

require large amounts of training data
•  Numerous affect categories
•  Texts/utterances may convey mixed, contrastive, or

sequences of emotions
•  Multiple affect targets and stimuli
•  Whose perspective?
•  May refer to emotional events without implicitly or
explicitly expressing the message producer’s view
Reference data – the elephant in the room
Human affect perception seems characterized by ...
!  subjectivity & expectations
!  contextual factors CHALLENGING

PROBLEM FOR
COMPUTATIONAL
!  no real ‘ground truth’ INFERENCE
Yet – affect and related experiences are key to the human condition,
and as such are critical to address in computational semantics.
A few summarizing observations
•  Affective meaning goes beyond sentiment and polarity
•  Affect is linked to naturalness, with implications for HCC
•  Useful applications and several challenges makes this an
interesting research area to engage with
•  Distinctions among affect concepts and attributes
•  Affect involves acceptable categories, not right vs. wrong
•  Intersubjective agreement tends to be relaxed
•  A methodological challenge to the ‘ground truth’ concept
-  What makes sense in a particular context?
Instructors: Affect-related tasks thus represent a pedagogical opportunity to

explore tasks without right/wrong answers and clear solutions, and help emerging
investigators develop skills to express and model such complex semantic problems.
Emotive Language Use
Topics:
•  How language users communicate affect and emotion across modalities in
text, speech, signed, and multimodal data
•  Links to sociolinguistic attributes of language users
•  Implications for and translation into features for computational analysis
Language data for affect sensing have many guises
•  Range of modes – standard/nonstandard, dialog/monolog/massive

interaction, speech/signed/text, etc.
•  On body or remotely (FB/Twitter) – temporally unfolding data
•  Privacy consideration – voice recognition, conveying personal information, etc.
•  Affect involves both what language users communicate and how they do it
Linguistic signals are rich and may involve meaningful
layers that open doors to affective semantics
(Alm-Arvius)
Language as window into cognition - how people think & conceptualize

Affect is subject to contextual variation
•  Socio-cultural and interpersonal factors
•  Cultural conventions
•  Social expectations Moderated affect
•  Social stratification expressions/perceptions
•  Taboos and rituals
•  Individuals’ social attributes as additional factors
•  In general in affect perception, individuals’ interpretation

may vary by factors such as …
•  mood and personality
•  emotional intelligence
•  gender (Cowie et al., 2010 and
•  boredom, fatigue studies cited therein;
Alm, 2009)
Linguistic data are, by nature, comparatively
natural, unobtrusive, and inexpensive to capture
FACE
POSTURE
GESTURE
HEART
SKIN
EYE GAZE
PUPILS
etc.
They are convenient to integrate in multimodal capture for affect sensing.

Emotions tend to be conveyed across modalities, e.g., in facial expressions.
(Ekman & Friesen, 1998)
Multimodal sensor capture is motivated by that
multimodality characterizes affect expression;
it can add vetting of linguistic signals
Speech/transcribed
Brain activity
(microphone)
(EEG)
Facial motion
Skin conductance (Kinect)
(GSR)
Cardiac activity Gaze

(Oximeter) (Eye-tracking)
25! (Wang et al., 2014)
Examples of potential linguistic affect signals,
per reports in the literature
•  Lexical contents •  Casuals, negatives (level/ratio)

•  Orthographic forms and conventions •  Affective or mental predicates
•  Terms of address or kinship •  Certain part-of-speech ratios
•  Diminutive or augmentative •  Theme repetitions
morphology, specific affixes
•  Speech acts: commands, warning,
•  Perspective markers, pronouns complimenting, thanking, apologizing,
•  Length of discourses condolences, congratulating, flaming
•  Politeness markers, honorifics, T/V •  Laughing, weeping, disfluencies,
pronouns stuttering, withdrawing, being
•  Intensifiers, quantifiers and inarticulate, speech errors
comparatives, evidentiality markers •  Ideophones, sound symbolism,
•  Mood, modals, hedges reduplications, onomatopoeia
•  Exclamations, insults, interjections, •  Voice/sign inflection: prosody (such as
curses, expletives, imprecations, case duration, pitch, intensity) and voice
markings quality; silence and pausing; changes
•  Syntactic constructions: left- in tempo, signing form, etc.
dislocation, inversion, topicalization or
focus, hedges, clefting, raising, and The obvious complication is that linguistic
grammatical complexity signals are ambiguous and wear multiple
hats – not just serving the expression of
affect or affect-related behaviors.
(Mohammad!and!Alm,!2015)! 26! (Besnier, 1990;

Reilly & Seibert, 2003; inter alia)
Subtle stylistic shifts in syntax can contribute
to affective meaning – as one function
Your canary frightened our cat this

morning.
Unmarked construction

This morning your canary frightened What your canary did was frighten our cat
our cat. this morning.
Fronting Pseudo cleft
It was your canary that frightened Your canary – she frightened our cat this
our cat this morning. morning.
It-cleft
Left dislocation

Clear affect encoding in text may involve various clues
!  Lexicon for affect states, contrastive states, affect-related words/expressions
!  Example sentence with affect-related lexical items:

They laughed and they wept; and Peter embraced the old Fire-drum. (HAPPY)
!  Lexical examples from affect sentences judges agreed upon:
(Mohammad and Alm, 2015) 28! (Alm, 2010)

Clear affect encoding in text may involve various clues
!  Acquired knowledge and human experience, e.g. physical lack and need
(or addiction, incapability, appearance, sleep deprivation/allowance, etc.):
He was hungry and thirsty, yet no one gave him anything; and when it became dark,
and they were about to close the gardens, the porter turned him out. (SAD)
!  Speech acts, e.g. cursing:

Let her be expelled from the congregation and the Church. (ANGRY-DISGUSTED)
!  Forms of direct speech, e.g. interjection:

“Mercy!” cried Karen. (FEARFUL)
!  Mixed emotions (affect sequencing):

He now felt glad at having suffered sorrow and trouble, because it enabled him to
enjoy so much better all the pleasure and happiness around him; for the great
swans swam round the new-comer, and stroked his neck with their beaks, as a welcome.
(HAPPY; mixed sad)
(Mohammad and Alm, 2015) 29! (Alm, 2010)

Classifying affect in sentences whose affect orientation
annotators highly agreed on
Data: Fairytale texts

Linguistic unit of affect: Sentence
Can a model capture affect as a linguistic

phenomenon? When dealing with sentences
that human judges of affect had highly
agreed upon, modeling identified emotion
quite well in sentences based on linguistic
features (substantially improving on random
assignment considering size of affect classes)
Linguistic features toward classification:

lexical, orthographic, story-related, syntactic,
100! lexically derived scores, affect history, poetic
80! structures, lexical sequences
60!
40! Related: Affect sequencing matters;
20! some degree of affect trajectory in stories
0!
All!emo'ons!
Pos+Neg+Neut!
Emo'on+Neut!
(Mohammad!and!Alm,!2015)! 30! (Alm, 2009)

Characterizing affective voices
An interactive genetic algorithm evolved

Users listened to computationally
modified utterances (such as SAD prosodic parameters
voices) and provided feedback based
on their subjective preference
Evolving copy-synthesized
utterances presented
with constant verbal content
(Mohammad and Alm, 2015)

31! (Alm, 2009)
Angry vs. sad voices contrasted by
basic voice inflection cues
Duration: SAD > ANGRY Intensity: ANGRY > SAD
Word!1 ! ! !!!!!!!!!!Word!2! Word!1 ! ! !!!!!!!!!!Word!2!
Global trend with local variation (Alm, 2009)
Intensity seems to help convey moderate stress (Paul et al., 2015)
(Mohammad and Alm, 2015) 32!

•  Linguistic affective signals are heterogeneous, pervasive, unobtrusive and
inexpensive to capture. They are rich in meaning, can be layered, and
offer a window into affect and human cognition. Both what and how may
involve affect expression.
•  Linguistic affect data are informative – either in isolation or in combination

with other sensing modalities.
•  Many linguistic cues potentially encode affect – but they tend to also wear
other hats. Variation can be expected. Sequencing/trajectory may matter.
•  Their prevalence is another factor to consider; while some features may

aid useful analysis on data subsets where they are present, they may not
be as useful for automated prediction/detection more broadly.
•  Attributes of language users (sociolinguistic variables) may play a role.

More work needed about production/perception.
(Mohammad and Alm, 2015) 33!

Linguistic Data
Topics:
•  Alternatives for conceptual computational modeling of affect in language,
including lessons learned from theoretical frameworks in affective science
•  Useful linguistic datasets and lexical resources for computational analysis—
from social media to domain-specific corpora
•  Issues and solutions for linguistic annotation of affect and emotion
Four theoretical affect perspectives
with a history
Social
Darwinian Jamesian Cognitive
constructivist
1. physical appraisal cultural

response constructs
evolved -> of events (practices,
2. emotion in context values)
pre- appraisal
survival-
disposed -> action regulated
related reactions readiness
universal social
automated
(basic) functions
satisfy
cultural
norms
(Cornelius,!2000;!and!see!Calvo!&!
D’Mello,!2010!for!an!extended,!
(Mohammad!and!Alm,!2015)! 35! comprehensive!view!of!frameworks)!
An example of one way to conceptualize
affect as categories
Emotional ~! Neutral
Positive Negative ~! Neutral
(+/W)!
(&)!
~!
A fuzzy ‘gray zone’ between affect and neutrality tends to characterize affect phenomena
Modeling and describing affect
blending & complex emotions

FEAR > HAPPINESS " relief
Basic emotion
categories (“big six”)
and emotion families Affect dimensions such as
(Ekman) valence-pleasantness and
activity-arousal (Russell);
semantic differentials (Osgood)
worry, anxiety,
panic " FEAR Appraisal
(Ortony;
Scherer) Free labeling
interpretative
agents’ action-
imagination of
readiness
informed by perceivers
context
Affective categories vs. affective dimensions
SAM
“Big Six” (Cornelius, 2000)
(Bradley & Lang, 1994)

Yet another Darwinian view – Plutchick
Take home message:
•  No agreed upon model of

affect and emotion
Anticipation •  Neutrality (non-emotion)
Trust is in the same limbo
Anger •  Preferences are quite
Disgust strong in this area
Fear •  Tip! Important to reason
Joy comprehensively about
Sadness design/application-
related labeling choices,
Surprise
with adequate grounding
in the affective sciences.
Ongoing debate:
Universality of perception of emotions
Margaret Mead Paul Ekman

Cultural anthropologist Psychologist
•  Circa 1950’s, Margaret Mead and others

•  Facial expressions, their meanings, culturally determined
•  Paul Ekman
•  Most influential in providing evidence for Darwin, not Mead
•  Universality of six emotions
Ongoing debate:
Universality of emotion perception

Lisa Barrett
Professor of Psychology
•  Grad school experiment on people’s ability to distinguish photos of

depression from anxiety
•  One is based on sadness, and the other on fear
•  Found agreement to be poor
•  Agreement drops for Ekman emotions when participants are given:
•  Just the pictures (no emotion word options)
•  Or, two scowling faces and asked if the two are feeling the same
emotion
Manually created lexical resources
•  Dictionary of Affect (Whissell)

http://sail.usc.edu/dal_app.php
•  Affective Norms for English Words (Texts) (Bradley & Lang)

http://csea.phhp.ufl.edu/media.html
•  Harvard General Inquirer categories (Stone etc.)

http://www.wjh.harvard.edu/~inquirer/
•  NRC Emotion Lexicon (Mohammad & Turney)

http://saifmohammad.com/WebPages/lexicons.html
•  MaxDiff Sentiment Lexicon (Kiritchenko, Zhu, & Mohammad)

Automatically generated lexical resources
•  SentiWordNet (Esuli & Sebastiani)

http://sentiwordnet.isti.cnr.it/
•  WordNet-Affect (Strapparava & Valitutti)

http://wndomains.fbk.eu/
•  NRC Twitter Lexicons (Mohammad, Kiritchenko, & Zhu)
–  Hashtag Emotion Lexicon, Hashtag Sentiment Lexicon,

Sentiment140 Emoticon Lexicon
Sample resources:
Annotated corpora and other products
Annotated corpora (affect categories):
•  Affective Text Dataset (Strapparava & Mihalcea) – news; headlines
http://web.eecs.umich.edu/~mihalcea/downloads.html#affective
•  Affect Dataset (Alm) – classic literary tales; sentences
http://people.rc.rit.edu/~coagla/
•  2012 US Presidential Elections – tweets (Mohammad et al.)
http://saifmohammad.com/WebDocs/ElectoralTweetsData.zip
•  Emotional Prosody Speech and Transcripts – actors/numbers
(Liberman et al.) https://catalog.ldc.upenn.edu/LDC2002S28
•  HUMAINE – multimodal (Douglas-Cowie et al.)
http://emotion-research.net/download/pilot-db/
Other:
•  EmotionML (Schröder et al.) http://www.w3.org/TR/emotionml/
•  ACII (multiple data formats), Interspeech (spoken language)
•  IEEE Trans. on Affective Comp. http://www.computer.org/web/tac
•  EMNLP 2014 Tutorial on Sentiment Analysis of Social Media Texts
(Mohammad & Zhu)
Video & slides: https://www.youtube.com/watch?v=zv16Xyph7Ss
http://www.saifmohammad.com/WebDocs/EMNLP2014-SentimentTutorial.pdf
Emotion datasets in Chinese
•  News and blog posts with Ekman emotions (Wang, 2014)
•  Ren-CECps blog emotion corpus (Quan & Ren, 2009)

•  The sentences are annotated with eight emotions: joy,
expectation, love, surprise, anxiety, sorrow, anger, and hate.
•  2013 Chinese Microblog Sentiment Analysis Evaluation

(CMSAE) Dataset of posts from Sina Weibo annotated with
seven emotions: anger, disgust, fear, happiness, like, sadness
and surprise.
•  The train set: 4000 instances (13252 sentences)
•  The test set: 10000 instances (32185 sentences)
http://tcci.ccf.org.cn/conference/2013/pages/page04 eva.html
Emotion datasets in Japanese
Japanese customer reviews corpus with the same

eight emotions used in the Chinese Ren-CECps Corpus
(Sun et al., 2014)
•  The annotated corpus has ~3K sentences.
•  Each adverb and sentence manually annotated for
association with the eight emotions and the degree
of emotion intensity (0.1 to 1.0)
•  Also created an adverb emotion lexicon
•  687 adverbs and their associations with the eight
emotions.
Emotion datasets in Dutch
Dutch: Dutch sentences annotated into one of eight octants of

Leary’s Rose
•  framework for interpersonal communication (Vaassen &
Daelemans, 2011)
•  evaluate the performance of several classification systems
More on the reference data problem
•  Lower human intersubjective agreement; broadly

impacts affect beyond NL&SP: “[o]btaining high inter-
observer agreement is a challenge in affect data
annotation” (Gunes & Schuller, 2013)
•  Lack of a stable ‘ground truth’
•  Ratings may convey varying acceptability
•  Methodological issues with ‘training’ annotators
•  Immense implications for the development and

evaluation of automated systems, yet the reference data
problem remains quite understudied
Collecting data and annotation
Collecting data:
•  Games with a Purpose, Master-Apprentice, Wizard of Oz
•  Harvesting news, social media, literary texts, etc.
•  Confederates/actors vs. naturalistic data
•  ‘Authenticity’ of naturalistic data
•  Text external vs. text internal perspectives
Collecting reference labels:

•  Independent annotations
•  Pairs/small groups, experts, crowds, GWOP
•  Self-reports or self-annotation
•  Surveys, hashtags, post/forum labels
•  Measurement-based
•  Vetting across modalities
Hashtag words as labels
•  Hashtagged words may act as labels of valence or
emotion categories:
Some jerk just stole my photo on #tumblr #grrr #anger
•  Hashtags labels are not always good labels:

•  Sarcasm
The reviewers want me to re-annotate the data. #joy

•  Unclear from rest of the message

Mika used my photo on tumblr. #anger
(Go, Bhayani, & Huang, 2009;

Mohammad, 2012a)
Affect ratings can be influenced by
domain expertise !
HD: High Distress LD: Low Distress ND: No Distress H: Happy
(Mohammad!and!Alm,!2015)! 51! (Homan et al., 2014)

Using annotation procedures to gain insights
into affect-related meaning phenomena
(Mohammad!and!Alm,!2015)! 52! (Hochberg et al., 2014a)

Annotation analysis can help understanding
fuzzy, yet systematic perception
Full agreement or within one rating

on the continuum for over 90% on
decision-style annotation
(Mohammad!and!Alm,!2015)! 53! (Hochberg et al., 2014b)

Annotation surveys can show trends in factors’
influence and how annotators’ agree/diverge
(Mohammad!and!Alm,!2015)! 54! (Hochberg et al., 2014a)

One way of selecting ‘core’ reference data
for modeling (if that is intended)
!
614!narra'ves! !center!of!gravity!! 663!! !secondary!raZngs!
672!narra'ves!
Final!dataset!for!
Full!agreement! +49! narra'ves! +9!
!
modeling!
!
I!
A!
(Mohammad!and!Alm,!2015)! 55! (Hochberg et al., 2014b)

Example 1 of data elicitation:
Rich narration over images –
social-emotional vs. factual-material inference Qs
Why is there money on Where are they? Why is this picture funny?
the table? What is their relationship? What does the crop
What is going on in this What kind of game are “circle” look like?
picture? they playing? Where are the two green
Who knows that the man has How often do they play men from?
cards behind his back? video games? Why is the man on the left
How rich is each person pointing and smiling?
in this picture?
(Mohammad!and!Alm,!2015)! 56! (Womack et al., 2014)
Example 2 of data elicitation:
Controlled experimental set-up with moderate stress
•  Stroop Test induces cognitive load

•  Color word is presented on
screen. Task is to speak the
font color, not the color word.
•  Induce stress with time and penalty
Unstressed version
•  Unstressed version has no time
limit and no penalty for wrong
answers
•  Stressed version adds a time
limit, and a monetary penalty for
each wrong answer
•  Rest period reduces effect of Stressed version
previous trial (Mohammad!and!Alm,!2015)! 57! (Bethamcerla et al., 2015)
•  Options for affect modeling, with an ongoing debate

•  No one fit all solution
•  Alternatives may be constrained by the application
•  Linguistic affect phenomena:

•  Capturable into lexical resources/datasets with reference labels
•  Multiple alternatives for their respective collection
•  Gradient semantics with narrow ‘core’ and wider ‘gray zone’
•  Contextual factors in interpretation (expertise)
•  Annotations reveal insights into fuzzy phenomena involving

language data, and can sort out ‘core’ vs. ‘periphery’ data subsets
Computational Modeling (Part 1):
Word-level Affect Associations
Topics:
•  Creating term-affect association lexicons: manually and automatically
•  Real-valued associations
•  Twitter-specific associations
•  Negation
Affect in different textual chunks
•  Words
•  Sentences, tweets, SMS messages
•  Paragraphs, documents, customer reviews,

blog posts, essays, stories
Word associations
Beyond denotative meaning, words have other

associations that often add to their meanings.
.
Associations with…
•  sentiment
•  emotions
•  social overtones
•  cultural implications
•  colours
•  music
Connotations.
Word-valence associations
•  Adjectives
•  reliable and stunning typically positive
•  rude and broken typically negative
•  Nouns and verbs

•  holiday and smiling typically positive
•  death and crying typically negative
Capture word-sentiment associations.
Word-emotion associations
Words have associations with emotions:

•  attack and public speaking with fear
•  yummy and vacation with joy
•  loss and disease with sadness
•  result and wait with anticipation
Goal: Capture word-emotion associations.
Manually Creating Valence and Emotion
Association Lexicons
Crowdsourcing affect lexicons
•  Benefits
•  Inexpensive
•  Convenient and time-saving
•  Especially for large-scale annotation
•  Challenges
•  Quality control
•  Malicious annotations
•  Inadvertent errors
•  Words used in different senses are associated with
different emotions.
Word-choice question
Q1. Which word is closest in meaning to cry?
  # car # tree # tears # olive
Peter Turney, AI2
!  Word sets generated automatically

!  Near-synonym taken from thesaurus
!  Distractors are randomly chosen
!  Guides annotator to desired sense
!  Aids quality control

!  If Q1 is answered incorrectly: Responses to the
remaining questions for the word are discarded
(Mohammad & Turney, 2013)

Association questions
Q2. How much is cry associated with the emotion sadness?

(for example, death and gloomy are strongly associated with sadness)
•  cry is not associated with sadness
•  cry is weakly associated with sadness
•  cry is moderately associated with sadness
•  cry is strongly associated with sadness
•  Eight such questions for Plutchik’s eight basic emotions

•  Two such questions for positive or negative valence
•  Each instance annotated by five MTurk workers
Better agreement when asked ‘associated with’ rather than

‘evoke’.

Resulting lexical resource
NRC Emotion Lexicon
•  Sense-level lexicon
•  Has valence and emotion associations for 24,200
word-sense pairs
•  Word-level lexicon
•  Union of emotions associated with different
senses
•  Has valence and emotion associations for 14,200
word types

How to manually create sentiment
lexicons with intensity values?
•  Humans are not good at giving real-valued scores

•  Difficult to be consistent across multiple annotations
•  Challenging to maintain consistency across annotators
•  0.8 for annotator may be 0.7 for another
•  Humans are much better at comparisons (Cohen, 2003)

•  Questions such as:
Is one word more positive than another?
•  Large number of annotations needed.
Need a method that preserves the comparison aspect,

without greatly increasing the number of annotations
needed.
Maximum Difference Scaling (MaxDiff)
•  The annotator is presented with four words (say, A, B, C,

and D) and asked:
•  which word is the most positive
•  which is the least positive
•  By answering just these two questions, five out of the six

inequalities are known, e.g.:
•  If A is most positive
•  and D is least positive, then we know:
A > B, A > C, A > D, B > D, C > D
MaxDiff
•  Each MaxDiff question presented to multiple annotators.
•  The responses translated into:

•  a ranking of all the terms
•  a real-valued score for all the terms (Orme, 2009)
•  If two words have very different degrees of association,

for example, if A >> D:
•  A will be chosen as most positive much more often than D
•  D will be chosen as least positive much more often than A
Leading to a ranked list such that:

•  A and D are significantly farther apart
•  their real-valued association scores significantly different
Dataset of real-valued sentiment scores
•  Selected ~1,500 terms from tweets

•  Regular English words
peace, jumpy
•  Tweet-specific terms
•  Hashtags and conjoined words
#inspiring, #happytweet, #needsleep
•  Creative spellings
amazzing, goooood
•  Negated terms
not nice, nothing better, not sad
•  Obtained MaxDiff annotations
(Kiritchenko, Zhu, & Mohammad, 2014)
Examples of sentiment scores from the
MaxDiff annotations
Term Sentiment Score

0 (most negative) to 1 (most positive)
awesomeness 0.9133
#happygirl 0.8125
cant waitttt 0.8000
don't worry 0.5750
not true 0.3871
cold 0.2750
#getagrip 0.2063
#sickening 0.1389

Robustness of the annotations
•  Divided the MaxDiff responses into two equal halves

•  Generated scores and ranking based on each set
individually
•  The two sets produced very similar results:
•  Average difference in scores was 0.04
•  Spearman’s rank coefficient between the two
rankings was 0.98

Real-valued valence associations
created using MaxDiff
Dataset used in:
•  SemEval-2015 Task 10 (Subtask E): Determining Prior
Polarity
•  Dataset available:
http://alt.qcri.org/semeval2015/task10/index.php?id=data-and-tools
New datasets to be used in:

•  SemEval-2016 Task 7: Determining sentiment intensity of
English and Arabic phrases.
•  Include phrases with modals and negators
•  Task website:
http://alt.qcri.org/semeval2016/task7/
Automatically Generating Valence and
Emotion Association Lexicons
Twitter-specific
valence association lexicons
•  Compiled a list of seed words by looking up synonyms

of excellent, good, bad, and terrible:
•  30 positive words
•  46 negative words
•  Used emoticons as seeds also like Go, Bhayani, &
Huang (2009)
•  Polled the Twitter API for tweets with seed-word
hashtags
•  A set of 775,000 tweets was compiled from April to
December 2012
(Mohammad, Kiritchenko, & Zhu, 2013)

Automatically generated new lexicons
•  A tweet is considered:
•  positive if it has a positive hashtag
•  negative if it has a negative hashtag
•  For every word w in the set of 775,000 tweets, an

association score is generated:
score(w) = PMI(w,positive) – PMI(w,negative)

PMI = pointwise mutual information
If score(w) > 0, then word w is positive
If score(w) < 0, then word w is negative

NRC Hashtag Sentiment Lexicon
•  w can be:
•  any unigram in the tweets: ~54,000 entries
•  any bigram in the tweets: ~316,000 entries
•  non-contiguous pairs (any two words) from the
same tweet: 308,000 entries
•  Multi-word entries incorporate context:
unpredictable story 0.4
unpredictable steering -0.7
Available for download:

Features of the Twitter lexicon
•  Connotation and not necessarily denotation

•  tears, party, vacation

•  Large vocabulary
•  covering wide variety of topics
•  covering words from informal language
•  including creative spellings, hashtags, conjoined
words
•  Seed hashtags have varying effectiveness

•  Study on sentiment predictability of different
hashtags (Kunneman, Liebrecht, & van den Bosch, 2014)
Hashtag words as labels for emotions
•  Hashtagged words may act as labels of valence or

emotion categories
Some jerk just stole my photo on #tumblr #grrr #anger
•  Generated emotion association lexicons from this

peudo-labeled data
•  Showed usefulness in automatic sentence-level
emotion classification for the Ekman emotions
(Mohammad, 2012a)
Ongoing debate:
Universality of emotion perception

Lisa Barrett
Professor of Psychology
•  Is there validity to the notion of a few basic

emotions?
•  Is it time to develop models for large numbers
of emotions?
Generating emotion association lexicon
for 500 emotions
NRC Hashtag Emotion Lexicon: About 20,000 words

associated with about 500 emotions
(Mohammad,!2012a;!!
Mohammad!&!Kiritchenko,!2013a)!!
Negation
Jack was not thrilled at the prospect of working weekends $
negator sentiment
label: negative
need to determine this word’s

sentiment when negated
The bill is not garbage, but we need a more focused effort $
negator sentiment
label: negative
need to determine this word’s

sentiment when negated
Handling negation
Jack was not thrilled at the prospect of working weekends $
in the list of sentiment

negators label: negative
scope of negation
The bill is not garbage, but we need a more focused effort $
in the list of sentiment

negators label: negative
scope of negation
Scope of negation: from negator until a punctuation

(or end of sentence) (Kiritchenko, Zhu, & Mohammad, 2014)
negative label tweets or sentences
positive label

affirmative contexts negated contexts
(in light grey) (in dark grey)

All the affirmative contexts All the negated contexts
Generate sentiment lexicon for

Generate sentiment lexicon for words in negated context
words in affirmative context (Kiritchenko, Zhu, & Mohammad, 2014)
Other recent approaches
to creating valence lexicons
•  Using neural networks and deep learning

techniques (Tang et al., 2014a)
•  Constructing domain-specific lexicons (Chetviorkin &
Loukachevitch, 2014)
•  Other such works (Makki, Brooks, & Milios, 2014; Chen &
Skiena, 2014)
Shared tasks at the word level
•  SemEval-2013, 2014, 2015 Sentiment Analysis in Twitter

(Subtask A): determine sentiment of term in context
•  Positive, negative, or neutral
•  unpredictable movie plot vs. unpredictable steering
https://www.cs.york.ac.uk/semeval-2013/task2/
http://alt.qcri.org/semeval2014/task9/, http://alt.qcri.org/semeval2015/task10/
•  SemEval-2015 Sentiment Analysis in Twitter (Subtask):

determine prior polarity of terms
•  Score between -1 (most negative) and 1 (most positive)
http://alt.qcri.org/semeval2015/task10/index.php?id=subtaske-readme
•  SemEval-2016 Determining Sentiment Intensity of English

and Arabic Phrases
•  Score between -1 (most negative) and 1 (most positive)
•  Many lexical resources available for affect associations

•  Manually created, automatically generated
•  Word-level, sense-level
•  Binary, real-valued associations with a number of affect
categories
•  Annotation practices
•  How you phrase the question matters
•  Comparative aspect of task can lead to finer annotations
•  Automatic methods
•  Naturally capture target data characteristics
•  Can generate large bigram, trigram,… lexicons
(capture more context)
•  Help capture impact of negators and other affect modifiers
Computational Modeling (Part 2):
Sentence-, Tweet-, Message-level
Classification
Topics:
•  Landscape of affect-related tasks
•  Subjectivity, valence, and emotion classification: commonalities and
differences
•  Shared tasks
Analysis of affect
Subjectivity Valence Emotions
objective (factual) joy, sadness, fear,

vs. subjective positive, negative, anger, guilt, pride,
(opinions/ or neutral? optimism,
attitude)? frustration,…!
Subjectivity
!  Early work on subjectivity (Wiebe et al., 2004; Wiebe & Riloff,
2005)
◦  Subjective: having opinions and attitude
◦  Objective: containing facts
!  Applications
◦  Question answering, information retreival, etc.
Query: Give details about the resolution of iPhone 5’s screen?

Relevant: iPhone 5 has 326 pixels per inch
Not relevant: the iPhone has a beautiful touch screen
 
 
Query: How was the iPhone 5’s screen received?

Relevant: the iPhone has a beautiful touch screen
Not relevant: iPhone 5 has 326 pixels per inch

Detecting subjectivity
!  A number of techniques proposed (Hatzivassiloglou & Wiebe,

2000; Riloff & Wiebe, 2003; Wiebe et al., 2004; Su & Markert,
2008; Lin, He, & Everson, 2011; Wang & Fu, 2010)
◦  Use patterns of word usage

◦  Identifying certain kinds of adjectives
◦  Detecting emotional terms
◦  Occurrences of certain discourse connectives
!  Opinion Finder is a popular freely available subjectivity

system (Wilson et al., 2005)
Detecting valence
Cognitive structure of emotions (Ortony, Clore, & Collins, 1990)
valenced reaction to
consequences of events actions of agents aspects of objects

(pleased/displeased) (approving/disapproving) (liking/disliking)
Thousands of papers on automatic valence classification

(surveys by Pang & Lee, 2008; Liu & Zhang, 2012; Martinez-Camara et
al., 2012)
(Ortony, Clore, & Collins, 1990)
(Mohammad!and!Alm,!2015)! ! 97!
Attitude of the
writer, reader, and other entities
•  Much of the work is focused on determining
attitude of the writer.
!  Is the speaker/writer explicitly expressing sentiment?
Consider:

General Tapioca was killed in an explosion.

General Tapioca was ruthlessly executed today.
Mass-murdered General Tapioca finally found and killed in battle.
Detecting valence towards a target
Detecting aspects of products/service and sentiment towards

these aspects (Popescu & Etzioni, 2005; Su et al., 2006; Qadir, 2009;
Zhang et al., 2010)
The lasagna was delicious, but we had to wait 40 minutes before being seated.
food: positive service: negative
SemEval-2014 and 2015 Shared Task: Aspect Based Sentiment

Analysis in the laptop, restaurant domains
Detecting stance
Determining from text whether the author is in favor of,

against, or neutral towards a proposition or target.
•  Prior work on debates and discussions in online forums
(Thomas et al., 2006; Somasundaran & Wiebe, 2009; Murakami &
Raymond, 2010; Anand et al., 2011; Walker et al., 2012; Hasan & Ng,
2013; Sridhar, Getoor, & Walker, 2014)
•  SemEval-2016 Shared Task: Detecting Stance in Tweets
Target: Donald Trump

Text: Jeb Bush is the only sane republican candidate for president.

Text expresses sentiment towards Jeb Bush.

Speaker is likely not in favor of Donald Trump.
Task website:
SemEval-2013
Sentiment Analysis in Twitter
•  International competition on sentiment analysis of tweets:

•  SemEval-2013 (co-located with NAACL-2013)
•  44 teams
•  Subtasks:
•  Is a given message positive, negative, or neutral?
•  tweet or SMS
•  Is a given term within a message positive, negative, or
neutral?
Best performing submission:

NRC-Canada (Mohammad, Kiritchenko, & Zhu, 2013)
Setup
•  Pre-processing:
•  URL -> http://someurl
•  UserID -> @someuser
•  Tokenization and part-of-speech (POS) tagging
(CMU Twitter NLP tool)
•  Classifier:
•  SVM with linear kernel
•  Evaluation:
•  Macro-averaged F-pos and F-neg

Features
Features Examples
sentiment lexicon #positive: 3, scorePositive: 2.2; maxPositive: 1.3; last:
0.6, scoreNegative: 0.8, scorePositive_neg: 0.4
word n-grams spectacular, like documentary
char n-grams spect, docu, visua
part of speech #N: 5, #V: 2, #A:1
negation #Neg: 1; ngram:perfect ngram:perfect_neg,
polarity:positive polarity:positive_neg
all-caps YES, COOL
punctuation #!+: 1, #?+: 0, #!?+: 0
word clusters probably, definitely, probly
emoticons :D, >:(
elongated words soooo, yaayyy

NRC-Canada’s rankings in SemEval 2013 Shared Task
(Sentiment Analysis in Twitter)
F-score
Classify Tweets: positive, negative, or neutral!
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
F-score
0.8 Classify SMS messages: positive, negative, or neutral!
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
(Mohammad,
(Mohammad 104!
Kiritchenko, & Zhu, 2013)
and Alm, 2015)
2 2
(Zhu, Kiritchenko, & Mohammad, 2014)
2 2
(Zhu, Kiritchenko, & Mohammad, 2014)
submissions: 30+
NRC system’s feature contributions
(on Tweets)
F-scores
0.7
0.6
0.5
0.4
0.3

On Movie Reviews Corpus
•  Data from rottentomatoes.com (Pang & Lee, 2005)

•  Train and test set up
•  Two-way classification: positive or negative

Word embeddings
!  Substantial improvements in vision, speech, and now NLP

by representing units with low-dimensional continuous
vectors
!  In NLP, improvements shown in a number of areas including
valence classification (Collobert et al., 2011; Mikolov et al., 2013; Le
& Mikolov, 2014)
!  Word embedding
◦  A vector with a few hundred dimensions
◦  Words closer in meaning are closer in this vector space
◦  Vectors are learned from un-annotated data; can be
combined with annotated data from target application
Expect improvements in affect detection using word

embeddings and deep learning.
Emotions
Six basic emotions
- generally associated with Ekman
•  Anger
•  Fear
•  Disgust
•  Joy
•  Sadness
•  Surprise
“Big Six” (Cornelius, 2000)
Work on fundamental affect categories
Writing style and vocabulary is different in different domains

•  Chat messages (Holzman & Pottenger)
•  Annotated 1200 instances
•  Classic literary tales (Alm)
•  Annotated at the sentence level: Affect Dataset
http://people.rc.rit.edu/~coagla/
•  News paper headlines (Strapparava & Mihalcea)

•  News paper headlines with intensity scores:
Text Affect Dataset
http://web.eecs.umich.edu/ mihalcea/downloads.html#affective
•  Blog posts (Aman & Szpakowicz)

•  Tweets (Mohammad et al.)
–  Tweets from the 2012 US presidential elections.
http://saifmohammad.com/WebDocs/ElectoralTweetsData.zip
Supervised machine learning approaches
•  Proposed by creators of different datasets and others.

(e.g., Chaffar & Inkpen, 2011; Mohammad, 2012c; Kirange &
Deshmukh, 2013)
•  Mostly binary classifiers for each affect category
•  Anger—NoAnger, Joy-NoJoy, etc.
•  Features drawn from:
•  Character and word ngrams
•  Valence association lexicons
•  Emotion association lexicons
•  Part of speech
•  Word clusters
•  Negation
Accuracies for emotion categories usually lower than for
valence classification.
Work on Plutchik’s model
•  Chat messages:
•  Brooks, et al. (2013) annotated ~27,000 chat messages
•  between thirty astrophysics collaborators
•  with 40 affect categories inspired by Plutchik’s
taxonomy
•  Tweets:
•  Mohammad (2012a) collected tweets that have hashtag
emotion words such as #anger and #sadness
•  showed that these hashtag words act as good
emotion labels for the rest of the tweets: distant
supervision
•  Suttles and Ide (2013) collected tweets with emoticons,

emoji, and hashtag words
•  developed an algorithm for binary classification of
tweets along the four opposing Plutchik dimensions.
Work on other small emotion sets
•  ISEAR Project: 3000 student respondents asked to report
situations in which they had experienced joy, fear, anger,
sadness, disgust, shame, or guilt.
http://emotion-research.net/toolbox/toolboxdatabase2006-10-13.2581092615
•  Thomas et al. (2014)
•  supervised machine learning
•  7-way emotion classification.
•  Pearl and Steyvers (2010)
•  Online GWAP
•  Politeness, rudeness, embarrassment, formality, persuasion,
deception, confidence, and disbelief
•  Experience Project: Portal where users share life experiences

www.experienceproject.com
•  Neviarouskaya, Prendinger, & Ishizuka (2009)
•  1000 sentences from the Experience Project
•  manually annotated for fourteen affect categories
Rule-based system
•  Affect of sentence determined by composing meaning

of component pieces
•  Developed rules such as:

•  Negation words (never, nothing) reverse polarity
•  Adverbs of doubt (scarcely, hardly) reverse polarity
•  Adverbs of falseness (wrongly) reverse the polarity
•  Prepositions (without, despite) neutralize attitude
•  Condition operators (if, even though) neutralize
attitude
•  Developed lexicons for attitude, affect modifiers,

and modals (degree of confidence)
(Neviarouskaya,!Prendinger,!&!
Ishizuka,!2010)!
Verb classes from VerbNet
selected for affect detection
1 Psychological state or emotional reaction
1.1 Object-centered (oriented) emotional state (adore)
1.2 Subject-driven change in emotional state (trans.) (charm)
1.3 Subject-driven change in emotional state (intrans.) (appeal to)
2 Judgment
2.1 Positive judgment (bless, honor)
2.2 Negative judgment (blame, punish)
…
•  For each verb class, developed set of rules that are applied
to detect affect.
This work can be a source of ideas for current statistical

models of compositionality, for capturing affect appropriately.
(Neviarouskaya,!Prendinger,!&!
Ishizuka,!2010)!
Work on other small emotion sets
•  Bollen, Pepe, & Mao (2011) analyzed ~9million tweets posted

in the second half of 2008
•  Profile of Mood States (POMS) (McNair, Lorr, & Droppleman, 1989)
•  POMS is a psychometric instrument that measures the
mood states of tension, depression, anger, vigor, fatigue,
and confusion.
•  Wang et al. (2012) compiled a set of 2.5 million tweets with
emotion-related.
•  Hashtags correspond to seven emotion categories: joy,
sadness, anger, love, fear, thankfulness, and surprise.
•  Machine learning algorithm to classify tweets into these
seven emotion categories
•  Most useful features included unigrams, bigrams, sentiment
and emotion lexicons (LIWC, MPQA, WordNet Affect), and
part of speech.
Personality traits
The Big Five personality traits or dimensions of personality

•  extroversion vs. introversion
•  sociable, assertive vs. aloof, shy
•  openness to experience vs. conventionality
•  intellectual, insightful vs. shallow, unimaginative
•  conscientiousness vs. spontaneous
•  self-disciplined, organized vs. inefficient, careless
•  emotional stability vs. neuroticism
•  calm, unemotional vs. insecure, anxious
•  agreeability vs. disagreeability
•  friendly, co-operative vs. antagonistic, fault-finding
Automatic detection of personality traits
From:
•  Stream of consciousness essays
•  Facebook posts, Twitter messages
•  Blog or forum posts
•  Literature
Features:
•  Ngrams not that useful
•  LIWC features for pronouns etc. useful
•  Sentiment and emotion features useful
–  Fine emotion categories more helpful than coarse
sentiment (Mohammad & Kiritchenko)
(Grijalva et al., 2014; Minamikawa & Yokoyama, 2011a, 2011b;
Schwartz et al., 2013b; Malti & Krettenauer, 2013; Mohammad &
Kiritchenko, 2013a, 2013b)
Shared tasks at the sentence level
•  SemEval-2007: Affective Text
http://nlp.cs.swarthmore.edu/semeval/tasks/task14/summary.shtml
•  SemEval-2013, 2014, 2015: Sentiment Analysis in Twitter

https://www.cs.york.ac.uk/semeval-2013/task2/
•  SemEval-2014, 2015: Aspect Based Sentiment Analysis

•  SemEval-2015: Sentiment Analysis of Figurative Language in

Twitter
•  Kaggle Competition: Sentiment Analysis on Movie reviews

https://www.kaggle.com/c/sentiment-analysis-on-movie-reviews
•  Rich landscape of affect-related tasks
•  Subjectivity, valence, emotions
•  Reader or writer perspective
•  Attitude towards a target
•  Many of the features and techniques used in valence
classification are also helpful in emotion classification
•  Additionally, for emotions:
•  Affect association lexicons
•  What else?
•  Need emotion classification shared tasks
•  Applications where emotion detection is shown to be useful
•  Application can guide the choice of affect labels to use
Visualizing Computational
Outcomes
Topics:
•  Common visualization techniques
•  Tracking emotions in large text corpora
•  Interactive Visualizations
(Mohammad!&!
Yang,!2011)!!
relative salience of trust words
(Mohammad!&!Yang,!2011)!!
relative salience of sadness words
(Mohammad!&!Yang,!2011)!!
!!
(Mohammad,!2011)!!
Gender differences in use of emotion words
Some of the claims made in the literature:

•  Women:
•  foster personal relations (Deaux & Major, 1987; Eagly &
Steffen, 1984)
•  share concerns and support others (Boneva et al., 2001)
•  Men:
•  communicate for social position
•  prefer to talk about activities (Caldwell & Peplau, 1982;
Davidson & Duberman, 1982)
What are gender differences in

how writers use emotion words in work-place email?
Study of the Enron Email Corpus
•  Discarded mails with less than 50 and more than 200

words
•  Identified gender of senders by name:
•  41 women, 89 men, 20 untagged
•  Mails from gender-unknown employees discarded
•  Over 30K mails remaining:
•  More sent by men than by women
•  Analyzed emotions words used
•  List of emotion associated words taken from NRC
Emotion Lexicon
(Mohammad & Yang, 2011)
(Mohammad &
Yang, 2011)
(Mohammad &
Yang, 2011)
Emotion word density: Average number of
emotion words in every X words
Brothers Grimm fairy tales ordered as per increasing negative word

density. X = 10,000.
(Mohammad,!2011)!!
Analysis of emotion words in books
Percentage of fear words in close proximity to occurrences of

America, China, Germany, and India in books.
(Mohammad,!2011)!!
Percentage of joy and anger words in close proximity
to occurrences of man and woman in books.
(Mohammad,!2011)!!
Visualizing a thesaurus
•  Thesauri such as Roget’s Thesaurus

•  group words by meaning
•  have a taxonomy: classes, sections, categories,…
•  are used widely to help find the right word to use in
a particular context
•  Traditional means of accessing a thesaurus are limited
•  Imagisaurus (Mohammad, 2015a)

•  interactive visualizer for Roget’s Thesaurus
•  connects thesaurus with the NRC Emotion Lexicon
http://www.purl.com/net/imagisaurus
136!
(Mohammad,!2015a)!!
Default view of
Roget’s thesaurus
Classes
Category view after the slider for negativity is moved to show only
the strongly negative categories
(Mohammad,!2015a)!!
Surprise and Positive
Surprise and Negative
(Mohammad,!2015a)!! 138!
•  Visualizations…
•  allow new intuitive ways to access data
•  quickly convey the structure of data
•  help obtain new insights
•  Interactivity…
•  allows abstraction of details while still allowing
access when needed
•  shows only that information which is relevant to
user needs
Good visualizations help us understand data, and they
can act as demos to convey information effectively.
Survey of Applications
Topics:
•  Political science: Social media analysis in electoral processes
•  Creative and fine arts: Literary analysis and music generation
•  Clinical: Mental health, cognitive health, and medical decision-making
•  Business and education: Leveraging personalized/macro-level affect
sensing
Political science
•  Identify current public opinion towards the candidates in an
election (nowcasting) (Golbeck & Hansen, 2011; Conover et al.,
2011b; Mohammad et al., 2015)
•  Identifying contentious issues (Maynard & Funk, 2011)
•  Detecting voter polarization (Conover et al., 2011a)
•  Predicting the number of votes a candidate will get

(forecasting) (Tumasjan et al., 2010; Bermingham & Smeaton,
2011; Lampos, Preotiuc-Pietro, & Cohn, 2013)
•  Skepticism at the extent to which this is possible possible

(Avello, 2012)
Creative and fine arts: Literature
Analyzing (collections of) literary texts w.r.t. affect:

•  Tracking the flow of emotions in novels, plays, and movie
scripts
•  Detecting patterns of sentiment common to large
collections of texts
•  Kurt Vonnegut on the `Shapes of Stories’
https://www.youtube.com/watch?v=oP3c1h8v2ZQ
•  Syuzhet package by Matthew Jockers:
https://cran.r-project.org/web/packages/syuzhet/
•  Tracking emotions of particular characters or entities over
time
(Hartner, 2013; Kleres, 2011; Mohammad, 2011, 2012b; Alm &
Sproat, 2005)
Generating music from literature:
Music that captures the change in the distribution of emotion words.
(Davis!&!Mohammad,!2014)!!
Challenges
•  Not changing existing music -- generating novel pieces
•  Paralysis of choice
•  Has to sound good
•  No one way is the right way
–  evaluation is tricky
Music-emotion associations
•  Major and Minor Keys
•  Major keys: happiness
•  Minor keys: sadness
•  Tempo
•  Fast tempo: happiness or excitement
•  Melody
•  A sequence of consonant notes: joy and calm
•  A sequence of dissonant notes: excitement, anger,
or unpleasantness
(Hunter et al., 2010; Hunter et al., 2008; Ali & Peynirciolu, 2010;
Webster & Weir, 2005)
TransProse
•  Three simultaneous piano melodies pertaining to the
dominant emotions.
•  Overall positiveness (or, negativeness) determines:
•  whether C major or C minor
•  base octave
•  Partition the novel into many small sections
•  For each section, if emotion density is high:
•  play many short notes
•  more dissonant notes
Pieces by TransProse
Three simultaneous piano melodies pertaining to the dominant emotions.
Examples
TransProse: www.musicfromtext.com
Music played 300,000 times since website launched in April 2014.
Pervasiveness of language data in the clinical domain
Prescriptions Recordings of physicians/patients

Medical ads
Patients
at home/on the go
Articles (PubMed)
Social media,
health forums
Medical records Patient-physician consultations
148!
Public and mental health
•  Cyber-bullying (Chen et al., 2012; Dadvar et al., 2013)
•  Health attributes at a community level (Johnsen, et

al., 2014; Eichstaedt et al., 2015)
•  Tracking well-being (Schwartz et al., 2013a; Paul & Dredze,

2011)
•  Developing robotic assistants and physio-

therapists for the elderly, disabled, and the sick
Public and mental health
•  Distress (Homan et al., 2014; Lehrman, Alm, & Proaño, 2012)
•  Suicide and risk factors (Jashinsky et al., 2014;

Matykiewicz, Duch, & Pestian, 2009; Pestian, Matykiewicz, Grupp-
Phelan, 2008; Poulin et al., 2014)
•  Depression and its severity (Schwartz et al., 2014;

Coppersmith, Drezde, & Harman, 2014; Howes, Purver, & McCabe,
2014; Lamers et al., 2014; Pennebaker, Mehl, & Niederhoffer,
2003; Rude, Gortner, & Pennebaker, 2004; Cherry, Mohammad, &
De Bruijn, 2012)
Dynamics of domestic abuse
•  Domestic abuse is a problem of pandemic

proportions
•  Taboo – domestic abuse discussions rare
•  Issues with survey methods
•  Use social media texts for analyzing reasons for

staying in or leaving abusive relationships, and
actions involving abusers and victims.
(Mohammad!and!Alm,!2015)! 151! (Schrading et al., 2015)

Twitter discussion about domestic abuse
#WhyIStayed -- why victims stayed in abusive relationships.
#WhyILeft -- why victims escaped them
From NLP analyses, micronarratives of staying and leaving emerge. Victims report:
•  staying due to, e.g., cognitive manipulation, dire financial straits, keeping the
nuclear family united, or experiencing shame.
•  leaving, e.g. when threats are made towards loved ones, gaining agency,
realizing their situation or self worth, or getting support from family/friends
(Mohammad!and!Alm,!2015)! 152! (Schrading et al., 2015)
Interplay between confidence and
correctness in diagnostic contexts
•  Medical misdiagnosis
•  Consequences for patients and unnecessary medical costs
•  Causes of errors
•  Lack of expertise, technical errors, and many more
•  Cognitive errors may be the most challenging to reduce
•  Studying the confidence-correctness interplay can

yield insights into relative importance of language
vs. multimodal markers
(Mohammad!and!Alm,!2015)! 153!(Bullard et al., 2014)

Diagnostic confidence as categories
in prediction problems
•  Correctness is reality
•  Confidence is belief
about correctness
(self-estimated)
Ideal vs. problematic

Comparing modalities
•  Language-based features appeared most informative—tap into physicians’

rich and tacit conceptual knowledge and understanding of a case.
•  Performance gains possible with added modalities.
•  Combined MM features " inappropriate representation. Need semantic
alignment without time overlap assumption.
Education and affect analysis
•  States such as attention and anxiety may impact learning.
Automated tutoring systems may monitor student users’ relevant
states, e.g., to personalize their learning experiences or as a
mechanism to deal with attrition in e-learning contexts.
•  A computer tutoring system that adjusted its response based on

the perceived confidence of the user resulted in more efficient
learning by the student. (Forbes-Riley & Litman, 2011)
•  Automated tutoring and student evaluation systems detect affect

based on user responses both to determine correctness of
responses and the user’s emotional state (Dogan, 2012; Li et al.,
2014). An intuitive finding is that learning improves when the
student is in a happy and calm state as opposed to anxious or
frustrated.
Business and affect analysis
•  Analysis of social media has been applied towards shaping
brand image, tracking customer response, and developing
automated dialogue systems for customer queries and
complaints (Ren & Quan, 2012; Gupta, Gilbert, & Fabbrizio,
2013; Bock et al., 2012).
•  This domain is expected to snowball in the coming years.
157!
Future Directions and
Wrap-up
Topics:
•  Emotions analysis for processing figurative language and metaphor
•  Understanding relationships between emotions
•  Enhancing evaluation procedures
•  Effective integration of NLP into multimodal affect analysis
•  Present and future tasks: What can emotion analysis do for your task?
Future directions
•  Affect and creative use of language (metaphors, sarcasm)

•  Detecting affect, detecting figurative language
•  What makes figurative language more emotional?
•  How is affect impacted by negators and other affect

modifiers?
•  Working on hundreds of affect categories

•  What is their relationship?
•  Is there a taxonomy?
•  Where can they be applied?
•  Deep neural networks and low-dimensional word

embeddings
Future directions
•  Rethinking evaluation procedures (Alm, 2011)

•  Extrinsic evaluation, usability and user satisfaction (Liu,
Lieberman, & Selker, 2003a), quality of life/safety
improvements
•  Increase importance of visualization in evaluation
•  Effective integration of linguistic data into multimodal

affect analysis
•  Multimodal data fusion approaches (Castellano, Kessous, &
Caridakis, 2008; Calvo & D’Mello, 2010; Gunes & Schuller, 2013)
•  Continue to address meaningful data synchronization
(cf. language and vision) and the multimodal fusion
challenges
•  Study strengths vs. weaknesses across modalities
•  Explore commonalities and differences between
language and other sensors by context
MOST IMPORTANTLY
•  What can analysis of affective states, or affect-

related conditions, activities, or experiences do
for you and your task? Please share your
thoughts, comments, and questions!
Thank you!
•  We look forward to continuing the discussion

during the rest of the conference or in future
interactions. Please stay in touch!
Tutorial resources
A note on references:
Citations to select relevant works are provided in two formats:
•  An appended standard reference list of works cited
•  An annotated bibliography with selected key sources
Please feel free to contact us about suggestions or
refinements to these lists, or to associated aspects of these
materials, for future versions.
A note on images:
Images in these materials tend to be from open repositories
or personal/own collaborative sources. We have attempted to
seek permission otherwise. Nonetheless, if you notice that
image has slipped through the cracks and ought not to be
included, please contact us so we can remove it in later
versions of the materials.
(Muhammad and Alm, 2015)
Annotated Bibliography
1. Alm, C. O. (2012). The role of affect in the computational modeling of

natural language. Language and Linguistics Compass (Computational and
Mathematical), 6(7), 416-430. SURVEY (GENERAL).
Written for interdisciplinary readers across career-levels, this succinct survey
article provides literature review and discussion about theoretical background
and applied topics of interest for analyzing affect in linguistic corpora and for
incorporating mechanisms for processing affect into language technology.
2. Cahn, J. (1990). The generation of affect in synthesized speech. Journal of

the American Voice I/O Society, 8, 1-19. SEMINAL/CLASSICAL.
This classical reading represents an early example of work on synthesizing
emotional speech. It describes the Affect Editor, a tool for generating
expressive speech, including a diverse set of speech parameters used for
modeling, and it discusses the implementation and results of human-based
evaluation. The work suggested that recognizable emotions could be
synthesized. Some of the findings included that sad stimuli were particularly
well identified, that lexical contents of sentences might influence perception,
and that category mix-up might occur between more affectively similar
concepts such as angry and disgusted. The paper also briefly reported on
observing individual preferences. Affective expressiveness continues to be a
research topic in speech science and technology communities.
3. Calvo, R. A., and D’Mello, S. (2010). Affect detection: An interdisciplinary

review of models, methods, and their applications. IEEE Transactions on
Affective Computing, 1(1), 18-37. SURVEY (GENERAL).
This is a comprehensive survey article useful for readers who wish to deepen
their understanding of the interdisciplinary landscape of work involved with
detecting affect, the affective sciences, and affective computing. Given
theoretical background and frameworks for modeling affect, the article dives
into an overview of studied human modalities that contribute affect signals
(with particular sections dedicated to spoken and written language),
including methods, resources, and multimodal integration. Discussion
synthesizes important topics and current/future research directions. The
article presents an opportunity to become familiar with IEEE Transactions on
Affective Computing.
4. Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., and
Kuksa, P. (2011). Natural language processing (almost) from scratch. Journal
of Machine Learning Research, 12, 2493–2537. EMPIRICAL (SYSTEM).
This paper presents a neural network framework that was applied to part-of-
speech tagging, chunking, named entity recognition, semantic role labeling,
and some other NLP tasks. It is one of the more recent papers on deep
learning in NLP that eschews task-specific engineering in favour of learning
common internal representations from data. Even though affect-related tasks
are not directly addressed in this paper, several deep learning papers on
valence classification draw inspiration from this work.
! 1
5. Coppersmith, G., Dredze, M., and Harman, C. (2014). Quantifying mental

health signals in Twitter. Proceedings of the Workshop on Computational
Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality
at the 52nd Annual Meeting of the Association for Computational Linguistics,
Baltimore, MD, USA, 51-60. APPLICATION (HEALTH).
This study from a recent workshop exemplifies the interest in textual data
and social media for exploring affect-related phenomena in the domain of
public health. Presenting an approach for gathering and studying microblog
data for conditions such as depression and PTSD, the paper also discusses
the opportunity for complementary use of natural language processing
techniques in relation to a tradition of survey analysis in health contexts. The
paper conveys some of the challenges (generational uses of social media,
less mention of uncommon conditions, etc.) as well as the usefulness of
interdisciplinary collaboration in this area.
6. Cornelius, R. R. (2000). Theoretical approaches to emotion. Proceedings

of the ISCA ITRW on Speech and Emotion (SpeechEmotion-2000),
Newcastle, Northern Ireland, UK, 3-10. SURVEY (THEORY).
Cornelius advocates for the need to take theoretical accounts into
consideration in emotion scholarship. He straightforwardly introduces four
“perspectives” from the discipline of psychology: Darwinian, Jamesian,
cognitive, and social constructivist, and also discusses how these views relate
to each other. While positioning the discussion within emotional speech
research, the author explicates some of the benefits of understanding where
one’s work fits theoretically and how theoretical views are merging, including
how that may aid the appreciation of assumptions involved or influence
investigatory questions and insights to evolve.
7. Cowie, R., Douglas-Cowie, E., Martin, J.-C., and Devillers, L. (2010). The
essential role of human databases for learning in and validation of affectively
competent agents. In Scherer, K. R., Bänziger, T., and Roesch, E. B. (Eds.)
Blueprint for Affective Computing: A Sourcebook. Oxford: Oxford University
Press, 151-165. SURVEY (RESOURCE).
This book chapter about data resources touches upon many of the issues and
topics under discussion with respect to affect data development, from the
perspective of the affect sciences and affective computing. Database
examples are covered for distinct modalities (albeit sparsely for text-oriented
work). The chapter includes summarizing projections about next
developments in this area.
8. Eichstaedt, J. C., Schwartz, H. A., Kern, M. L., Park, G., Labarthe, D. R.,
Merchant, R. M., Jha, S., Agrawal, M., Dziurzynski, L. A., Sap, M., Weeg, C.,
Larson, E. E., Ungar, L. H., and Seligman, M. E. (2015). Psychological
language on Twitter predicts county-level heart disease mortality.
Psychological Science, 26(2), 159-169. APPLICATION (HEALTH).
Traditional approaches for determining psychological states of people involve
in-person or phone conversations. Such approaches are time intensive and
expensive. This paper aims at determining psychological state, at the
! 2
community level, from tweets posted by the community. Specifically, it

analyzes language in tweets and finds correlations of certain features with
heart disease rates at the level of counties. Lexicons for anger, anxiety,
positive and negative emotions, positive and negative social relationships,
and engagement and disengagement were found to be useful. This is an
interesting example of bridging the analysis of language (in this case tweets)
with non-linguistic information (in this case heart disease data).
9. Kiritchenko, S., Zhu, X., and Mohammad, S. M. (2014). Sentiment

analysis of short informal texts. Journal of Artificial Intelligence Research, 50,
723–762. EMPIRICAL (SYSTEM).
This paper gives details about the NRC-Canada system that came first in
various sentiment-related shared tasks in Sem-Eval-2013 and 2014. A
number of different kinds of features were used. Ablation experiments
showed that the Twitter-specific valence-association lexicons were the most
useful. These automatically generated lexicons capture non-standard
language, such as creative spellings and word elongations. Methods to
capture impact of negation on sentiment are also described. The paper
additionally describes a maximum difference scaling approach to obtain
reliable fine-grained annotations of sentiment.
10. Liu, H., Lieberman, H., and Selker, T. (2003). A model of textual affect
sensing using real-world knowledge. Proceedings of the International
Conference on Intelligent User Interfaces, Miami, FL, USA, 125-132.
SEMINAL/CLASSICAL.
In this early paper on affect processing with text, the main application of
interest was an affective email interface (EmpathyBuddy). The work involved
interpreting affect in terms of fundamental emotion categories, using a
textual resource of commonsense knowledge (Open Mind Commonsense). A
user study evaluated the email client. Users interacted with three client
versions, including the sensing-based version. They assessed the system for
“entertainment, interactivity, intelligence, and adoption”. The results
suggested that the authors’ main approach was perceived as more
intelligent, adoptable, and interactive.
11. Mohammad, S. M., and Turney, P. D. (2013). Crowdsourcing a word–

emotion association lexicon. Computational Intelligence, 29 (3), 436-465.
EMPIRICAL (RESOURCE).
This paper describes the creation of a large word—affect association lexicon
by crowdsourcing. Several techniques are employed for quality control, most
notably with the use of a separate word-choice question that ascertains
whether the annotator knows the meaning of the target word. The question
also guides the annotator to the desired sense of the word for which
annotations are needed. The resulting lexicon, the NRC Emotion Lexicon, has
entries for over 14,000 words and about 25,000 word senses. Each instance
is marked for associations with eight emotions, as well as positive and
negative sentiment. The lexicon is widely used by researchers and system
builders for various affect-related tasks.
! 3
12. Mohammad, S. M., and Kiritchenko, S. (2013). Using nuances of emotion

to identify personality. Proceedings of the International Conference on
Weblogs and Social Media (ICWSM-13), Boston, MA, 27-30. EMPIRICAL
(SYSTEM).
This paper describes the collection and use of tweets with emotion word
hashtags for automatic emotion detection. Experiments show that the
emotion word hashtags act as good labels of emotions in the rest of the
tweet. Thus the data can be used for training machine learning systems for
emotion classification. The success of this approach also means that one can
now quickly compile training data for any emotion which is used as a hashtag
in tweets. Experiments are performed in an extrinsic task for personality trait
classification, where it is shown that emotion-based features from hundreds
of emotion categories are more useful than using features from a handful of
affect categories (such as positive and negative sentiment, or the Big Six
emotion categories).
13. Neviarouskaya, A., Prendinger, H., and Ishizuka, M. (2010). Recognition

of affect, judgment, and appreciation in text. Proceedings of the 23rd
International Conference on Computational Linguistics, Beijing, China, 806-
814. EMPIRICAL (SYSTEM).
Even though a majority of current approaches in NLP are statistical, rule-
based systems are more interpretable, for example, for understanding why a
sentence was classified as having a certain emotion by the system. This
paper presents a rule-based system for detecting emotions at sentence level.
It employs a number of manually created lexicons for attitude, affect
modifiers, and even a lexicon that captures the confidence signified by modal
verbs. At the heart of the system is a method to combine affect-related
information from different pieces of the text using rules. Developing
compositional models statistically is a major area of research these days, and
this work can be a source of ideas in developing composition models that
capture affect appropriately.
14. Ortony, A., Clore, G. L., and Collins, A. (1990). The Cognitive Structure
of Emotions. Cambridge: Cambridge University Press. SEMINAL/CLASSICAL.
Several authors have proposed mutually conflicting theories about emotions,
and till date many key aspects of emotions are hotly debated. This book by
Ortony, Clore, and Collins presents one such theoretical framework that
argues that emotions are valence reactions. The valence reaction is broken
down into several sub-categories and these subcategories are broken down
into further sub-categories, based on whether the valence reaction was to
the consequences of events, aspects of objects, whether the person approves
or disapproves it, etc. Ideas on the theoretical underpinnings of emotions
and different kinds of valence reaction can be helpful for developing
instructions for affect annotations, as well as for developing features that can
be useful in automatic affect classification.
! 4
15. Osgood, C. E., Suci, G. J., and Tannenbaum, P. (1957). The

Measurement of Meaning. Urbana, USA: University of Illinois Press.
SEMINAL/CLASSICAL.
This work studies the nature of meaning. One of its most influential
experiments involves asking people to rate the meanings of concepts along
several dimensions such as fair—unfair, strong—weak, safe—dangerous, etc.
Factor analysis of the responses is used to show that the three dimensions
conveying most of the variance in meaning across concepts are that of
evaluativeness (good—bad), potency (strong—weak), and activity (active—
passive). This work has influenced scholars in a number of fields including
linguistics, psychology, mass communications, and natural language
processing.
16-17. Brief mention of two book manuscripts. SURVEY (GENERAL).

Scherer, K. R., Bänziger, T., and Roesch, E. B. (Eds.) (2010). Blueprint for
Affective Computing: A Sourcebook. Oxford: Oxford University Press.
In this collection, the editors gather nineteen chapters into seven sections on
a range of topics pertinent for exploring affect in human- and machine-
oriented research. Chapters range from topics such as “Emotions in
interpersonal interactions” (Parkinson) to “Emotion in artificial neural
networks” (Roesch, Korsten, Fragopanagos, and Taylor), with five chapters
specifically devoted to “Approaches to an implementation of affectively
competent agents”.
Schuller, B., and Batliner, A. (2014). Computational Paralinguistics: Emotion,
Affect and Personality in Speech and Language Processing. Chichester: Wiley.
This recent book is divided into “Foundations” and “Modelling”, each
comprising various chapters, such as “Taxonomies”, “Functional aspects”,
and “Corpus engineering” in the first part, and “Linguistic features”,
“Machine-based modelling”, and “’Hands-on’: Existing toolkits and practical
tutorial” in the second part. In the preface, the authors explain that a goal is
“to provide the reader with a sort of map presenting an overview of the field,
and useful for finding one’s way through. The scale of this map is medium-
sized, and we can only display a few of the houses in this virtual
paralinguistic ‘city’ with their interiors, on an exemplary basis.”
18. Strapparava, C., and Mihalcea, R. (2007). SemEval-2007 Task 14:

Affective text. Proceedings of SemEval-2007, Prague, Czech Republic, 70–74.
EMPIRICAL (RESOURCE).
This is a task-description paper of an early shared task competition on
automatically detecting valence and emotions in text. The dataset chosen
was a collection of newspaper headlines. Annotators were asked to give
scores between 0 and 100 for each of the Big Six emotions and positive and
negative valence. No training data was provided, and so only unsupervised
systems were able to participate. Nonetheless, the testset created as part of
this shared task has subsequently been used by supervised systems by
splitting it into new train—test partitions. One of the key distinctions of this
work compared to the shared tasks proposed in the last few years is that this
is one of the few datasets that has fine-grained annotation for the degree of
! 5
affect. Several applications would benefit from a system that can predict
the degree of affect in text.
19. Turney, P.D. (2002). Thumbs up or thumbs down? Semantic orientation

applied to unsupervised classification of reviews. Proceedings of the 40th
Annual Meeting of the Association for Computational Linguistics, Philadelphia,
PA, 417-424. EMPIRICAL (SYSTEM).
This paper presents a way to classify customer reviews as positive
(recommended) or negative (not recommended). At the heart of the method
is a way to determine the degree of positiveness (or negativeness) of a word
by calculating the mutual information score between it and sets of positive
and negative seed terms. This fundamental approach is still used (possibly
with minor modification) for creating sentiment and emotion association
lexicons.
20. Wiebe, J., Wilson, T., Bruce, R., Bell, M., and Martin, M. (2004). Learning
subjective language. Computational Linguistics, 30(3), 277–308.
SEMINAL/CLASSICAL.
This article presents early, comprehensive efforts to automatically detect
subjective language. Supervised classification is performed on a number of
datasets using various features, including low-frequency words, collocations,
and words that are distributionally similar to pre-chosen seed words. The
paper is also an excellent resource for understanding the principles
underpinning subjective language.
! 6
References
• Ali, O., and Peynirciolu, Z. (2010). Intensity of emotions conveyed and elicited by
familiar and unfamiliar music. Music Perception: An Interdisciplinary Journal, 27(3),
177–182.
• Alm, C. O. (2009). Affect in Text and Speech. Saarbrücken: VDM Verlag.
• Alm, C. O. (2010). Characteristics of high agreement affect annotation in text.
Proceedings of the 4th Linguistic Annotation Workshop at the 48th Annual Meeting of
the Association for Computational Linguistics, Uppsala, Sweden, 118-122.
• Alm, C. O. (2011). Subjective natural language problems: Motivations, applications,
characterizations, and implications. Proceedings of the 49th Annual Meeting of the
Association for Computational Linguistics, Portland, OR, 107-112.
• Alm, C. O. (2012). The role of affect in the computational modeling of natural
language. Language and Linguistics Compass (Computational and Mathematical),
6(7), 416-430.
• Alm, C. O., and Sproat, R. (2005). Emotional sequencing and development in fairy
tales. In Tao, J., Tan, T., and Picard, R. W. (Eds.) First International Conference on
Affective Computing and Intelligent Interaction, Beijing, China. LNCS 3784. Berlin
Heidelberg: Springer-Verlag, 668-674.
• Alm-Arvius, C. (1998). Introduction to Semantics. Lund: Studentlitteratur.
• Aman, S., and Szpakowicz, S. (2007). Identifying expressions of emotion in text. In
Matoušek, V., and Mautner, P. (Eds.) Text, Speech and Dialogue. LNCS 4629. Berlin
Heidelberg: Springer-Verlag, 196–205.
• Anagnostopoulos, C.N., Iliou, T., and Giannoukos, I. (2015). Features and
classifiers for emotion recognition from speech: A survey from 2000 to 2011.
Artificial Intelligence Review, 43(2), 155–177.
• Anand, P., Walker, M., Abbott, R., Tree, J. E. F., Bowmani, R., and Minor, M.
(2011). Cats rule and dogs drool!: Classifying stance in online debate. Proceedings
of the 2nd Workshop on Computational Approaches to Subjectivity and Sentiment
Analysis, Portland, OR, 1–9.
• Andreasen, N. G., and Pfohl, B. (1976). Linguistic analysis of speech in affective
disorders. Archives of General Psychiatry, 33(11), 1361–1367.
• Avello, D. G. (2012). ”I wanted to predict elections with Twitter and all I got was
this lousy paper” – A balanced survey on election prediction using Twitter data.
arXiv, 1204.6441.
• Baker, C. F., Fillmore, C. J., and Lowe, J. B. (1998). The Berkeley FrameNet
project. Proceedings of the Annual Meeting of the Association for Computational
Linguistics, Stroudsburg, PA, 86–90.
• Barrett, L. F. (2006). Are emotions natural kinds? Perspectives on Psychological
Science, 1(1), 28–58.
• Bellegarda, J. (2010). Emotion analysis using latent affective folding and
embedding. Proceedings of the NAACL-HLT 2010 Workshop on Computational
Approaches to Analysis and Generation of Emotion in Text, Los Angeles, CA.
• Bermingham, A., and Smeaton, A. F. (2011). On using Twitter to monitor political
sentiment and predict election results. Proceedings of the IJCNLP Workshop on
Sentiment Analysis where AI meets Psychology, Chiang Mai, Thailand, 2–10.
• Besnier, N. (1990). Language and affect. Annual Review of Anthropology, 19, 419-
451.
1
• Bethamcherla, V. P., Paul, W., Alm, C. O., Bailey, R., Geigel, J., and Wang, L.
(2015). Face-speech sensor fusion for non-invasive stress detection. Proceedings of
1st Joint Conference on Facial Analysis, Animation and Audio-Visual Speech
Processing, Vienna, Austria. Forthcoming.
• Bock, R., Gluge, S., Wendemuth, A., Limbrecht, K., Walter, S., Hrabal, D., and
Traue, H. C. (2012). Intraindividual and interindividual multimodal emotion
analyses in human-machine-interaction. IEEE International Multi-Disciplinary
Conference on Cognitive Methods in Situation Awareness and Decision Support
(CogSIMA), New Orleans, LA, 59–64.
• Bogdanova, D., Rosso, P., and Solorio, T. (2012). On the impact of sentiment and
emotion based features in detecting online sexual predators. Proceedings of the 3rd
Workshop in Computational Approaches to Subjectivity and Sentiment Analysis,
Jeju, Korea, 110–118.
• Bollen, J., Pepe, A., and Mao, H. (2011). Modeling public mood and emotion:
Twitter sentiment and socio-economic phenomena. Proceedings of the 5th
International AAAI Conference on Weblogs and Social Media, Barcelona, Spain,
450-453.
• Boneva, B., Kraut, R., and Frohlich, D. (2001). Using e-mail for personal
relationships. American Behavioral Scientist, 45(3), 530–549.
• Bradley, M. M., and Lang, P. J. (1994). Measuring emotion: The self-assessment
manikin and the semantic differential. Journal of Behavioral Therapy and
Experimental Psychiatry, 25(1), 49-59.
• Bradley, M. M., and Lang, P. J. (1999). Affective norms for English words (ANEW):
Stimuli, instruction manual and affective ratings. Technical report C-1, Gainesville,
FL. The Center for Research in Psychophysiology, University of Florida.
• Bradley, M. M., and Lang, P. J. (2007). Affective norms for English text (ANET):
Affective ratings of text and instruction manual. Tech. Rep. No. D-1. University of
Florida, Gainesville, FL.
• Brody, S., and Diakopoulos, N. (2011). Cooooooooooooooollllllllllllll!!!!!!!!!!!!!!:
Using word lengthening to detect sentiment in microblogs. Proceedings of the
Conference on Empirical Methods in Natural Language Processing, Edinburgh,
Scotland, 562–570.
• Brooks, M., Kuksenok, K., Torkildson, M. K., Perry, D., Robinson, J. J., Scott, T. J.,
Anicello, O., Zukowski, A., Harris, P., and Aragon, C. R. (2013). Statistical affect
detection in collaborative chat. Proceedings of the 2013 Conference on Computer
Supported Cooperative Work, San Antonio, TX, 317–328.
• Bullard, J., Alm, C. O., Yu, Q., Shi, P., and Haake, A. (2014). Towards multimodal
modeling of physicians’ diagnostic confidence and self-awareness using medical
narratives. Proceedings of 25th International Conference on Computational
Linguistics, Dublin, Ireland, 1718–1727.
• Bühler, K. (1934). Sprachtheorie: Die Darstellungsfunktion der Sprache. Stuttgart:
Gustav Fischer Verlag.
• Cahn, J. (1990). The generation of affect in synthesized speech. Journal of the
American Voice I/O Society, 8, 1-19.
• Caldwell, M. and Peplau, L. (1982). Sex differences in same-sex friendships. Sex
Roles, 8, 721-732.
2
• Calvo, R. A., and D’Mello, S. (2010). Affect detection: An interdisciplinary review of

models, methods, and their applications. IEEE Transactions on Affective
Computing, 1(1), 18-37.
• Carvalho, P., Sarmento, L., Silva, M. J., and De Oliveira, E. (2009). Clues for
detecting irony in user-generated contents: Oh...!! It’s “so easy”;-). Proceedings of
the 1st International CIKM Workshop on Topic-sentiment Analysis for Mass Opinion,
Hong-Kong, China, 53–56.
• Castellano, G., Kessous, L., and Caridakis, G. (2008). Emotion recognition through
multiple modalities: Face, body gesture, speech. In Peter, C., and Beale, R. (Eds.)
Affect and Emotion in Human-Computer Interaction: From Theory to Applications.
LNCS 4868. Berlin Heidelberg: Springer-Verlag, 92-103.
• Chaffar, S., and Inkpen, D. (2011). Using a heterogeneous dataset for emotion
analysis in text. In Butz, C., and Lindgras, P. (Eds.) Advances in Artificial
Intelligence. LNCS 6657. Berlin Heidelberg: Spring-Verlag, 62-67.
• Chen, Y., and Skiena, S. (2014). Building sentiment lexicons for all major
languages. Proceedings of the 52nd Annual Meeting of the Association for
Computational Linguistics, Baltimore, MD, 383–389.
• Chen, Y., Zhou, Y., Zhu, S., and Xu, H. (2012). Detecting offensive language in
social media to protect adolescent online safety. IEEE International Conference on
Privacy, Security, Risk and Trust (PASSAT) and 2012 International Conference on
Social Computing (SocialCom), Amsterdam, 71–80.
• Cherry, C., Mohammad, S. M., and De Bruijn, B. (2012). Binary classifiers and
latent sequence models for emotion detection in suicide notes. Biomedical
Informatics Insights, 5(Suppl 1), 147.
• Chetviorkin, I., Moscow, L. G., and Loukachevitch, N. (2014). Two-step model for
sentiment lexicon extraction from Twitter streams. Proceedings of the Workshop on
Computational Approaches to Subjectivity and Sentiment Analysis, Baltimore, MD,
67–72.
• Cohen, S. H. (2003). Maximum difference scaling: Improved measures of
importance and preference for segmentation. Technical report, Sawtooth Software,
Inc., 1–17
• Collier, G. (1985). Emotional Expression. Hillsdale, N. J.: Lawrence Erlbaum
Associates.
• Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., and Kuksa, P.
(2011). Natural language processing (almost) from scratch. Journal of Machine
Learning Research, 12, 2493–2537.
• Conover, M. D., Ratkiewicz, J., Francisco, M., Gonc, B., Flammini, A., and Menczer,
F. (2011a). Political polarization on Twitter. Networks, 133(26), 89–96.
• Conover, M. D., Goncalves, B., Ratkiewicz, J., Flammini, A., and Menczer, F.
(2011b). Predicting the political alignment of Twitter users. IEEE 3rd International
Conference on Privacy Security Risk and Trust and IEEE 3rd International
Conference on Social Computing, 192–199.
• Coppersmith, G., Dredze, M., and Harman, C. (2014). Quantifying mental health
signals in Twitter. Proceedings of the Workshop on Computational Linguistics and
Clinical Psychology: From Linguistic Signal to Clinical Reality at the 52nd Annual
Meeting of the Association for Computational Linguistics, Baltimore, MD, USA, 51-
60.
• Cornelius, R. R. (2000). Theoretical approaches to emotion. Proceedings of the
3
ISCA ITRW on Speech and Emotion (SpeechEmotion-2000), Newcastle, Northern

Ireland, UK, 3-10.
• Cowie, R., Douglas-Cowie, E., Martin, J.-C., and Devillers, L. (2010). The essential
role of human databases for learning in and validation of affectively competent
agents. In Scherer, K. R., Bänziger, T., and Roesch, E. B. (Eds.) Blueprint for
Affective Computing: A Sourcebook. Oxford: Oxford University Press, 151-165.
• Dadvar, M., Trieschnigg, D., Ordelman, R., and de Jong, F. (2013). Improving
cyberbullying detection with user context. In Serdyukov et al. (Eds.) Advances in
Information Retrieval. LNCS 7814. Berlin Heidelberg: Springer-Verlag, 693–696.
• Darwin, C. (1998 [1890]). The expression of the emotions in man and animals
[selected excerpts]. In Jenkins, J. M., Oatley, K., and Stein, N. L. (Eds.) Human
Emotions: A Reader. Oxford: Blackwell, 13-20.
• Davidson, L. and Duberman, L. (1982). Friendship: Communication and
interactional patterns in same-sex dyads. Sex Roles, 8, 809–822.
• Davis, H., and Mohammad, S. (2014). Generating music from literature.
Proceedings of the 3rd Workshop on Computational Linguistics for Literature (CLFL),
Gothenburg, Sweden, 1–10.
• Deaux, K. and Major, B. (1987). Putting gender into context: An interactive model
of gender-related behavior. Psychological Review, 94(3), 369–389.
• Dogan, H. (2012). Emotion, confidence, perception and expectation case of
mathematics. International Journal of Science and Mathematics Education, 10(1),
49–69.
• Douglas-Cowie, E., Cowie, R., Sneddon, I., Cox, C., Lowry, O., McRorie, M., Martin,
J.-C., Devillers, L., Abrilian, S., Batliner, A., Amir, N., Karpousiz, K., and Martin, J.-
C. (2007). The HUMAINE database: Addressing the collection and annotation of
naturalistic and induced emotional data. In Paiva, A., Prada, R., and Picard, R. W.
(Eds.) Second International Conference on Affective Computing and Intelligent
Interaction, Lisbon, Portugal. LNCS 4738. Berlin Heidelberg: Springer-Verlag, 488-
500.
• Eagly, A. and Steffen, V. (1984). Gender stereotypes stem from the distribution of
women and men into social roles. Journal of Personality and Social Psychology,
46(4), 735–754.
• Eichstaedt, J. C., Schwartz, H. A., Kern, M. L., Park, G., Labarthe, D. R., Merchant,
R. M., Jha, S., Agrawal, M., Dziurzynski, L. A., Sap, M., Weeg, C., Larson, E. E.,
Ungar, L. H., and Seligman, M. E. (2015). Psychological language on Twitter
predicts county-level heart disease mortality. Psychological Science, 26(2), 159-
169.
• Ekman, P. (1992). An argument for basic emotions. Cognition and Emotion, 6(3),
169–200.
• Ekman, P. (1994). All emotions are basic. In Ekman, P., and Davidson, R. J. (Eds.)
The Nature of Emotion: Fundamental Questions. New York: Oxford University
Press, 15-19.
• Ekman, P., and Friesen, W. V. (1998 [1971]). Constants across culture in the face
and emotion. In Jenkins, J. M., Oatley, K., and Stein, N. L. (Eds.) Human Emotions:
A Reader. Oxford: Blackwell, 63-72.
• Esuli, A., and Sebastiani, F. (2006). SentiWordNet: A publicly available lexical
resource for opinion mining. Proceedings of 5th Conference on Language Resources
and Evaluation, Genova, IT, 417–422.
4
• Foolen, A. (1997). The expressive function of language: Towards a cognitive

semantic approach. In Niemeier, S., and Dirven, R. (Eds.) The Language of
Emotions: Conceptualization, Expression, and Theoretical Foundation, Amsterdam:
John Benjamins, 15–31.
• Forbes-Riley, K., and Litman, D. (2011). Designing and evaluating a wizarded
uncertainty-adaptive spoken dialogue tutoring system. Computer Speech and
Language, 25(1), 105-126.
• Francisco, V., and Gervás, P. (2006). Automated mark up of affective information in
English texts. In Sojka, P., Kopeček, I., and Pala, K. (Eds.) Text, Speech and
Dialogue. LNCS 4188. Berlin Heidelberg: Springer-Verlag, 375–382.
• Frege, G. (1952 [1892]). On sense and reference. In Geach, P., and Black, M.
(Eds.) Translations from the Philosophical Writings of Gottlob Frege. Oxford:
Blackwell, 56–78.
• Frijda, N. H. (1988). The laws of emotion. American Psychologist, 43(5), 349.
• Genereux, M., and Evans, R. P. (2006). Distinguishing affective states in weblogs.
Proceedings of AAAI-2006 Spring Symposium on Computational Approaches to
Analysing Weblogs, Stanford, CA, 27– 29.
• Go, A., Bhayani, R., and Huang, L. (2009). Twitter sentiment classification using
distant supervision. Technical report, Stanford University. 1–6.
• Gobron, S., Ahn, J., Paltoglou, G., Thelwall, M., and Thalmann, D. (2010). From
sentence to emotion: A real-time three-dimensional graphics metaphor of emotions
extracted from text. The Visual Computer, 26(6-8), 505–519.
• Golbeck, J., and Hansen, D. (2011). Computing political preference among Twitter
followers. Proceedings of the SIGCHI Conference on Human Factors in Computing
Systems, New York, NY, 1105–1108.
González-Ibáñez, R., Muresan, S., and Wacholder, N. (2011). Identifying sarcasm
in Twitter: A closer look. Proceedings of the 49th Annual Meeting of the
Association for Computational Linguistics, Portland, OR, 581–586.
• Grijalva, E., and Harms, P. D. (2014). Narcissism: An integrative synthesis and
dominance complementarity model. The Academy of Management Perspectives,
28(2), 108–127.
• Gunes, H., and Schuller, B. (2013). Categorical and dimensional affect analysis in
continuous input: Current trends and future directions. Journal of Image and Vision
Computing, 31(2), 120-136.
• Gupta, N., Gilbert, M., and Fabbrizio, G. D. (2013). Emotion detection in email
customer care. Computational Intelligence, 29(3), 489–505.
• Halliday, M. A. K. (1996). Linguistic function and literary style: An inquiry into the
language of William Golding’s The Inheritors. In Weber, J. J. (Ed.) The Stylistics
Reader: From Roman Jakobson to the Present. London: Arnold, 56–86.
• Hartner, M. (2013). The lingering after-effects in the readers mind – An
investigation into the affective dimension of literary reading. Journal of Literary
Theory Online. [Review of Burke, M. (2011). Literary Reading, Cognition and
Emotion. An Exploration of the Oceanic Mind. New York/London: Routledge.]
• Hasan, K. S., and Ng, V. (2013). Stance classification of ideological debates: Data,
models, features, and constraints. Proceedings of the 6th International Joint
Conference on Natural Language Processing, Nagoya, Japan, 1348–1356.
• Hatzivassiloglou, V., and McKeown, K. R. (1997). Predicting the semantic
orientation of adjectives. Proceedings of the 8th Conference of European Chapter of
5
the Association for Computational Linguistics, Madrid, Spain, 174–181.

• Hatzivassiloglou, V., and Wiebe, J. M. (2000). Effects of adjective orientation and
gradability on sentence subjectivity. Proceedings of the 18th Conference on
Computational Linguistics, Saarbrucken, Germany, 299–305.
• Hochberg, L., Alm, C. O., Rantanen, E. M., DeLong, C. M., and Haake, A. (2014a).
Decision style in a clinical reasoning corpus. Proceedings of the 2014 Workshop on
Biomedical Natural Language Processing, Baltimore, MD, USA, 83-87.
• Hochberg, L., Alm, C. O., Rantanen, E. M., Yu, Q., DeLong, C. M., and Haake, A.
(2014b). Towards automatic annotation of clinical decision-making style.
Proceedings of LAW VIII - The 8th Linguistic Annotation Workshop at the 25th
International Conference on Computational Linguistics, Dublin, Ireland, 129-138.
• Holzman, L. E., and Pottenger, W. M. (2003). Classification of emotions in internet
chat: An application of machine learning using speech phonemes. Technical report,
Leigh University.
• Homan, C. M., Johar, R., Liu, T., Lytle, M., Silenzio, V., and Alm, C. O. (2014).
Toward macro-insights for suicide prevention: Analyzing fine-grained distress at
scale. Proceedings of the Workshop on Computational Linguistics and Clinical
Psychology: From Linguistic Signal to Clinical Reality at the 52nd Annual Meeting of
the Association for Computational Linguistics, Baltimore, MD, USA, 107-117.
• Howes, C., Purver, M., and McCabe, R. (2014). Linguistic indicators of severity and
progress in online text-based therapy for depression. Proceedings of the Workshop
on Computational Linguistics and Clinical Psychology: From Linguistic Signal to
Clinical Reality at the 52nd Annual Meeting of the Association for Computational
Linguistics, Baltimore, MD, USA, 7-16.
• Hu, M., and Liu, B. (2004). Mining and summarizing customer reviews. Proceedings
of the 10th ACM SIGKDD International Conference on Knowledge Discovery and
Data Mining, New York, NY, USA, 168–177.
• Hunter, P.G., Schellenberg, G., and Schimmack, U. (2008). Mixed affective
responses to music with conflicting cues. Cognition & Emotion, 22(2):327–352.
• Hunter, P.G., Schellenberg, G., and Schimmack, U. (2010). Feelings and
perceptions of happiness and sadness induced by music: Similarities, differences,
and mixed emotions. Psychology of Aesthetics, Creativity, and the Arts, 4(1):47.
• Irsoy, O., and Cardie, C. (2014). Deep recursive neural networks for
compositionality in language. Proceedings of Advances in Neural Information
Processing Systems, Montreal, Quebec, 2096–2104.
• Jakobson, R. (1996). Closing statement: Linguistics and poetics. In Weber, J. J.
(Ed.) The Stylistics Reader: From Roman Jakobson to the Present. London: Arnold,
10-35.
• Jashinsky, J., Burton, S. H., Hanson, C. L., West, J., Giraud-Carrier, C., Barnes, M.
D., and Argyle, T. (2014). Tracking suicide risk factors through Twitter in the US.
Crisis, 35(1), 51-59.
• Jia, L., Yu, C., and Meng, W. (2009). The effect of negation on sentiment analysis
and retrieval effectiveness. Proceedings of the 18th ACM Conference on Information
and Knowledge Management, New York, NY, USA, 1827–1830.
• John, D., Boucouvalas, A. C., and Xu, Z. (2006). Representing emotional
momentum within expressive internet communication. Proceedings of the 24th
IASTED International Conference on Internet and Multimedia Systems and
Applications, Anaheim, CA, 183–188.
6
• Johnsen, J.A. K., Vambheim, S. M., Wynn, R., and Wangberg, S. C. (2014).
Language of motivation and emotion in an internet support group for smoking
cessation: explorative use of automated content analysis to measure regulatory
focus. Psychology Research and Behavior Management, 7, 19–29.
• Jurgens, D., Mohammad, S. M., Turney, P., and Holyoak, K. (2012). Semeval-2012
Task 2: Measuring degrees of relational similarity. Proceedings of the 6th
International Workshop on Semantic Evaluation, SemEval’12, Montreal, Canada,
356–364.
• Karpathy, A., Joulin, A., and Li, F.-F. (2014). Deep fragment embeddings for
bidirectional image sentence mapping. Proceedings of Advances in Neural
Information Processing Systems, Montreal, Quebec, 1889–1897.
• Katz, J. J., and Fodor, J. A. (1963). The structure of a semantic theory. Language,
39(2), 170–210.
• Kilgarriff, A. (1997). “I don’t believe in word senses”. Computers and the
Humanities, 31(2), 91–113.
• Kirange, D. K., and Deshmukh, R. R.. (2013). Emotion classification of news
headlines using SVM. Asian Journal of Computer Science and Information
Technology, 2(5), 104-106.
• Kiritchenko, S., Zhu, X., and Mohammad, S. M. (2014). Sentiment analysis of short
informal texts. Journal of Artificial Intelligence Research, 50, 723–762.
• Kleres, J. (2011). Emotions and narrative analysis: A methodological approach.
Journal for the Theory of Social Behaviour, 41(2), 182–202.
• Kramer, A. D. (2012). The spread of emotion via Facebook. Proceedings of the
SIGCHI Conference on Human Factors in Computing Systems, 767–770.
• Ku, L.W., Liang, Y.T., and Chen, H.-H. (2006). Opinion extraction, summarization
and tracking in news and blog corpora. Proceedings of the AAAI Spring Symposium:
Computational Approaches to Analyzing Weblogs, Vol. 100107.
• Kunneman, F., Liebrecht, C., and van den Bosch, A. (2014). The (un)predictability
of emotional hashtags in Twitter. Proceedings of the 5th Workshop on Language
Analysis for Social Media, Gothenburg, Sweden, 26–34.
• Lamers, S. M. A., Truong, K. P., Steunenberg, B., de Jong, F. and Westerhof, G. J.
(2014). Applying prosodic speech features in mental health care: An exploratory
study in a life-review intervention for depression. Proceedings of the Workshop on
Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical
Reality at the 52nd Annual Meeting of the Association for Computational Linguistics,
Baltimore, MD, USA, 61-68.
• Lampos, V., Preotiuc-Pietro, D., and Cohn, T. (2013). A user-centric model of
voting intention from social media. Proceedings of the 51st Annual Meeting of the
Association for Computational Linguistics, Sofia, Bulgaria, 993–1003.
• Lassen, D. S., and Brown, A. R. (2011). Twitter: The electoral connection? Social
Science Computer Review, 29(4), 419–436.
• Lazaridou, A., Bruni, E., and Baroni, M. (2014). Is this a wampimuk? Cross-modal
mapping between distributional semantics and the visual world. Proceedings of the
52ndAnnual Meeting of the Association for Computational Linguistics, Baltimore, MD,
USA, 1403–1414.
• Le, Q. V., and Mikolov, T. (2014). Distributed representations of sentences and
documents. arXiv preprint arXiv:1405.4053.
7
• Lehrman, M. T., Alm, C. O., and Proaño, R. A. (2012). Detecting distressed vs. non-
distressed affect states in short forum texts. Proceedings of the Workshop on
Language in Social Media at the Conference of the North American Chapter of the
Association for Computational Linguistics-Human Language Technologies, Montreal,
Canada, 9-18.
• Li, J., Zhou, G., Wang, H., and Zhu, Q. (2010). Learning the scope of negation via
shallow semantic parsing. Proceedings of the 23rd International Conference on
Computational Linguistics, Beijing, China, 671–679.
• Li, X. G., Li, S. M., Jiang, L. R., and Zhang, S. B. (2013). Study of English
pronunciation quality evaluation system with tone and emotion analysis capabilities.
Applied Mechanics and Materials, 475-476, 318–323.
• Liberman, M., Davis, K., Grossman, M., Martey, N., and Bell, J. (2002). Emotional
Prosody Speech and Transcripts LDC2002S28. Web Download. Philadelphia:
Linguistic Data Consortium.
• Lin, C., He, Y., and Everson, R. (2011). Sentence subjectivity detection with
weakly-supervised learning. Proceedings of the 5th International Joint Conference
on Natural Language Processing, 1153–1161.
• Lindquist, K. A., Wager, T. D., Kober, H., Bliss-Moreau, E., and Barrett, L. F.
(2012). The brain basis of emotion: A meta-analytic review. Behavioral and Brain
Sciences, 35(3), 121–143.
• Liu, B., and Zhang, L. (2012). A survey of opinion mining and sentiment analysis.
In Aggarwal, C. C., and Zhai, C. (Eds.) Mining Text Data. New York, NY: Springer
Science+Business Media, 415–463.
• Liu, H., Lieberman, H., and Selker, T. (2003a). A model of textual affect sensing
using real-world knowledge. Proceedings of the International Conference on
Intelligent User Interfaces, Miami, FL, USA, 125-132.
• Liu, H., Selker, T., and Lieberman, H. (2003b). Visualizing the affective structure of
a text document. Proceedings of Conference on Human Factors in Computing
Systems, Ft. Lauderdale, FL, USA, 740–741.
• Lu, B., and Tsou, B. K. (2010). CityU-DAC: Disambiguating sentiment-ambiguous
adjectives within context. Proceedings of the 5th International Workshop on
Semantic Evaluation, 292–295.
• Lyons, J. (1995). Linguistic Semantics: An Introduction. Cambridge: Cambridge
University Press.
• Makki, R., Brooks, S., and Milios, E. E. (2014). Context-specific sentiment lexicon
expansion via minimal user interaction. Proceedings of the International Conference
on Information Visualization Theory and Applications, Rome, Italy, 178–186.
• Malti, T., and Krettenauer, T. (2013). The relation of moral emotion attributions to
prosocial and antisocial behavior: A meta-analysis. Child Development, 84(2), 397–
412.
• Martinez-Camara, E., Martın-Valdivia, M. T., Urenalopez, L. A., and Montejoraez, A.
R. (2012). Sentiment analysis in Twitter. Natural Language Engineering, 20(1), 1–
28.
• Matykiewicz, P., Duch, W., and Pestian, J. P. (2009). Clustering semantic spaces of
suicide notes and newsgroup articles. Proceedings of the Workshop on Current
Trends in Biomedical Natural Language Processing, Boulder, CO, 179-184.
8
• Maynard, D., and Funk, A. (2011 [2012]). Automatic detection of political opinions
in tweets. In Garcia-Castro, R. et al. (Eds.) The Semantic Web: ESWC 2011
Workshops, Heraklion, Greece. LNCS 7117, 88–99.
• Mihalcea, R., Banea, C., and Wiebe, J. (2007). Learning multilingual subjective
language via cross-lingual projections. Proceedings of the 45th Annual Meeting of
the Association of Computational Linguistics, Prague, Czech Republic, 976-983.
• Mihalcea, R., and Liu, H. (2006). A corpus-based approach to finding happiness.
Proceedings of AAAI-2006 Spring Symposium on Computational Approaches to
Analysing Weblogs, 139–144.
• Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. (2013).
Distributed representations of words and phrases and their compositionality.
Proceedings of Advances in Neural Information Processing Systems, 3111–3119.
• Minamikawa, A., and Yokoyama, H. (2011a). Blog tells what kind of personality you
have: egogram estimation from Japanese weblog. Proceedings of the ACM 2011
Conference on Computer Supported Cooperative Work, 217–220, Hangzhou, China.
• Minamikawa, A., and Yokoyama, H. (2011b). Personality estimation based on
weblog text classification. In Mehtrota, K. G. et al. (Eds.) Modern Approaches in
Applied Intelligence. LNCS 6704. Berlin Heidelberg: Springer-Verlag, 89–97.
• Mohammad, S. M. (2011). From once upon a time to happily ever after: Tracking
emotions in novels and fairy tales. Proceedings of the 5th ACL-HLT Workshop on
Language Technology for Cultural Heritage, Social Sciences, and Humanities,
Portland, OR, USA, 105–114.
• Mohammad, S. M. (2012a). #Emotional tweets. Proceedings of the 1st Conference
on Lexical and Computational Semantics, Montreal, Canada, 246–255.
• Mohammad, S. M. (2012b). From once upon a time to happily ever after: Tracking
emotions in mail and books. Decision Support Systems, 53(4), 730–741.
• Mohammad, S. M. (2012c). Portable features for classifying emotional text.
Proceedings of the Conference of the North American Chapter of the Association for
Computational Linguistics-Human Language Technologies, Montreal, Canada, 587–
591.
• Mohammad, S. M. (2015a). Imagisaurus: An interactive visualizer of valence and
emotion in the Roget's Thesaurus. Proceedings of the ACL Workshop on
Computational Approaches to Subjectivity and Sentiment Analysis, Lisbon,
Portugal. Forthcoming.
• Mohammad, S. M. (2015b) Sentiment analysis: Detecting valence, emotions, and
other affectual states from text. Emotion Measurement. Forthcoming.
• Mohammad, S. M., Dunne, C., and Dorr, B. (2009). Generating high-coverage
semantic orientation lexicons from overtly marked words and a thesaurus.
Proceedings of the Conference on Empirical Methods in Natural Language
Processing, Waikiki, Hawaii, 599–608.
• Mohammad, S. M., and Kiritchenko, S. (2013a). Using hashtags to capture fine
emotion categories from tweets. Computational Intelligence, 31(2), 301–326.
• Mohammad, S. M., and Kiritchenko, S. (2013b). Using nuances of emotion to
identify personality. Proceedings of the International Conference on Weblogs and
Social Media (ICWSM-13), Boston, MA, 27-30.
• Mohammad, S. M., Kiritchenko, S., and Zhu, X. (2013). NRC-Canada: Building the
state-of-the-art in sentiment analysis of tweets. Proceedings of the 7th
International Workshop on Semantic Evaluation Exercises (SemEval-2013), Atlanta,
9
GA, USA, 321-327.

• Mohammad, S. M., and Turney, P. D. (2010). Emotions evoked by common words
and phrases: Using Mechanical Turk to create an emotion lexicon. Proceedings of
the NAACL-HLT 2010 Workshop on Computational Approaches to Analysis and
Generation of Emotion in Text, Los Angeles, CA, 26-34.
• Mohammad, S. M., and Turney, P. D. (2013). Crowdsourcing a word–emotion
association lexicon. Computational Intelligence, 29 (3), 436-465.
• Mohammad, S. M., and Yang, T. (2011). Tracking sentiment in mail: How genders
differ on emotional axes. Proceedings of the 2nd Workshop on Computational
Approaches to Subjectivity and Sentiment Analysis (WASSA 2.011), Portland, OR,
70–79.
• Mohammad, S. M., Zhu, X., Kiritchenko, S., and Martin, J. (2015). Sentiment,
emotion, purpose, and style in electoral tweets. Information Processing and
Management, 51(4), 480–499.
• Montague, R. (1974). Formal Philosophy: Selected Papers of Richard Montague.
Thomason, R. (Ed.) New Haven, CT: Yale University Press.
• Montero, C. S., Munezero, M., and Kakkonen, T. (2014). Investigating the role of
emotion-based features in author gender classification of text. In Gelbukh, A. (Ed.)
Computational Linguistics and Intelligent Text Processing: 15th International
Conference, CICLing 2014, Kathmandu, Nepal. LNCS 8404. Berlin Heidelberg:
Springer-Verlag, 98-114.
• Murakami, A., and Raymond, R. (2010). Support or oppose? Classifying positions in
online debates from reply activities and opinion expressions. Proceedings of the
International Conference on Computational Linguistics, Beijing, China, 869–875.
• Nalisnick, E. T., and Baird, H. S. (2013b). Extracting sentiment networks from
Shakespeare’s plays. IEEE 12th International Conference on Document Analysis and
Recognition (ICDAR), 758–762.
• Neviarouskaya, A., Prendinger, H., and Ishizuka, M. (2009). Compositionality
principle in recognition of fine-grained emotions from text. Proceedings of the 3rd
• International Conference on Weblogs and Social Media (ICWSM-09), San Jose, CA,
278–281.
• Neviarouskaya, A., Prendinger, H., and Ishizuka, M. (2010). Recognition of affect,
judgment, and appreciation in text. Proceedings of the 23rd International
Conference on Computational Linguistics, Beijing, China, 806-814
• Orme, B. (2009). Maxdiff analysis: Simple counting, individual-level logit, and HB.
Sawtooth Software, Inc.
• Ortony, A., Clore, G. L., and Collins, A. (1990). The Cognitive Structure of
Emotions. Cambridge: Cambridge University Press.
• Osgood, C. E. (1969). On the whys and wherefores of E, P, and A. Journal of
Personality and Social Psychology, 12 (3), 194–199.
• Osgood, C. E., Suci, G. J., and Tannenbaum, P. (1957). The Measurement of
Meaning. Urbana, USA: University of Illinois Press.
• Pak, A., and Paroubek, P. (2010). Twitter as a corpus for sentiment analysis and
opinion mining. Proceedings of the 7th Conference on International Language
Resources and Evaluation, Valletta, Malta, 1320-1326.
• Pang, B., and Lee, L. (2008). Opinion mining and sentiment analysis. Foundations
and Trends in Information Retrieval, 2(1–2), 1–135.
• Parrot, W. G. (Ed.) (2001). Emotions in Social Psychology. Philadelphia: Psychology
10
Press.
• Paul, M. J., and Dredze, M. (2011). You are what you tweet: Analyzing Twitter for
public health. Proceedings of the 5th International AAAI Conference on Weblogs and
Social Media, 265–272.
• Paul, W., Alm, C. O., Bailey, R., Geigel, J., and Wang, L. (2015). Stressed out:
What speech tells us about stress. Proceedings of Interspeech, Dresden, Germany.
Forthcoming.
• Pearl, L., and Steyvers, M. (2010). Identifying emotions, intentions, and attitudes
in text using a game with a purpose. Proceedings of the NAACL-HLT 2010 Workshop
on Computational Approaches to Analysis and Generation of Emotion in Text, Los
Angeles, CA, 71-79.
• Pennebaker, J. W., Mehl, M. R., and Niederhoffer, K. G. (2003). Psychological
aspects of natural language use: Our words, our selves. Annual Review of
Psychology, 54(1), 547–577.
• Pestian, J. P., Matykiewicz, P., and Grupp-Phelan, J. (2008). Using natural language
processing to classify suicide notes. Proceedings of the Workshop on Current Trends
in Biomedical Natural Language Processing, Columbus, OH, 96-97.
• Picard, R. (1997). Affective Computing. Cambridge: MIT Press.
• Plutchik, R. (1962). The Emotions. New York: Random House.
• Plutchik, R. (1980). A general psychoevolutionary theory of emotion. Emotion:
Theory, Research, and Experience, 1(3), 3–33.
• Plutchik, R. (2001). The nature of emotions: Human emotions have deep
evolutionary roots, a fact that may explain their complexity and provide tools for
clinical practice. American Scientist, 89(4), 344–350.
• Polanyi, L., and Zaenen, A. (2004). Contextual valence shifters. Exploring Attitude
and Affect in Text: Theories and Applications (AAAI Spring Symposium Series).
• Pontiki, M., Galanis, D., Pavlopoulos, J., Papageorgiou, H., Androutsopoulos, I., and
Manandhar, S. (2014). SemEval-2014 Task 4: Aspect based sentiment analysis.
Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval
2014) at the 25th International Conference on Computational Linguistics, Dublin,
Ireland, 27-35.
• Popescu, A.-M., and Etzioni, O. (2005). Extracting product features and opinions
from reviews. Proceedings of the Conference on Human Language Technology and
Empirical Methods in Natural Language Processing, Vancouver, Canada, 339–346.
• Poulin, C., Shiner, B., Thompson, P., Vepstas, L., Young-Xu, Y., Goertzel, B., Watts,
B., Flashman, L., and McAllister, T. (2014). Predicting the risk of suicide by
analyzing the text of clinical notes. PLOS ONE 9(1).e85733.
• Purver, M., and Battersby, S. (2012). Experimenting with distant supervision for
emotion classification. Proceedings of the 13th Conference of the European Chapter
of the Association for Computational Linguistics, Avignon, France, 482–491.
• Qadir, A. (2009). Detecting opinion sentences specific to product features in
customer reviews using typed dependency relations. Proceedings of the Workshop
on Events in Emerging Text Types (eETTs ’09), Borovets, Bulgaria, 38–43.
• Quan, C., and Ren, F. (2009). Construction of a blog emotion corpus for Chinese
emotional expression analysis. Proceedings of the 2009 Conference on Empirical
Methods in Natural Language Processing, Singapore, 1446-1454.
• Quan, C., and Ren, F. (2014). Visualizing emotions from chinese blogs by textual
emotion analysis and recognition techniques. International Journal of Information
11
Technology and Decision Making, 1–20.

• Ragin, A. B., and Oltmanns, T. F. (1983). Predictability as an index of impaired
verbal communication in schizophrenic and affective disorders. British Journal of
Psychiatry, 143(6), 578–583.
• Reilly, J., and Seibert, L. (2003). Language and emotion. In Davidson, R. J.,
Scherer, K. R., and Goldsmith, H. H. (Eds.) Handbook of Affective Sciences.
Oxford: Oxford University Press, 535-559.
• Ren, F., and Quan, C. (2012). Linguistic-based emotion analysis and recognition for
measuring consumer satisfaction: An application of affective computing.
Information Technology and Management, 13(4), 321–332.
• Reyes, A., Rosso, P., and Veale, T. (2013). A multidimensional approach for
detecting irony in Twitter. Language Resources and Evaluation, 47(1), 239–268.
• Riloff, E., and Wiebe, J. (2003). Learning extraction patterns for subjective
expressions. Proceedings of the 2003 Conference on Empirical Methods in Natural
Language Processing, Sapporo, Japan, 105–112.
• Rosenthal, S., Nakov, P., Kiritchenko, S., Mohammad, S. M., Ritter, A., and
Stoyanov, V. (2015). SemEval-2015 Task 10: Sentiment analysis in Twitter.
Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval-
2015) Denver, CO, 451-463.
• Rosenthal, S., Nakov, P., Ritter, A., and Stoyanov, V. (2014). SemEval-2014 Task
9: Sentiment Analysis in Twitter. Proceedings of the 8th International Workshop on
Semantic Evaluation (SemEval-2014) at the 25th International Conference on
Computational Linguistics, Dublin, Ireland, 73-80.
• Rude, S., Gortner, E.-M., and Pennebaker, J. (2004). Language use of depressed
and depression-vulnerable college students. Cognition and Emotion, 18(8), 1121–
1133.
• Russell, B. (1912). The Problems of Philosophy. Oxford: Oxford University Press.
• Russell, J. A. (1980). A circumplex model of affect. Journal of Personality and Social
psychology, 39(6), 1161.
• Russell, J. A., and Fernández-Dols, J. M. (1998 [1997]). What does a facial
expression mean? [selection]. In Jenkins, J. M., Oatley, K., and Stein, N. L. (Eds.)
Human Emotions: A Reader. Oxford: Blackwell, 73–77.
• Salameh, M., Mohammad, S. M., and Kiritchenko, S. (2015). Sentiment after
translation: A case-study on Arabic social media posts. Proceedings of the North
American Chapter of Association of Computational Linguistics, Denver, CO, 767-
777.
• Scherer, K. R. (2003). Vocal communication of emotion: A review of research
paradigms. Speech Communication, 40(1-2), 227-256.
• Scherer, K. R., Bänziger, T., and Roesch, E. B. (Eds.) (2010). Blueprint for Affective
Computing: A Sourcebook. Oxford: Oxford University Press.
• Schrading, N., Alm, C. O., Ptucha, R., and Homan, C. M. (2015). #WhyIStayed,
#WhyILeft: Microblogging to make sense of domestic abuse. Proceedings of Human
Language Technologies: The Annual Conference of the North American Chapter of
the Association for Computational Linguistics, Denver, CO, USA, 1281-1286.
• Schröder, M., Baggia, P., Burkhardt, F., Pelachaud, C., Peter, C., and Zovato, E.
(2011). EmotionML - An upcoming standard for representing emotions and related
states. Proceedings of the 4th International Conference on Affective Computing and
Intelligent Interaction, Memphis, TN, 316-325.
12
• Schuller, B., and Batliner, A. (2014). Computational Paralinguistics: Emotion, Affect

and Personality in Speech and Language Processing. Chichester: Wiley.
• Schwartz, H., Eichstaedt, J., Kern, M., Dziurzynski, L., Agrawal, M., Park, G.,
Lakshmikanth, S. K., Jha, S., Seligman, M. E. P., Ungar, L. H., and Lucas, R. E.
(2013a). Characterizing geographic variation in well-being using tweets.
Proceedings of the 7th International AAAI Conference on Weblogs and Social Media,
Ann Arbor, MI, 583-591.
• Schwartz, H. A., Eichstaedt, J. C., Kern, M. L., Dziurzynski, L., Ramones, S. M.,
Agrawal, M., Shah, A., Kosinski, M., Stillwell, D., Seligman, M. E., and Ungar, L. H.
(2013b). Personality, gender, and age in the language of social media: The open-
vocabulary approach. PloS One, 8(9), 1–16.
• Schwartz, H. A, Eichstaedt, J. C., Kern, M. L., Park, G., Sap, M., Stillwell, D.,
Kosinski, M., and Ungar, L. H. (2014). Towards assessing changes in degree of
depression through Facebook. Proceedings of the Workshop on Computational
Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality at the
52nd Annual Meeting of the Association for Computational Linguistics, Baltimore, MD,
USA, 118-125.
• Socher, R., Perelygin, A., Wu, J. Y., Chuang, J., Manning, C. D., Ng, A. Y., and
Potts, C. (2013). Recursive deep models for semantic compositionality over a
sentiment treebank. Proceedings of the Conference on Empirical Methods in Natural
Language Processing, Seattle, WA, 1631-1642.
• Somasundaran, S. and Wiebe, J. (2009). Recognizing stances in online
debates. Proceedings of the Joint conference of the 47th Annual Meeting of the
Association for Computational Linguistics and the 4th International Joint Conference
on Natural Language Processing of the Asian Federation of Natural Language
Processing, Singapore, 226–234.
• Sridhar, Dhanya, Getoor, Lise, and Walker, Marilyn. (2014). Collective stance
classification of posts in online debate forums. Proceedings of the Joint Workshop
on Social Dynamics and Personal Attributes in Social Media, Baltimore, MD, 109–
117.
• Stone, P., Dunphy, D. C., Smith, M. S., Ogilvie, D. M. (1966). The General Inquirer:
A Computer Approach to Content Analysis. Cambridge: MIT Press.
• Strapparava, C., and Mihalcea, R. (2007). SemEval-2007 Task 14: Affective text.
Proceedings of SemEval-2007, Prague, Czech Republic, 70–74.
• Strapparava, C., and Mihalcea, R. (2010). Annotating and identifying emotions in
text. In Armano, G., et al. (Eds.) Intelligent Information Access, SCI 301. Berlin
Heidelberg: Springer-Verlag, 21–38.
• Strapparava, C., and Valitutti, A. (2004). WordNet-Affect: An affective extension of
WordNet. Proceedings of the 4th International Conference on Language Resources
and Evaluation, Lisbon, Portugal, 1083–1086.
• Su, F., and Markert, K. (2008). From words to senses: a case study of subjectivity
recognition. Proceedings of the 22nd International Conference on Computational
Linguistics, Manchester, UK, 825–832.
• Su, Q., Xiang, K., Wang, H., Sun, B., and Yu, S. (2006). Using pointwise mutual
information to identify implicit features in customer reviews. Proceedings of the 21st
International Conference on Computer Processing of Oriental Languages,
Singapore, 22–30.
• Suero Montero, C., and Suhonen, J. (2014). Emotion analysis meets learning
13
analytics: On-line learner profiling beyond numerical data. Proceedings of the 14th
Koli Calling International Conference on Computing Education Research, 165–169.
• Sun, Y., Quan, C., Kang, X., Zhang, Z., and Ren, F. (2014). Customer emotion
detection by emotion expression analysis on adverbs. Information Technology and
Management, 1–9.
• Suttles, J., and Ide, N. (2013). Distant supervision for emotion classification with
discrete binary values. Computational Linguistics and Intelligent Text Processing,
Samos, Greece. Berlin Heidelberg: Springer-Verlag, 121– 136.
• Taboada, M., Brooke, J., Tofiloski, M., Voll, K., and Stede, M. (2011). Lexicon-based
methods for sentiment analysis. Computational Linguistics, 37(2), 267–307.
• Tang, D., Wei, F., Qin, B., Liu, T., and Zhou, M. (2014a). Coooolll: A deep learning
system for twitter sentiment classification. Proceedings of the 8th International
Workshop on Semantic Evaluation (SemEval 2014), Dublin, Ireland, 208–212.
• Tang, D., Wei, F., Qin, B., Zhou, M., and Liu, T. (2014b). Building large-scale
Twitter-specific sentiment lexicon: A representation learning approach. Proceedings
of the International Conference on Computational Linguistics, Dublin, Ireland, 172–
182.
• Thelwall, M., Buckley, K., and Paltoglou, G. (2011). Sentiment in Twitter events.
Journal of the American Society for Information Science and Technology, 62(2),
406–418.
• Thomas, B., Dhanya, K. A., and Vinod, P. (2014). Synthesized feature space for
multiclass emotion classification. IEEE 1st International Conference on Networks and
Soft Computing (ICNSC), Guntur, 188–192.
• Thomas, M., Pang, B., and Lee, L. (2006). Get out the vote: Determining support or
opposition from congressional floor-debate transcripts. Proceedings of the
Conference on Empirical Methods in Natural Language Processing, Sydney,
Australia, 327–335.
• Tumasjan, A., Sprenger, T. O., Sandner, P. G., and Welpe, I. M. (2010). Election
forecasts with Twitter: How 140 characters reflect the political landscape. Social
Science Computer Review, 29(4), 402–418.
• Turney, P. D. (2002). Thumbs up or thumbs down? Semantic orientation applied to
unsupervised classification of reviews. Proceedings of the 40th Annual Meeting of the
Association for Computational Linguistics, Philadelphia, PA, 417-424.
• Turney, P., and Littman, M. L. (2003). Measuring praise and criticism: Inference of
semantic orientation from association. ACM Transactions on Information Systems,
21(4), 315-346.
• Vaassen, F., and Daelemans, W. (2011). Automatic emotion classification for
interpersonal communication. Proceedings of the ACL Workshop on Computational
Approaches to Subjectivity and Sentiment Analysis, Portland, OR, 104–110.
• Vaidyanathan, P., Prud’hommeaux, E., Alm, C. O., Pelz, J. B., and Haake, A.
(2015). Alignment of eye movements and spoken language for semantic image
understanding. Proceedings of the 11th International Conference on Computational
Semantics, London, UK, 76-81.
• Wang, C., and Wang, F. (2012). A bootstrapping method for extracting sentiment
words using degree adverb patterns. IEEE International Conference on Computer
Science and Service System (CSSS), 2173–2176.
14
• Wang, L., Bailey, R., Geigel, J., Alm, C. O., Bethamcherla, V., Krithika, S., John, B.,
and Kilroy, T. (2014). Sensor fusion for cognitive load and stress monitoring and
detection. RIT-VA, Canandaigua, NY. Poster.
• Wang, W., Chen, L., Thirunarayan, K., and Sheth, A. P. (2012). Harnessing Twitter
"big data" for automatic emotion identification. Proceedings of the 2012 ASE/IEEE
International Conference on Social Computing and 2012 ASE/IEEE International
Conference on Privacy, Security, Risk and Trust, Washington, DC, 587–592.
• Wang, X., and Fu, G.-H. (2010). Chinese subjectivity detection using a sentiment
density-based naive Bayesian classifier. IEEE International Conference on Machine
Learning and Cybernetics (ICMLC), Vol. 6, 3299–3304.
• Wang, Z. (2014). Segment-based fine-grained emotion detection for chinese
text. Proceedings of the 3rdCIPS-SIGHAN Joint Conference on Chinese Language
Processing, Wuhan, China. 52–60.
• Walker, M. A., Anand, P., Abbott, R., and Grant, R. (2012). Stance classification
using dialogic properties of persuasion. Proceedings of the Conference of the North
American Chapter of the Association for Computational Linguistics-Human Language
Technologies, Montreal, Canada, 592–596.
• Warriner, A. B., Kuperman, V., and Brysbaert, M. (2013). Norms of valence,
arousal, and dominance for 13,915 English lemmas. Behavior Research Methods,
45(4), 1191–1207.
• Webster, G. and Weir, C. (2005). Emotional responses to music: Interactive effects
of mode, texture, and tempo. Motivation and Emotion, 29(1), 19–39.
• Whissell, C. (2000). Phonoemotional profiling: A description of the emotional
ﬂavour of English texts on the basis of the phonemes employed in them.
Perceptual and Motor Skills, 91(2), 617-648.
• Wiebe, J., and Riloff, E. (2005). Creating subjective and objective sentence
classifiers from unannotated texts. Proceedings of the 6th International Conference
on Computational Linguistics and Intelligent Text Processing, Mexico City,
Mexico, 486–497.
• Wiebe, J., and Riloff, E. (2011). Finding mutual benefit between subjectivity
analysis and information extraction. IEEE Transactions on Affective Computing,
2(4), 175–191.
• Wiebe, J., Wilson, T., Bruce, R., Bell, M., and Martin, M. (2004). Learning subjective
language. Computational Linguistics, 30(3), 277–308.
• Wilson, T., Hoffmann, P., Somasundaran, S., Kessler, J., Wiebe, J., Choi, Y., Cardie,
C., Riloff, E., and Patwardhan, S. (2005). Opinionfinder: A system for subjectivity
analysis. Proceedings of the Conference on Empirical Methods in Natural Language
Processing, Interactive Demonstrations, Vancouver, Canada, 34–35.
• Wilson, T., Kozareva, Z., Nakov, P., Rosenthal, S., Stoyanov, V., and Ritter, A.
(2013). SemEval-2013 Task 2: Sentiment analysis in Twitter. Proceedings of the 7th
International Workshop on Semantic Evaluation (SemEval-2013), Atlanta, GA, 312-
320.
• Womack, K., McGowen, V., Alm, C. O., Pelz, J., Haake, A., and Shi, P. (2014).
Analyzing multimodal behaviors of students with autism spectrum disorders.
Effective Access Technology Conference, Rochester, NY. Poster.
• Zhang, L., Liu, B., Lim, S. H., and O’Brien-Strain, E. (2010). Extracting and ranking
product features in opinion documents. Proceedings of the 23rd International
Conference on Computational Linguistics, Beijing, China, 1462–1470.
15
• Zhu, X., Guo, H., Mohammad, S., and Kiritchenko, S. (2014). An empirical study on
the effect of negation words on sentiment. Proceedings of the 52nd Annual Meeting
of the Association for Computational Linguistics, Baltimore, Maryland, 304–313.
• Zhu, X., Sobhani, P., and Guo, H. (2015). Long short-term memory over recursive
structures. International Conference on Machine Learning, Lile, France.
16

Recommended Banget Nih

Hochgeladen von

Dokumentinformationen

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Recommended Banget Nih

Hochgeladen von

Copyright:

Verfügbare Formate

!

Cecilia Ovesdotter Alm

Tutorial at EMNLP 2015, Lisboa, Portugal

UIUC RIT, USA

• Convey why you should care about analysis of affect and

• Provide a broad background on terminology, conceptual

• Give an overview of affect processing which spans both

• Exemplify the area’s relevance in interesting application

• Springboard a conversation as we head into EMNLP

Emotive Language Use

BREAK! Computational Modeling (Part 1)

Future Directions and Wrap-up

Language serves social and interpersonal functions. Affective meaning is

Besides recognition, characterization, or generation of affect states, this

• Early recognized as important (Bühler, Jakobson, Lyons, Halliday) yet

• Beyond ‘propositional’ meaning which tends to disregard emotive

(1) Bill Clinton was President of the United States.

• HCC as a theoretical/functional paradigm acknowledges…

• the central role of humans in computing

• Language as sensor of the body area network for personalized computing

• link to a human individual and a group of individuals

Linguistic sensing of affective states or affect-related behaviors and experiences

Consider a person who reads stories. Storytelling

(Mohammad!and!Alm,!2015)! 11! (Alm, 2009)

• Customer service and branding

• Affective interfaces and dialog systems

• Medical support systems

• Public health exploration

We will return to some of these application domains later on.

Linguistic Fake vs.

• Emotional gist of an utterance is not simply the sum of

• Negation and modals impact affect of the text, without

• Emotions are often not explicitly stated.

• Prosodic information often absent in text.

• Different degrees of affect depending on sense and context.

• Difficult to interpret creative uses of language such as sarcasm,

• Most machine learning algorithms for affect analysis

• Texts/utterances may convey mixed, contrastive, or

Human affect perception seems characterized by ...

! subjectivity & expectations

! contextual factors CHALLENGING

Instructors: Affect-related tasks thus represent a pedagogical opportunity to

• Range of modes – standard/nonstandard, dialog/monolog/massive

Language as window into cognition - how people think & conceptualize

• Socio-cultural and interpersonal factors

• Individuals’ social attributes as additional factors

• In general in affect perception, individuals’ interpretation

They are convenient to integrate in multimodal capture for affect sensing.

Cardiac activity Gaze

• Lexical contents • Casuals, negatives (level/ratio)

(Mohammad!and!Alm,!2015)! 26! (Besnier, 1990;

Your canary frightened our cat this

! Lexicon for affect states, contrastive states, affect-related words/expressions

! Example sentence with affect-related lexical items:

(Mohammad and Alm, 2015) 28! (Alm, 2010)

! Speech acts, e.g. cursing:

! Forms of direct speech, e.g. interjection:

! Mixed emotions (affect sequencing):

(Mohammad and Alm, 2015) 29! (Alm, 2010)

Data: Fairytale texts

Can a model capture affect as a linguistic

Linguistic features toward classification:

(Mohammad!and!Alm,!2015)! 30! (Alm, 2009)

An interactive genetic algorithm evolved

•  Convey why you should care about analysis of affect and

•  Provide a broad background on terminology, conceptual

•  Give an overview of affect processing which spans both

•  Exemplify the area’s relevance in interesting application

•  Springboard a conversation as we head into EMNLP

•  Early recognized as important (Bühler, Jakobson, Lyons, Halliday) yet

•  Beyond ‘propositional’ meaning which tends to disregard emotive

•  HCC as a theoretical/functional paradigm acknowledges…

•  the central role of humans in computing

•  Language as sensor of the body area network for personalized computing

•  link to a human individual and a group of individuals

•  Customer service and branding

•  Affective interfaces and dialog systems

•  Medical support systems

•  Public health exploration

•  Emotional gist of an utterance is not simply the sum of

•  Negation and modals impact affect of the text, without

•  Emotions are often not explicitly stated.

•  Prosodic information often absent in text.

•  Different degrees of affect depending on sense and context.

•  Difficult to interpret creative uses of language such as sarcasm,

•  Most machine learning algorithms for affect analysis

•  Texts/utterances may convey mixed, contrastive, or

!  subjectivity & expectations

!  contextual factors CHALLENGING

•  Range of modes – standard/nonstandard, dialog/monolog/massive

•  Socio-cultural and interpersonal factors

•  Individuals’ social attributes as additional factors

•  In general in affect perception, individuals’ interpretation

•  Lexical contents •  Casuals, negatives (level/ratio)

!  Lexicon for affect states, contrastive states, affect-related words/expressions

!  Example sentence with affect-related lexical items:

!  Speech acts, e.g. cursing:

!  Forms of direct speech, e.g. interjection:

!  Mixed emotions (affect sequencing):

•  Linguistic affect data are informative – either in isolation or in combination

•  Their prevalence is another factor to consider; while some features may

•  Attributes of language users (sociolinguistic variables) may play a role.

•  No agreed upon model of

•  Circa 1950’s, Margaret Mead and others

•  Grad school experiment on people’s ability to distinguish photos of

•  Dictionary of Affect (Whissell)

•  Affective Norms for English Words (Texts) (Bradley & Lang)

•  Harvard General Inquirer categories (Stone etc.)

•  NRC Emotion Lexicon (Mohammad & Turney)

•  MaxDiff Sentiment Lexicon (Kiritchenko, Zhu, & Mohammad)

•  SentiWordNet (Esuli & Sebastiani)

•  WordNet-Affect (Strapparava & Valitutti)

•  NRC Twitter Lexicons (Mohammad, Kiritchenko, & Zhu)

–  Hashtag Emotion Lexicon, Hashtag Sentiment Lexicon,

•  Ren-CECps blog emotion corpus (Quan & Ren, 2009)

•  2013 Chinese Microblog Sentiment Analysis Evaluation

•  Lower human intersubjective agreement; broadly

•  Lack of a stable ‘ground truth’

•  Ratings may convey varying acceptability

•  Methodological issues with ‘training’ annotators

•  Immense implications for the development and

•  Hashtags labels are not always good labels:

•  Unclear from rest of the message

•  Stroop Test induces cognitive load

•  Options for affect modeling, with an ongoing debate

•  Linguistic affect phenomena:

•  Annotations reveal insights into fuzzy phenomena involving

•  Sentences, tweets, SMS messages

•  Paragraphs, documents, customer reviews,

•  Nouns and verbs

!  Word sets generated automatically

!  Guides annotator to desired sense

!  Aids quality control

•  Eight such questions for Plutchik’s eight basic emotions