Beruflich Dokumente
Kultur Dokumente
Abstract: Changes in the framing of topical news have been shown to foreshadow significant
therefore an important problem, which existing research has not considered. Previous
approaches are manual surveys, which rely on human effort and are consequently
isolate framing change trends over several years. We demonstrate our approach by
isolating framing change periods that correlate with previously known framing changes.
We have prepared a new dataset, consisting of over 12,000 articles from seven news
topics or domains in which earlier surveys have found framing changes. Finally, our
work highlights the predictive utility of framing change detection, by identifying two
Chaitanya Shivade
Munindar Singh
Opposed Reviewers:
Response to Reviewers: We thank the editors for their valuable feedback. We address the main points below.
Editor Comment:
My main concern with this paper deals with the evaluation of the approach. More
precisely, the experimental section illustrates a series of case studies or scenarios
where the frame change is identified through a sudden polarity drift (Figures 10.16) that
is shown to correlate with some well-known fact or event studied in the literature. The
point is: how much is this evaluation anecdotal, and to what extent can it be
quantitatively measured? In all Figures from 10 through 16, several peaks and sudden
changes can be observed in the polarity distribution (e.g., Figure 12, class 5, years
2000 through 2003, or Figure 13, class 1, year 2006, to mention just a few): do they all
correspond to frame changes? If not, how can they be detected/studied? The paper
states that the dataset was annotated by experts: how? Can such annotation be used
for quantitative evaluation of the approach?
Powered by Editorial Manager® and ProduXion Manager® from Aries Systems Corporation
Response:
This comment concerns two primary aspects of the paper: (i) defining a framing
change, and in particular, isolating framing changes, and filtering out polarity drifts that
do not correspond to framing changes (ii) quantitative evaluation of the approach. We
address each aspect below.
We have added a section entitled Defining Framing Changes. We summarize the main
points here.
Since language and human behavior are not strictly deterministic, the measurement of
any temporally disparate pair of news corpora using adjective polarity (or any other
numerical metric) would result in different representative values of the two corpora.
Therefore, in this sense, any pair of news corpora can be said to have undergone a
framing change.
Further, individual metrics are susceptible to noisy readings due to imprecise data and
measurement. In particular, such an effect may cause sudden isolated spikes between
successive measurements.
This motivates the question of how a framing change is defined, in the context of our
computational measurements. The usual social science definition is that a framing
change is a shift in the way that a specific topic is presented to an audience. To isolate
such changes computationally, we use the following key observations from ground
truth framing changes: (i) framing changes take place as trends that are consistent
over at least $k$ years (ii) framing changes must be consistent across multiple
measurements.
Our aim in this paper is to begin from a set of time series such as the ones in figures
10 to 16, and isolate such trends. The requirement motivated by our first condition,
namely, that framing changes must last at least k years, is easy to satisfy by imposing
such a numerical threshold.
Our approach thus identifies polarity drifts that are both correlated (quantitatively
measured by correlations between different measures of polarity) and sustained (by
the imposition of a threshold of duration). We point out that our approach filters out
isolated drifts in individual polarity measures, since such drifts are uncorrelated across
multiple measures. Further, we note that the magnitude of individual drifts matters only
indirectly to our approach, to the extent that a larger drift, if consistent across multiple
polarity measures, may have higher correlation than a smaller drift that is also
correlated.
In order to do so, we study the literature pertaining to framing changes in the domains
we examine. We identify large-scale studies conducted by reputed organizations such
as the National Cancer Institute, the Columbia Journalism Review, Pew Research, and
so on. These studies examine news and media publishing in a particular domain over a
period of time, as we do, and manually identify changes in the framing of domain news
during these periods.
The studies we rely on for ground truth sometimes provide quantitative justification for
Powered by Editorial Manager® and ProduXion Manager® from Aries Systems Corporation
their findings. These studies therefore provide an expert annotation of framing changes
in our domains, for the periods we examine..
Given that the data sources and coverage between our analysis and that of prior
surveys are usually quite different, the correlations we obtain appear quite substantial.
However, quantitative evaluation remains challenging for the reasons we point out.
This paper follows the spirit of recent work in seeking to develop the study of framing
into a computational science. We acknowledge that our methods may undergo
refinement to tackle broader ground truth data, of a wider temporal and geographical
scope. Nonetheless, we posit that our methods and results have scientific value, and
hope that future work will provide greater coverage of ground truth.
Please note that the underlying data preparation requires social science expertise and
cannot be effectively crowdsourced via a platform such as Mechanical Turk. We
therefore hope that our approach piques the interest of social scientists and leads them
to pursue more comprehensive studies of framing in news media that would enable
improvements in computational methods.
Editor Comment:
* What do the annotators tag in the dataset? The paper just state that two raters code a
random sample of articles from each domain, reporting Cohen's kappa. But what do
they code? Frames, or frame changes? If so, how is a frame change defined?
Response:
To ensure that the articles returned by our term search procedure are indeed relevant
to each domain, a random sample of articles from each domain dataset was coded for
relevance by two raters.
Please refer to the section on Defining Framing Changes for a discussion on how we
treat the problem of identifying changes in framing.
Editor Comment:
* With regards to Figures 1 and 2, the authors state that the peak in the LGBT domain
immediately precedes a frame change. But, does this hold also for other peaks of other
domains? Such as, drones in 2004, obesity in 2005, or smoking in 2005?
Response:
Powered by Editorial Manager® and ProduXion Manager® from Aries Systems Corporation
Whereas we do not claim that this correlation is true for all domains, we posit that it
motivates the utility of adjective polarity in the study of framing changes.
Editor Comment:
* What are the differences of the proposed approach with respect to an approach that
detects just frames (instead of frame changes), but then look at changes in the
detected frames...? See, e.g., the following references:
** Alashri et al., "Climate Change" Frames Detection and Categorization Based on
Generalized Concepts", International Journal of Semantic Computing, 2016
** Tsur et al., "A Frame of Mind: Using Statistical Models for Detection of Framing and
Agenda Setting Campaigns", ACL 2015
Response:
We have added paragraphs to the Related Work section, detailing the novel
contributions of our work and drawing distinctions between this paper and earlier
approaches. We summarize this discussion here.
We note that our approach is similar in spirit to Tsur et al’s [9] work, in that both that
work and this paper apply a topic modeling strategy to analyze framing as a time
series. However, we highlight the following key differences and contributions of our
work. Firstly, as both Sheshadri and Singh [8] and Tsur et al point out, framing is a
subjective aspect of communication. Therefore, a computational analysis of framing
should ideally differentiate subjective aspects from fact-based and objective
components of communication. Since adjectives in and of themselves are incapable of
communicating factual information, we take them to be artifacts of how an event or
topic is framed. In contrast, generic n-grams (as used by Tsur et al) do not provide this
distinction.
Further, Tsur et al rely upon estimating ``changes in framing'' using changes in the
relative frequencies of n-grams associated with various topics or frames. Whereas
such an approach is useful in evaluating which of a set of frames may be dominant at
any given time, it does not measure ``framing changes'' in the sense originally
described in [5]. In contrast, our work estimates changes in framing using consistent
polarity drifts of adjectives associated with individual frames. Our approach may also
be applied to each of a number of frames independently of the others, as opposed to
Tsur et al.
Editor Comment:
I have consulted with another academic editor, Dr. Marco Lippi, and we agree that a
desk rejection was premature. Nevertheless, experience tells us that all reviewers
provide a perspective similar to at least some other readers. For this reason I am
requesting that you revise the manuscript to address issues raised in the desk
rejection. Perhaps you made some revisions in your appeal. However, these were not
apparent to me with track changes. Please revise the manuscript itself, including a
rebuttal that identifies the specific location of the modifications you made in response
to the original decision.
Response:
Editor Comment:
Response:
Our updated related work section discusses alternative approaches in detail, and
Powered by Editorial Manager® and ProduXion Manager® from Aries Systems Corporation
describes our novel contributions over these approaches.
Editor Comment:
Response:
The dataset and code are available online at the following link:
https://drive.google.com/open?id=1zAH__Y1lcdriuwUcjZsKmvaqYtzAjyZ9
All our results are reproducible from the data and code in the above mentioned
repository. We will provide a guide to run our code.
Editor Comment:
4. An overview diagram for the proposed approach would help the reader understand
the flow of the proposed approach.
Response:
Editor Comment:
5. The results are presented but not discussed. The section should be renamed to
"Results and Discussion" and appropriate discussion should be added with each pair of
graphs.
Response:
We have expanded our analysis of the results for each domain, including adding a
quantitative precision-recall analysis based on ground truth data.
Editor Comments:
This research is focused on detecting framing changes in topical news. The authors
argue that the public opinion varies with the way the news is framed. The research
lacks motivation as it is not clear what benefits can be achieved if frame changes are
detected. Moreover, the problem is already discussed and presented in articles [4,5].
This paper seems to provide more empirical evidence in support to the existing
research [4,5]. Hence, the research contribution is unclear.
Furthermore, following points are worth considering:-
1. The related work should be discussed in detail highlighting the
advantages/limitations of existing approaches.
2. The dataset and codes are not available online.
3. The comparison of research with state of the art approaches and manual techniques
has not been conducted.
4. An overview diagram for the proposed approach would help the reader understand
the flow of the proposed approach.
5. The results are presented but not discussed. The section should be renamed to
"Results and Discussion" and appropriate discussion should be added with each pair of
graphs.
Powered by Editorial Manager® and ProduXion Manager® from Aries Systems Corporation
Responses:
Comment on Motivation: “The research lacks motivation as it is not clear what benefits
can be achieved if frame changes are detected.”
Framing changes have been shown to have commercial and legislative consequences,
and have also been shown to foreshadow public attention changes. We cite five
example articles here [1-5] and can readily provide more as necessary. A large body of
literature in the fields of Political Science and Communication addresses the manual
identification of framing changes in specific domains. Whereas we cite two examples
here [6-7], additional examples are available – please let us know. However, existing
work does not attempt to address the problem of computationally detecting framing
changes. Our work is the first attempt at this problem, which has significant
commercial, public, and legislative import. Our results substantially agree with the
results of earlier human surveys, and further have shown predictive utility for legislative
and public response. Our work therefore has significant scientific and potential
commercial value.
Revised sections:
We emphasize that our work is the first attempt at computationally modeling changes
in framing. The closest previous efforts in this area are those of [10] and [11]. We
describe our novel contributions over these efforts in detail in the Related Work
section. We are unaware of any other relevant related work and would be happy to
learn of any such work from the Editor.
This statement is incorrect. We have clearly stated in our submission that all data and
code will be made available, and are available online at the following link:
https://drive.google.com/open?id=1zAH__Y1lcdriuwUcjZsKmvaqYtzAjyZ9
All our results are reproducible from the data and code in the above mentioned
repository. We will provide a guide to run our code.
“The comparison of research with state of the art approaches and manual techniques
has not been conducted.”
Please refer to our responses above to the comment on motivation and comment #1.
4) “An overview diagram for the proposed approach would help the reader
understand the flow of the proposed approach.”
Powered by Editorial Manager® and ProduXion Manager® from Aries Systems Corporation
We are grateful for this suggestion and will incorporate an overview diagram illustrating
our approach. However, this is a simple suggestion for presentation that may easily be
addressed in a revision.
5) “The results are presented but not discussed. The section should be renamed to
"Results and Discussion" and appropriate discussion should be added with each pair of
graphs.”
The Results section discusses our results for each domain, using both a qualitative
comparison with manual surveys (by other authors) and by highlighting the predictive
utility of the returned result. We show that our results both agree with previous manual
surveys, and are also able to predict significant public and legislative response in each
domain. We will rename this section to “Results and Discussion”.
References:
1. A. C. Gunther, The persuasive press inference: Effects of mass media on perceived
public opinion. Commun. Res. 25, 486–504 (1998).
3.G. King, B. Schneer, A. White, How the news media activate public expression and
influence national agendas. Science 358, 776–780 (2017).
9 O. Tsur, D. Calacci, and D. Lazer, A Frame of Mind: Using Statistical Models for
Detection of Framing and Agenda Setting Campaigns. Proceedings of the 53rd Annual
Meeting of the Association for Computational Linguistics and the 7th International Joint
Conference on Natural Language Processing (Volume 1: Long Papers).
Additional Information:
Question Response
Financial Disclosure CS has a commercial affiliation to Amazon. The funder provided support in the form
of salaries for this author, but did not have any additional role in the study design, data
Enter a financial disclosure statement that collection and analysis, decision to publish, or preparation of the manuscript. The
describes the sources of funding for the specific roles of these authors are articulated in the `author contributions' section.
work included in this submission. Review
the submission guidelines for detailed
requirements. View published research
articles from PLOS ONE for specific
examples.
Powered by Editorial Manager® and ProduXion Manager® from Aries Systems Corporation
and will appear in the published article if
the submission is accepted. Please make
sure it is accurate.
Unfunded studies
Enter: The author(s) received no specific
funding for this work.
Funded studies
Enter a statement with the following details:
• Initials of the authors who received each
award
• Grant numbers awarded to each author
• The full name of each funder
• URL of each funder website
• Did the sponsors or funders play any role in
the study design, data collection and
analysis, decision to publish, or preparation
of the manuscript?
• NO - Include this sentence at the end of
your statement: The funders had no role in
study design, data collection and analysis,
decision to publish, or preparation of the
manuscript.
• YES - Specify the role(s) played.
* typeset
Use the instructions below to enter a The above commercial affiliation does not alter our adherence to PLOS ONE policies
competing interest statement for this on sharing
submission. On behalf of all authors, data and materials.
disclose any competing interests that
could be perceived to bias this
work—acknowledging all financial support
and any other relevant financial or non-
financial competing interests.
Powered by Editorial Manager® and ProduXion Manager® from Aries Systems Corporation
NO authors have competing interests
* typeset
• Human participants
• Human specimens or tissue
• Vertebrate animals or cephalopods
• Vertebrate embryos or tissues
• Field research
Powered by Editorial Manager® and ProduXion Manager® from Aries Systems Corporation
Format for specific study types
Field Research
Data Availability Yes - all data are fully available without restriction
Powered by Editorial Manager® and ProduXion Manager® from Aries Systems Corporation
A Data Availability Statement describing
where the data can be found is required at
submission. Your answers to this question
constitute the Data Availability Statement
and will be published in the article, if
accepted.
Describe where the data may be found in All data and most or all code will be made available in a Github/Google Drive
full sentences. If you are copying our repository.
sample text, replace any instances of XXX
with the appropriate details.
Powered by Editorial Manager® and ProduXion Manager® from Aries Systems Corporation
and contact information or URL).
• This text is appropriate if the data are
owned by a third party and authors do
not have permission to share the data.
* typeset
Additional data availability information: Tick here if the URLs/accession numbers/DOIs will be available only after acceptance
of the manuscript for publication so that we can ensure their inclusion before
publication.
Powered by Editorial Manager® and ProduXion Manager® from Aries Systems Corporation
Cover letter
NORTH CAROLINA
STATE UNIVERSITY Karthik Sheshadri
Karthik Sheshadri
Department of Computer Science
Raleigh, NC 27695-8206, USA
North Carolina State University
May 28, 2019 Phone: (919) 798-0203
E-mail: ksheshah@ncsu.edu
NORTH CAROLINA
STATE UNIVERSITY Karthik Sheshadri
Karthik Sheshadri
Department of Computer Science
Raleigh, NC 27695-8206, USA
North Carolina State University
March 3, 2020 Phone: (919) 798-0203
E-mail: ksheshah@ncsu.edu
* kshesha@ncsu.edu
Abstract
Changes in the framing of topical news have been shown to foreshadow significant
public, legislative, and commercial events. Automated detection of framing changes is
therefore an important problem, which existing research has not considered. Previous
approaches are manual surveys, which rely on human effort and are consequently
limited in scope. We make the following contributions. We systematize discovery of
framing changes through a fully unsupervised computational method that seeks to
isolate framing change trends over several years. We demonstrate our approach by
isolating framing change periods that correlate with previously known framing changes.
We have prepared a new dataset, consisting of over 12,000 articles from seven news
topics or domains in which earlier surveys have found framing changes. Finally, our
work highlights the predictive utility of framing change detection, by identifying two
domains in which framing changes foreshadowed substantial legislative activity, or
preceded judicial interest.
Related Work
The Media Frames Corpus, compiled by Card et al. [11], studies three topics
(Immigration, Smoking, and same-sex marriages), and identifies fifteen framing
dimensions in each. We identify two major limitations of their work. Firstly, Card et al.
study framing as a static detection problem, identifying which dimensions appear in a
given news article. However, research in sociology [10] shows that most news topics
feature a dominant frame (or dominant dimension in the terminology of [11]). Further,
for a generic news topic, the dominant frame is not necessarily one of fifteen previously
chosen dimensions, but can instead be an unknown arbitrary frame specific to the topic
under consideration. For example, in the example given in the Introduction and
Contributions section, the dominant frame related to the privacy of individuals, which is
not one of the fifteen dimensions described in Card et al. [11].
Secondly, Sheshadri and Singh [12] showed that public and legislative reaction tend
to occur only after changes in the dominant frame. That finding motivates an approach
to framing that focuses on identifying and detecting changes in the dominant frame of a
news domain.
Sheshadri and Singh further propose two simple metrics that they motivate as
measures of domain framing: framing polarity and density. They define framing polarity
as the average frequency of occurrence in a domain corpus of terms from a benchmark
sentiment lexicon. Framing density is measured using an entropic approach that counts
the number of terms per article required to distinguish a current corpus from an earlier
one.
We identify the following limitations of the aforementioned measures (introduced
in [12]). Firstly, both measures make no effort to associate a given news article with a
particular frame. Prior work does not support the inherent assumption that all articles
in a given domain belong to a particular frame [10, 11]. We enhance understanding by
analyzing each domain using several distinct frames.
Contributions
This paper contributes a fully unsupervised and data-driven natural language based
approach to detecting framing change trends over several years in domain news
publishing. To the best of our knowledge, this paper is the first to address framing
change detection, a problem of significant public and legislative import. Our approach
agrees with and extends the results of earlier manual surveys, which required human
data collection and were consequently limited in scope. Our approach removes this
restriction by being fully automated. Our method can thus be run simultaneously over
all news domains, limited only by the availability of real-time news data. Further, we
show that our approach yields results that foreshadow periods of legislative activity.
This motivates the predictive utility of our method for legislative activity, a problem of
significant import.
Further, we contribute a Framing Changes Dataset, which is a collection of over
12,000 news articles from seven news topics or domains. In four of these domains,
surveys carried out in earlier research have shown framing to change. In two domains,
periods with significant legislative activity are considered. Our individual domain
datasets within the framing changes dataset cover the years in which earlier research
found framing changes, as well as periods ranging up to ten years before and after the
change. Our dataset is the first to enable computational modeling of framing change
trends. We plan to release the dataset with our paper. We note that a fraction of the
articles in this dataset were used earlier for the analysis in [12].
Data Sources
We use two Application Programming Interfaces (APIs) to create our datasets.
Benchmark Datasets
We identified three open source benchmark review datasets from which to create our
adjective probability distribution. Together, these datasets provide about 150 million
reviews of various restaurants, services and products, with each review rated from one
to five. Given the large volume of reviews from different sources made available by these
datasets, we assume that they provide a sufficiently realistic representation of all
adjectives in the English language.
We rely primarily on the Trip Advisor dataset to create our adjective probability
distribution. We identified two other benchmark datasets, namely, the Yelp Challenge
dataset and the Amazon review dataset. Due to the fact that these datasets together
comprise about 150 million reviews, it is computationally infeasible for us to include
them in our learning procedure. Instead, we learned distributions from these datasets
for sample adjectives, to serve as a comparison with and as verification of our overall
learned distribution. The resulting distributions for these adjectives appeared
substantially similar to those of the corresponding adjectives in our learned distribution.
We therefore conclude that our learned distribution provides a valid representation of all
adjectives in the English language. We describe each dataset below.
Trip Advisor
The Trip Advisor dataset consists of 236,000 hotel reviews. Each review provides text,
an overall rating, and aspect specific ratings for the following seven aspects: Rooms,
Cleanliness, Value, Service, Location, Checkin, and Business. We limit ourselves to
using the overall rating of each review.
Amazon
The Amazon dataset provides approximately 143 million reviews from 24 product
categories such as Books, Electronics, Movies, and so on. The dataset uses the JSON
format and includes reviews comprising a rating, review text, and helpfulness votes.
Additionally, the JSON string encodes product metadata such as a product description,
category information, price, brand, and image features.
Polarity of Adjectives
For each adjective in the English language, we are interested in producing a probability
distribution that describes the relative likelihood of the adjective appearing in a review
whose rating is r. For our data, r ranges from one to five.
We began by compiling a set of reviews from the Trip Advisor dataset for each
rating from one to five. We used the Stanford CoreNLP parser [22] to parse each of the
five sets of reviews so obtained. We thus obtained sets of parses corresponding to each
review set. From the set of resultant parses, we extracted all words that were assigned a
part-of-speech of ‘JJ’ (adjective). Our search identified 454,281 unique adjectives.
For each unique adjective a, we counted the number of times it occurred in our set of
parses corresponding to review ratings one to five. We denote this by Ni , with
N1 N2 N5
1 ≤ i ≤ 5. Our probability vector for adjective a is then { Saa , Saa , . . . , Saa } where
Sa = Na1 + Na2 + Na3 + Na4 + Na5 .
Additionally, we recorded the rarity of each adjective as S1a . This estimates a
probability distribution P , with 454,281 rows and six columns.
Table 2 shows example entries from our learned probability distribution. As can be
seen from the table, our learned distribution not only correctly encodes probabilities
(the adjective ‘great’ has nearly 80% of its probability mass in the classes four and five,
whereas the adjective ‘horrible’ has nearly 80% of its mass in classes one and two), but
also implicitly learns an adjective ranking such as the one described in De Melo et
al. [23]. To illustrate this ranking, consider that the adjective ‘excellent’ has 60% of its
probability mass in class five, whereas the corresponding mass for the adjective ‘good’ is
only 38%.
For visual illustration, we depict our learned probability distribution as a heatmap in
Table 3.
Motivated by our learned probability distribution, we posit that classes 1 represents
negativity, class 2 to 4 represent neutrality, and class 5 represents positivity.
For a majority of our domains (five out of seven), we use a threshold of q > −∞,
that is, no adjectives are excluded. For the remaining two domains, (drones and LGBT
rights), we employ a threshold of q > 10−4 .
The trends in our results appeared to be fairly consistent across a reasonable range
of threshold values.
Corpus-Specific Representations
A domain corpus is a set of news articles from a given domain. Let a given domain have
m years in its period of interest with annual domain corpora T1 , T2 , . . . , Tm .
Corpus Clustering
An overall domain corpus is therefore T = T1 ∪ T2 ∪ . . . ∪ Tm .
We assume that a corpus has k unique frames. We adopt a standard topic modeling
approach to estimate frames. We use the benchmark Latent Dirichlet Allocation
(LDA) [24] approach to model k = 5 topics (that is, frames) in each domain corpus. We
extract the top l = 20 terms v from each frame. We also extract the set of all unique
nouns in T . We define a cluster as the set of nouns v ∩ T . We thus generate k clusters,
each representing a unique frame.
8/34
columns (see the Polarity of Adjectives section). We estimate the annual cluster polarity
of c as the vector of column-wise averages of Ai . Let Pc = {P1 , P2 , . . . , Pm } be the set
of annual cluster polarities so obtained.
Annual polarities for representative clusters from each of our domains are shown in
figures 11 to 15.
4
Drones
2
2003 2005 2007 2009 2011
1.00 Immigration
0.80
0.60
0.40
2000 2003 2005 2007 2009 2011 2013 2015 2017
4.00
2.00
LGBT Rights
0.00
1996 1999 2002 2005 2008 2011 2014
Fig 1. The average number of adjectives per article, shown for our domains over their
respective periods of interest. This metric serves as a measure of the subjectivity of
news in a domain. Notice that in the domain LGBT rights, the peak in this measure
immediately precedes a framing change identified in an earlier study [25].
10.00
Obesity
0.00
1990 1993 1996 1999 2002 2005 2008
40.00
30.00 Smoking
20.00
10.00
5.00
Surveillance
4.00
3.00
2.00
C1j
To measure the correlation of subset Ti–j , we compute its matrix of correlation
coefficients [27] K. We reshape K into a vector of size f × 1 where f = i ∗ j, and
evaluate its median, l. We find the maximum value of l, lmax , over all possible values of
i and j. We denote the values of i and j corresponding to lmax as imax and jmax . We
return Timax –jmax as our period of maximum correlation (PMC).
We note that the smaller the duration of a PMC, the greater the possibility that our
class vectors may have a high correlation in the period due to random chance. To
compensate for this effect, we employ a threshold whereby a period is not considered as
a candidate for the domain PMC unless it lasts at least y years. We uniformly employ a
value of y = 3 in this paper.
Our approach thus identifies polarity drifts that are both correlated (quantitatively
measured by correlations between different measures of polarity) and sustained (by the
imposition of a threshold of duration). We point out that our approach filters out
isolated drifts in individual polarity measures, since such drifts are uncorrelated across
multiple measures. Further, we note that the magnitude of individual drifts matters
only indirectly to our approach, to the extent that a larger drift, if consistent across
multiple polarity measures, may have higher correlation than a smaller drift that is also
correlated.
A block diagram depicting our overall approach is shown in figure 3.
Quantitative Evaluation
We now discuss a partial quantitative evaluation of our approach using a
Precision-Recall analysis. Our analysis relies on ground truth annotation of framing
changes, as detailed in the section below.
We are unable to conduct a full precision-recall analysis over all domains due to the
limitations we discuss in the following sections, as well as in the Qualitative Analysis
and Discussion section. However, we expect that our partial analysis is representative of
the general performance of the approach.
LDA
Precision-Recall Analysis
To gain confidence that our approach successfully identifies framing changes, we
conduct a precision-recall analysis on our data. We consider each year in each domain
as a data point in our analysis. We calculate overall precision and recall over all data
points in our domains. We consider a data point a true positive or true negative if both
a ground truth study and our approach labeled it as corresponding to a framing change,
or otherwise, respectively. We refer to a data point that was labeled as a positive (or
negative) by our approach, but which is a negative (or positive) according to the
relevant ground truth survey as a false positive or false negative, respectively.
Fig 4. Our estimated clusters for the domain abortion. Each cluster is said to
represent a unique frame. The frame discussed in cluster 1 (characterized by the terms
‘abortion’ and ‘ban’) concerns a proposed ban on abortion. We analyze this cluster, and
find that our estimated PMC (Figure 15) coincides with the period immediately
preceding the Partial Birth Abortion Act of 2003.
Fig 5. Our estimated clusters for the domain drones. Each cluster is said to represent
a unique frame. The frame discussed in cluster 1 concerns the use of drones against
terrorist targets. Our analysis of this cluster returns a PMC of 2009 to 2011 (Figure 17).
Our PMC immediately foreshadows the Federal Aviation Administration’s
Modernization and Reform Act of 2012.
Fig 7. Our estimated clusters for the domain obesity. Each cluster is said to represent
a unique frame. We posit that cluster 2 (characterized by the terms ‘food’, ‘diet’, and
‘make’) represents societal causes of obesity (see the Obesity section). We analyze this
cluster and estimate a PMC of 2005 to 2007 (Figure 13). Our PMC agrees with the
findings of an earlier human survey [2].
Fig 9. Our estimated clusters for the domain surveillance. Each cluster is said to
represent a unique frame. The frame of cluster 3, characterized by the terms ‘national‘,
‘security’, and ‘agency’, discusses the Snowden revelations of 2013. We analyze this
cluster and estimate a PMC of 2013 to 2014 (Figure 12). Our PMC coincides exactly
with the period following the Snowden revelations. Additionally, we note that the
Columbia Journalism Review [29] found that following the Snowden revelations, news
coverage of Surveillance changed to a narrative focusing on individual rights and digital
privacy [12].
Results
We find that our periods of maximum correlation correlate substantially with framing
changes described in earlier surveys [2, 29, 31, 32], and also foreshadow legislation.
Our computed class vectors are depicted in figures 11 to 15. We discuss each domain
below.
Smoking
The NCI published a monograph discussing the influence of the news media on tobacco
use [28]. On page 337, the monograph describes how, during the period 2001 to 2003,
American news media had progressed towards tobacco control frames. It states that
55% of articles in this period reported progress on tobacco control, whereas only 23%
reported setbacks.
In contrast, the monograph finds (also on page 337) that between 1985 to 1996,
tobacco control frames (11) were fairly well balanced with pro-tobacco frames (10). We
extracted a dataset of over 2,000 articles from 1990 to 2007.
Our approach returns a PMC of 2001 to 2003 (see figure 11) for this domain. Since
no studies cover the period 1997 to 2000 [28], we interpret the findings described in the
monograph to imply that the change towards tobacco control frames predominantly
began in 2000, and ended in 2003. This domain therefore contributes three true
positives (2001 to 2003) and one false negative (2000), with no false positives, to our
precision-recall analysis.
Surveillance
The CJR [29] found that following the Snowden revelations, news coverage of
Surveillance in the US changed to a narrative focusing on individual rights and digital
privacy [12]. We compiled a dataset consisting of approximately 2,000 surveillance
articles from the New York Times for the period 2010 to 2016.
0.240 Class 1
0.220
0.200
0.180
0.110
Class 2
0.100
0.150
0.140
Class 3
0.130
0.120
0.260
0.240
0.220
Class 4
Class 5
0.340
0.320
0.300
0.280
0.260
1990 1993 1996 1999 2002 2005
Fig 11. Annual polarities for cluster 3, (characterized by the terms ‘cancer’ and
‘smoke’), from Figure 8 from the domain smoking for the classes 1 to 5. The PMC is
shown with solid lines in square markers, and coincides exactly with a framing change
described in an earlier NCI monograph.
Obesity
Kim and Willis [2] found that the framing of obesity news underwent changes between
the years 1997 and 2004. During this period, Kim and Willis found that the fraction of
news frames attributing responsibility for obesity to social causes increased significantly.
Prior to this period, obesity tended to be framed as an issue of individual responsibility.
For example, obesity news after the year 2000 has often criticized food chains for their
excessive use of sugar in fast food, as shown in the NYT snippet in the Introduction and
Contributions section. We compiled a dataset of over 3,000 articles from the New York
Times (since Kim and Willis [2] restrict their study to Americans) from 1990 to 2009.
The clusters we estimate for this domain are shown in Figure 7. Cluster 2 addresses
possible causes of obesity, with a particular focus on dietary habits. We posit that this
cluster represents societal causes more than individual ones (since individual causes, as
shown in the NYT snippet of the Introduction and Contributions section tend to discuss
topics such as fitness and sedentary lifestyles, as opposed to food content). We observe
that the PMC for this domain (2005 to 2007) is characterized by increased positivity,
shown by classes 4 and 5, and decreased negativity (class 1). Our results for this
domain thus agree with the findings of Kim and Willis [2].
We were unable to use this domain in our precision-recall analysis, since Kim and
Willis, to the best of our knowledge, do not specify a precise period during which the
framing change took place.
However, since Figures 2 and 3 of Kim and Willis [2] show a dramatic increase of
social causes in 2004, and a corresponding marked decline of individual causes, we
conclude a substantial agreement between their findings and our results.
LGBT Rights
We compiled a dataset of over 3,000 articles from the period 1996 to 2015 in this domain.
Figure 6 depicts our estimated clusters. Cluster 3 represents a frame that discusses the
subject of same-sex marriage and its legality. We note that the Supreme Court ruled to
legalize same-sex marriages in the US in the year 2015. Our class vectors for this domain
are shown in figure 14. We obtained two PMCs with nearly identical correlation scores
(0.999 for the period 2006 to 2008, and 0.989 for the period 2013 to 2015). Figure 14
highlights the period 2013 to 2015 immediately preceding the judicial interest of 2015.
We were unable to identify a prior study that discusses the framing of LGBT news
over our entire period of interest. However, we use the findings reported in Gainous et
al. [32] as our ground truth for this domain. Gainous et al. studied the framing of
LGBT related publishing in the New York Times over the period 1988 to 2012, and
found a dramatic increase in equality frames between approximately 25 in 2008 and
approximately 110 in 2012. Correspondingly, our findings of Figure 14 show that
between 2008 and 2012, there was a dramatic increase in the measures of classes 4 and 5
0.20
0.15
0.14
0.12 Class 2
0.1
0.14
0.12 Class 3
0.1
0.24
0.22
Class 4
0.2
0.18
Class 5
0.35
0.3
2010 2011 2012 2013 2014 2015 2016
Fig 12. Annual polarities for a representative cluster (characterized by the terms
‘national‘, ‘security’, and ‘agency’) from the domain surveillance for the classes 1 to 5.
The PMC is shown with solid lines in square markers.
Class 1
0.25
0.2
0.120 Class 2
0.100
0.15
0.14
Class 3
0.13
0.12
0.24
Class 4
0.22
0.2
0.36 Class 5
0.34
0.32
0.3
0.28
1990 1993 1996 1999 2002 2005 2008
Fig 13. Annual polarities for cluster 2 (characterized by the terms ‘diet’, ‘food’, and
‘make’) from Figure 7 from the domain obesity for the classes 1 to 5. The PMC is shown
with solid lines in square markers. We posit that this cluster represents societal causes
of obesity (see the Obesity section). We observe that the PMC for this cluster (2005 to
2007) agrees with the findings of Kim and Willis [2].
Abortion
The Partial-Birth Abortion Ban Act was enacted in 2003. We obtained 248 articles for
the period 2000 to 2003, for this domain. We obtain a PMC of 2001 to 2003 for this
domain, as shown in figure 15.
Immigration
We study the framing of immigration news in the United Kingdom. We obtained about
3,600 articles on the subject of Immigration from the Guardian API for the period 2000
to 2017. For this domain, we carried out our analysis on the article titles (rather than
the full text). Since the Guardian returns full length articles, we found that this design
choice allows us to produce a more focused domain corpus than the one generated by
the full article text. We depict our estimated class vectors and PMC in figure 16.
We analyze the frame of cluster 2 in Figure 10. This cluster deals with the issue of
asylum seekers to the United Kingdom. In the period beginning immediately before the
year 2000,a new peak in asylum claims to the United Kingdom of 76,040 had been
reached [33]. This event coincided with a high-profile terrorist act by a set of Afghan
asylum seekers [33].
These events resulted in increased border refusals and the final 2002 white paper on
“Secure Borders, Safe Haven.” We estimate a PMC of 2000 to 2002 (Figure 16). Our
PMC coincides exactly with the period immediately foreshadowing the government
white paper.
Drones
We obtained nearly 4,000 articles on this domain for the period 2003 to 2012. We
obtain a PMC of 2009 to 2011 for this domain, as shown in Figure 17.
Our PMC immediately foreshadows the Federal Aviation Administration’s
Modernization and Reform Act of 2012.
Predictive Utility
The aforementioned two domains (immigration and drones) highlight the predictive
utility of news framing. Whereas we did not find earlier surveys that coincide with our
PMCs for these domains, we note that these PMCs foreshadowed substantial legislative
activity. This observation suggests that PMCs estimated through real-time monitoring
of domain news may yield predictive utility for legislative and commercial activity.
0.180
0.160
0.120
Class 2
0.115
0.110
0.105
0.150 Class 3
0.140
0.130
0.250
0.240 Class 4
0.230
0.220
0.350
Class 5
0.340
0.330
1996 1998 2000 2002 2004 2006 2008 2010 2012 2014
Fig 14. Annual polarities for cluster 3, characterized by the terms ‘gay’, ‘rights’, and
‘marriage’, in Figure 6 from the domain LGBT Rights for the classes 1 to 5. We obtain
two PMCs with nearly identical correlation scores, namely, 2006 to 2008 and 2013 to
2015. The PMC of 2013 to 2015 is shown with solid lines in square markers,
immediately preceding the judicial interest of 2015.
0.26
0.24
0.12
0.11
Class 2
0.10
0.09
0.14
Class 3
0.13
0.12
Class 4
0.22
0.20
0.18
0.30
Class 5
0.28
0.26
0.400
0.300
0.200 Class 1
0.15
0.10
0.05 Class 2
0.15
0.10
Class 3
0.25
0.20
0.15 Class 4
0.50
0.40
0.30
0.20 Class 5
0.160
0.140
0.100
Class 2
0.090
Class 3
0.140
0.130
Class 4
0.260
0.240
0.400
0.380
Class 5
0.360
0.340
Conclusion
We highlight a problem of significant public and legislative importance, framing change
detection. We contribute an unsupervised natural language based approach that detects
framing change trends over several years in domain news publishing. We identify a key
characteristic of such changes, namely, that during frame changes, the polarity of
adjectives describing cooccurring nouns changes cumulatively over multiple years. Our
approach agrees with and extends the results of earlier manual surveys. Whereas such
surveys depend on human effort and are therefore limited in scope, our approach is fully
automated and can simultaneously run over all news domains. We contribute the
Framing Changes Dataset, a collection of over 12,000 news articles from seven domains
in which framing has been shown to change by earlier surveys. We will release the
dataset with our paper. Our work suggests the predictive utility of automated news
monitoring, as a means to foreshadow events of commercial and legislative import.
Our work represents one of the first attempts at a computational modeling of
framing and framing changes. We therefore claim that our approach produces promising
results, and that it will serve as a baseline for more sophisticated analysis over wider
temporal and geographical data.
Ethics Statement
Our study involved no human or animal subjects.
Funding Statement
CS has a commercial affiliation to Amazon. The funder provided support in the form of
salaries for this author, but did not have any additional role in the study design, data
collection and analysis, decision to publish, or preparation of the manuscript. The
specific roles of these authors are articulated in the ‘author contributions’ section.
Author Contributions
KS and CS conceived the research and designed the method. KS prepared the datasets
and performed the analysis. KS and MPS designed the evaluation approach. KS, CS,
and MPS wrote the paper.
References
1. Gunnars K. Ten Causes of Weight Gain in America; 2015.
https://www.healthline.com/nutrition/10-causes-of-weight-gain#section12.
2. Kim SH, Willis A. Talking about Obesity: News Framing of Who Is Responsible
for Causing and Fixing the Problem. Journal of Health Communication.
2007;12(4):359–376.
3. Flegal K, Carroll M, Kit B, Ogden C. Prevalence of Obesity and Trends in the
Distribution of Body Mass Index Among US Adults, 1999–2010. Journal of the
American Medical Association. 2012;307(5):491–497.
* kshesha@ncsu.edu
Abstract
Changes in the framing of topical news have been shown to foreshadow significant
public, legislative, and commercial events. Automated detection of framing changes is
therefore an important problem, which existing research has not considered. Previous
approaches are manual surveys, which rely on human effort and are consequently
limited in scope. We make the following contributions. We systematize discovery of
framing changes through a fully unsupervised computational method that seeks to
isolate framing change trends over several years. We demonstrate our approach by
isolating framing change periods that correlate with previously known framing changes.
We have prepared a new dataset, consisting of over 12,000 articles from seven news
topics or domains in which earlier surveys have found framing changes. Finally, our
work highlights the predictive utility of framing change detection, by identifying two
domains in which framing changes foreshadowed substantial legislative activity, or
preceded judicial interest.
Related Work
The Media Frames Corpus, compiled by Card et al. [11], studies three topics
(Immigration, Smoking, and same-sex marriages), and identifies fifteen framing
dimensions in each. We identify two major limitations of their work. Firstly, Card et al.
study framing as a static detection problem, identifying which dimensions appear in a
given news article. However, research in sociology [10] shows that most news topics
feature a dominant frame (or dominant dimension in the terminology of [11]). Further,
for a generic news topic, the dominant frame is not necessarily one of fifteen previously
chosen dimensions, but can instead be an unknown arbitrary frame specific to the topic
under consideration. For example, in the example given in the Introduction and
Contributions section, the dominant frame related to the privacy of individuals, which is
not one of the fifteen dimensions described in Card et al. [11].
Secondly, Sheshadri and Singh [12] showed that public and legislative reaction tend
to occur only after changes in the dominant frame. That finding motivates an approach
to framing that focuses on identifying and detecting changes in the dominant frame of a
news domain.
Sheshadri and Singh further propose two simple metrics that they motivate as
measures of domain framing: framing polarity and density. They define framing polarity
as the average frequency of occurrence in a domain corpus of terms from a benchmark
sentiment lexicon. Framing density is measured using an entropic approach that counts
the number of terms per article required to distinguish a current corpus from an earlier
one.
We identify the following limitations of the aforementioned measures (introduced
in [12]). Firstly, both measures make no effort to associate a given news article with a
particular frame. Prior work does not support the inherent assumption that all articles
in a given domain belong to a particular frame [10, 11]. We enhance understanding by
analyzing each domain using several distinct frames.
Contributions
This paper contributes a fully unsupervised and data-driven natural language based
approach to detecting framing change trends over several years in domain news
publishing. To the best of our knowledge, this paper is the first to address framing
change detection, a problem of significant public and legislative import. Our approach
agrees with and extends the results of earlier manual surveys, which required human
data collection and were consequently limited in scope. Our approach removes this
restriction by being fully automated. Our method can thus be run simultaneously over
all news domains, limited only by the availability of real-time news data. Further, we
show that our approach yields results that foreshadow periods of legislative activity.
This motivates the predictive utility of our method for legislative activity, a problem of
significant import.
Further, we contribute a Framing Changes Dataset, which is a collection of over
12,000 news articles from seven news topics or domains. In four of these domains,
surveys carried out in earlier research have shown framing to change. In two domains,
periods with significant legislative activity are considered. Our individual domain
datasets within the framing changes dataset cover the years in which earlier research
found framing changes, as well as periods ranging up to ten years before and after the
change. Our dataset is the first to enable computational modeling of framing change
trends. We plan to release the dataset with our paper. We note that a fraction of the
articles in this dataset were used earlier for the analysis in [12].
Data Sources
We use two Application Programming Interfaces (APIs) to create our datasets.
Benchmark Datasets
We identified three open source benchmark review datasets from which to create our
adjective probability distribution. Together, these datasets provide about 150 million
reviews of various restaurants, services and products, with each review rated from one
to five. Given the large volume of reviews from different sources made available by these
datasets, we assume that they provide a sufficiently realistic representation of all
adjectives in the English language.
We rely primarily on the Trip Advisor dataset to create our adjective probability
distribution. We identified two other benchmark datasets, namely, the Yelp Challenge
dataset and the Amazon review dataset. Due to the fact that these datasets together
comprise about 150 million reviews, it is computationally infeasible for us to include
them in our learning procedure. Instead, we learned distributions from these datasets
for sample adjectives, to serve as a comparison with and as verification of our overall
learned distribution. The resulting distributions for these adjectives appeared
substantially similar to those of the corresponding adjectives in our learned distribution.
We therefore conclude that our learned distribution provides a valid representation of all
adjectives in the English language. We describe each dataset below.
Trip Advisor
The Trip Advisor dataset consists of 236,000 hotel reviews. Each review provides text,
an overall rating, and aspect specific ratings for the following seven aspects: Rooms,
Cleanliness, Value, Service, Location, Checkin, and Business. We limit ourselves to
using the overall rating of each review.
Amazon
The Amazon dataset provides approximately 143 million reviews from 24 product
categories such as Books, Electronics, Movies, and so on. The dataset uses the JSON
format and includes reviews comprising a rating, review text, and helpfulness votes.
Additionally, the JSON string encodes product metadata such as a product description,
category information, price, brand, and image features.
Polarity of Adjectives
For each adjective in the English language, we are interested in producing a probability
distribution that describes the relative likelihood of the adjective appearing in a review
whose rating is r. For our data, r ranges from one to five.
We began by compiling a set of reviews from the Trip Advisor dataset for each
rating from one to five. We used the Stanford CoreNLP parser [22] to parse each of the
five sets of reviews so obtained. We thus obtained sets of parses corresponding to each
review set. From the set of resultant parses, we extracted all words that were assigned a
part-of-speech of ‘JJ’ (adjective). Our search identified 454,281 unique adjectives.
For each unique adjective a, we counted the number of times it occurred in our set of
parses corresponding to review ratings one to five. We denote this by Ni , with
N1 N2 N5
1 ≤ i ≤ 5. Our probability vector for adjective a is then { Saa , Saa , . . . , Saa } where
Sa = Na1 + Na2 + Na3 + Na4 + Na5 .
Additionally, we recorded the rarity of each adjective as S1a . This estimates a
probability distribution P , with 454,281 rows and six columns.
Table 2 shows example entries from our learned probability distribution. As can be
seen from the table, our learned distribution not only correctly encodes probabilities
(the adjective ‘great’ has nearly 80% of its probability mass in the classes four and five,
whereas the adjective ‘horrible’ has nearly 80% of its mass in classes one and two), but
also implicitly learns an adjective ranking such as the one described in De Melo et
al. [23]. To illustrate this ranking, consider that the adjective ‘excellent’ has 60% of its
probability mass in class five, whereas the corresponding mass for the adjective ‘good’ is
only 38%.
For visual illustration, we depict our learned probability distribution as a heatmap in
Table 3.
Motivated by our learned probability distribution, we posit that classes 1 represents
negativity, class 2 to 4 represent neutrality, and class 5 represents positivity.
For a majority of our domains (five out of seven), we use a threshold of q > −∞,
that is, no adjectives are excluded. For the remaining two domains, (drones and LGBT
rights), we employ a threshold of q > 10−4 .
The trends in our results appeared to be fairly consistent across a reasonable range
of threshold values.
Corpus-Specific Representations
A domain corpus is a set of news articles from a given domain. Let a given domain have
m years in its period of interest with annual domain corpora T1 , T2 , . . . , Tm .
Corpus Clustering
An overall domain corpus is therefore T = T1 ∪ T2 ∪ . . . ∪ Tm .
We assume that a corpus has k unique frames. We adopt a standard topic modeling
approach to estimate frames. We use the benchmark Latent Dirichlet Allocation
(LDA) [24] approach to model k = 5 topics (that is, frames) in each domain corpus. We
extract the top l = 20 terms v from each frame. We also extract the set of all unique
nouns in T . We define a cluster as the set of nouns v ∩ T . We thus generate k clusters,
each representing a unique frame.
8/34
columns (see the Polarity of Adjectives section). We estimate the annual cluster polarity
of c as the vector of column-wise averages of Ai . Let Pc = {P1 , P2 , . . . , Pm } be the set
of annual cluster polarities so obtained.
Annual polarities for representative clusters from each of our domains are shown in
figures 11 to 15.
4
Drones
2
2003 2005 2007 2009 2011
1.00 Immigration
0.80
0.60
0.40
2000 2003 2005 2007 2009 2011 2013 2015 2017
4.00
2.00
LGBT Rights
0.00
1996 1999 2002 2005 2008 2011 2014
Fig 1. The average number of adjectives per article, shown for our domains over their
respective periods of interest. This metric serves as a measure of the subjectivity of
news in a domain. Notice that in the domain LGBT rights, the peak in this measure
immediately precedes a framing change identified in an earlier study [25].
10.00
Obesity
0.00
1990 1993 1996 1999 2002 2005 2008
40.00
30.00 Smoking
20.00
10.00
5.00
Surveillance
4.00
3.00
2.00
C1j
To measure the correlation of subset Ti–j , we compute its matrix of correlation
coefficients [27] K. We reshape K into a vector of size f × 1 where f = i ∗ j, and
evaluate its median, l. We find the maximum value of l, lmax , over all possible values of
i and j. We denote the values of i and j corresponding to lmax as imax and jmax . We
return Timax –jmax as our period of maximum correlation (PMC).
We note that the smaller the duration of a PMC, the greater the possibility that our
class vectors may have a high correlation in the period due to random chance. To
compensate for this effect, we employ a threshold whereby a period is not considered as
a candidate for the domain PMC unless it lasts at least y years. We uniformly employ a
value of y = 3 in this paper.
Our approach thus identifies polarity drifts that are both correlated (quantitatively
measured by correlations between different measures of polarity) and sustained (by the
imposition of a threshold of duration). We point out that our approach filters out
isolated drifts in individual polarity measures, since such drifts are uncorrelated across
multiple measures. Further, we note that the magnitude of individual drifts matters
only indirectly to our approach, to the extent that a larger drift, if consistent across
multiple polarity measures, may have higher correlation than a smaller drift that is also
correlated.
A block diagram depicting our overall approach is shown in figure 3.
Quantitative Evaluation
We now discuss a partial quantitative evaluation of our approach using a
Precision-Recall analysis. Our analysis relies on ground truth annotation of framing
changes, as detailed in the section below.
We are unable to conduct a full precision-recall analysis over all domains due to the
limitations we discuss in the following sections, as well as in the Qualitative Analysis
and Discussion section. However, we expect that our partial analysis is representative of
the general performance of the approach.
LDA
Precision-Recall Analysis
To gain confidence that our approach successfully identifies framing changes, we
conduct a precision-recall analysis on our data. We consider each year in each domain
as a data point in our analysis. We calculate overall precision and recall over all data
points in our domains. We consider a data point a true positive or true negative if both
a ground truth study and our approach labeled it as corresponding to a framing change,
or otherwise, respectively. We refer to a data point that was labeled as a positive (or
negative) by our approach, but which is a negative (or positive) according to the
relevant ground truth survey as a false positive or false negative, respectively.
Fig 4. Our estimated clusters for the domain abortion. Each cluster is said to
represent a unique frame. The frame discussed in cluster 1 (characterized by the terms
‘abortion’ and ‘ban’) concerns a proposed ban on abortion. We analyze this cluster, and
find that our estimated PMC (Figure 15) coincides with the period immediately
preceding the Partial Birth Abortion Act of 2003.
Fig 5. Our estimated clusters for the domain drones. Each cluster is said to represent
a unique frame. The frame discussed in cluster 1 concerns the use of drones against
terrorist targets. Our analysis of this cluster returns a PMC of 2009 to 2011 (Figure 17).
Our PMC immediately foreshadows the Federal Aviation Administration’s
Modernization and Reform Act of 2012.
Fig 7. Our estimated clusters for the domain obesity. Each cluster is said to represent
a unique frame. We posit that cluster 2 (characterized by the terms ‘food’, ‘diet’, and
‘make’) represents societal causes of obesity (see the Obesity section). We analyze this
cluster and estimate a PMC of 2005 to 2007 (Figure 13). Our PMC agrees with the
findings of an earlier human survey [2].
Fig 9. Our estimated clusters for the domain surveillance. Each cluster is said to
represent a unique frame. The frame of cluster 3, characterized by the terms ‘national‘,
‘security’, and ‘agency’, discusses the Snowden revelations of 2013. We analyze this
cluster and estimate a PMC of 2013 to 2014 (Figure 12). Our PMC coincides exactly
with the period following the Snowden revelations. Additionally, we note that the
Columbia Journalism Review [29] found that following the Snowden revelations, news
coverage of Surveillance changed to a narrative focusing on individual rights and digital
privacy [12].
Results
We find that our periods of maximum correlation correlate substantially with framing
changes described in earlier surveys [2, 29, 31, 32], and also foreshadow legislation.
Our computed class vectors are depicted in figures 11 to 15. We discuss each domain
below.
Smoking
The NCI published a monograph discussing the influence of the news media on tobacco
use [28]. On page 337, the monograph describes how, during the period 2001 to 2003,
American news media had progressed towards tobacco control frames. It states that
55% of articles in this period reported progress on tobacco control, whereas only 23%
reported setbacks.
In contrast, the monograph finds (also on page 337) that between 1985 to 1996,
tobacco control frames (11) were fairly well balanced with pro-tobacco frames (10). We
extracted a dataset of over 2,000 articles from 1990 to 2007.
Our approach returns a PMC of 2001 to 2003 (see figure 11) for this domain. Since
no studies cover the period 1997 to 2000 [28], we interpret the findings described in the
monograph to imply that the change towards tobacco control frames predominantly
began in 2000, and ended in 2003. This domain therefore contributes three true
positives (2001 to 2003) and one false negative (2000), with no false positives, to our
precision-recall analysis.
Surveillance
The CJR [29] found that following the Snowden revelations, news coverage of
Surveillance in the US changed to a narrative focusing on individual rights and digital
privacy [12]. We compiled a dataset consisting of approximately 2,000 surveillance
articles from the New York Times for the period 2010 to 2016.
0.240 Class 1
0.220
0.200
0.180
0.110
Class 2
0.100
0.150
0.140
Class 3
0.130
0.120
0.260
0.240
0.220
Class 4
Class 5
0.340
0.320
0.300
0.280
0.260
1990 1993 1996 1999 2002 2005
Fig 11. Annual polarities for cluster 3, (characterized by the terms ‘cancer’ and
‘smoke’), from Figure 8 from the domain smoking for the classes 1 to 5. The PMC is
shown with solid lines in square markers, and coincides exactly with a framing change
described in an earlier NCI monograph.
Obesity
Kim and Willis [2] found that the framing of obesity news underwent changes between
the years 1997 and 2004. During this period, Kim and Willis found that the fraction of
news frames attributing responsibility for obesity to social causes increased significantly.
Prior to this period, obesity tended to be framed as an issue of individual responsibility.
For example, obesity news after the year 2000 has often criticized food chains for their
excessive use of sugar in fast food, as shown in the NYT snippet in the Introduction and
Contributions section. We compiled a dataset of over 3,000 articles from the New York
Times (since Kim and Willis [2] restrict their study to Americans) from 1990 to 2009.
The clusters we estimate for this domain are shown in Figure 7. Cluster 2 addresses
possible causes of obesity, with a particular focus on dietary habits. We posit that this
cluster represents societal causes more than individual ones (since individual causes, as
shown in the NYT snippet of the Introduction and Contributions section tend to discuss
topics such as fitness and sedentary lifestyles, as opposed to food content). We observe
that the PMC for this domain (2005 to 2007) is characterized by increased positivity,
shown by classes 4 and 5, and decreased negativity (class 1). Our results for this
domain thus agree with the findings of Kim and Willis [2].
We were unable to use this domain in our precision-recall analysis, since Kim and
Willis, to the best of our knowledge, do not specify a precise period during which the
framing change took place.
However, since Figures 2 and 3 of Kim and Willis [2] show a dramatic increase of
social causes in 2004, and a corresponding marked decline of individual causes, we
conclude a substantial agreement between their findings and our results.
LGBT Rights
We compiled a dataset of over 3,000 articles from the period 1996 to 2015 in this domain.
Figure 6 depicts our estimated clusters. Cluster 3 represents a frame that discusses the
subject of same-sex marriage and its legality. We note that the Supreme Court ruled to
legalize same-sex marriages in the US in the year 2015. Our class vectors for this domain
are shown in figure 14. We obtained two PMCs with nearly identical correlation scores
(0.999 for the period 2006 to 2008, and 0.989 for the period 2013 to 2015). Figure 14
highlights the period 2013 to 2015 immediately preceding the judicial interest of 2015.
We were unable to identify a prior study that discusses the framing of LGBT news
over our entire period of interest. However, we use the findings reported in Gainous et
al. [32] as our ground truth for this domain. Gainous et al. studied the framing of
LGBT related publishing in the New York Times over the period 1988 to 2012, and
found a dramatic increase in equality frames between approximately 25 in 2008 and
0.20
0.15
0.14
0.12 Class 2
0.1
0.14
0.12 Class 3
0.1
0.24
0.22
Class 4
0.2
0.18
Class 5
0.35
0.3
2010 2011 2012 2013 2014 2015 2016
Fig 12. Annual polarities for a representative cluster (characterized by the terms
‘national‘, ‘security’, and ‘agency’) from the domain surveillance for the classes 1 to 5.
The PMC is shown with solid lines in square markers.
Class 1
0.25
0.2
0.120 Class 2
0.100
0.15
0.14
Class 3
0.13
0.12
0.24
Class 4
0.22
0.2
0.36 Class 5
0.34
0.32
0.3
0.28
1990 1993 1996 1999 2002 2005 2008
Fig 13. Annual polarities for cluster 2 (characterized by the terms ‘diet’, ‘food’, and
‘make’) from Figure 7 from the domain obesity for the classes 1 to 5. The PMC is shown
with solid lines in square markers. We posit that this cluster represents societal causes
of obesity (see the Obesity section). We observe that the PMC for this cluster (2005 to
2007) agrees with the findings of Kim and Willis [2].
Abortion
The Partial-Birth Abortion Ban Act was enacted in 2003. We obtained 248 articles for
the period 2000 to 2003, for this domain. We obtain a PMC of 2001 to 2003 for this
domain, as shown in figure 15.
Immigration
We study the framing of immigration news in the United Kingdom. We obtained about
3,600 articles on the subject of Immigration from the Guardian API for the period 2000
to 2017. For this domain, we carried out our analysis on the article titles (rather than
the full text). Since the Guardian returns full length articles, we found that this design
choice allows us to produce a more focused domain corpus than the one generated by
the full article text. We depict our estimated class vectors and PMC in figure 16.
We analyze the frame of cluster 2 in Figure 10. This cluster deals with the issue of
asylum seekers to the United Kingdom. In the period beginning immediately before the
year 2000,a new peak in asylum claims to the United Kingdom of 76,040 had been
reached [33]. This event coincided with a high-profile terrorist act by a set of Afghan
asylum seekers [33].
These events resulted in increased border refusals and the final 2002 white paper on
“Secure Borders, Safe Haven.” We estimate a PMC of 2000 to 2002 (Figure 16). Our
PMC coincides exactly with the period immediately foreshadowing the government
white paper.
Drones
We obtained nearly 4,000 articles on this domain for the period 2003 to 2012. We
obtain a PMC of 2009 to 2011 for this domain, as shown in Figure 17.
Our PMC immediately foreshadows the Federal Aviation Administration’s
Modernization and Reform Act of 2012.
Predictive Utility
The aforementioned two domains (immigration and drones) highlight the predictive
utility of news framing. Whereas we did not find earlier surveys that coincide with our
PMCs for these domains, we note that these PMCs foreshadowed substantial legislative
0.180
0.160
0.120
Class 2
0.115
0.110
0.105
0.150 Class 3
0.140
0.130
0.250
0.240 Class 4
0.230
0.220
0.350
Class 5
0.340
0.330
1996 1998 2000 2002 2004 2006 2008 2010 2012 2014
Fig 14. Annual polarities for cluster 3, characterized by the terms ‘gay’, ‘rights’, and
‘marriage’, in Figure 6 from the domain LGBT Rights for the classes 1 to 5. We obtain
two PMCs with nearly identical correlation scores, namely, 2006 to 2008 and 2013 to
2015. The PMC of 2013 to 2015 is shown with solid lines in square markers,
immediately preceding the judicial interest of 2015.
0.26
0.24
0.12
0.11
Class 2
0.10
0.09
0.14
Class 3
0.13
0.12
Class 4
0.22
0.20
0.18
0.30
Class 5
0.28
0.26
0.400
0.300
0.200 Class 1
0.15
0.10
0.05 Class 2
0.15
0.10
Class 3
0.25
0.20
0.15 Class 4
0.50
0.40
0.30
0.20 Class 5
0.160
0.140
0.100
Class 2
0.090
Class 3
0.140
0.130
Class 4
0.260
0.240
0.400
0.380
Class 5
0.360
0.340
Conclusion
We highlight a problem of significant public and legislative importance, framing change
detection. We contribute an unsupervised natural language based approach that detects
framing change trends over several years in domain news publishing. We identify a key
characteristic of such changes, namely, that during frame changes, the polarity of
adjectives describing cooccurring nouns changes cumulatively over multiple years. Our
approach agrees with and extends the results of earlier manual surveys. Whereas such
surveys depend on human effort and are therefore limited in scope, our approach is fully
automated and can simultaneously run over all news domains. We contribute the
Framing Changes Dataset, a collection of over 12,000 news articles from seven domains
in which framing has been shown to change by earlier surveys. We will release the
dataset with our paper. Our work suggests the predictive utility of automated news
monitoring, as a means to foreshadow events of commercial and legislative import.
Our work represents one of the first attempts at a computational modeling of
framing and framing changes. We therefore claim that our approach produces promising
results, and that it will serve as a baseline for more sophisticated analysis over wider
temporal and geographical data.
Ethics Statement
Our study involved no human or animal subjects.
Funding Statement
CS has a commercial affiliation to Amazon. The funder provided support in the form of
salaries for this author, but did not have any additional role in the study design, data
collection and analysis, decision to publish, or preparation of the manuscript. The
specific roles of these authors are articulated in the ‘author contributions’ section.
Author Contributions
KS and CS conceived the research and designed the method. KS prepared the datasets
and performed the analysis. KS and MPS designed the evaluation approach. KS, CS,
and MPS wrote the paper.
References
1. Gunnars K. Ten Causes of Weight Gain in America; 2015.
https://www.healthline.com/nutrition/10-causes-of-weight-gain#section12.
2. Kim SH, Willis A. Talking about Obesity: News Framing of Who Is Responsible
for Causing and Fixing the Problem. Journal of Health Communication.
2007;12(4):359–376.
3. Flegal K, Carroll M, Kit B, Ogden C. Prevalence of Obesity and Trends in the
Distribution of Body Mass Index Among US Adults, 1999–2010. Journal of the
American Medical Association. 2012;307(5):491–497.
* kshesha@ncsu.edu
Abstract
Changes in the framing of topical news have been shown to foreshadow significant
public, legislative, and commercial events. Automated detection of framing changes is
therefore an important problem, which existing research has not considered. Previous
approaches are manual surveys, which rely on human effort and are consequently
limited in scope. We make the following contributions. We systematize discovery of
framing changes through a fully unsupervised computational method that seeks to
isolate framing change trends over several years. We demonstrate our approach by
isolating framing change periods that correlate with previously known framing changes.
We have prepared a new dataset, consisting of over 12,000 articles from seven news
topics or domains in which earlier surveys have found framing changes. Finally, our
work highlights the predictive utility of framing change detection, by identifying two
domains in which framing changes foreshadowed substantial legislative activity, or
preceded judicial interest.
Related Work
The Media Frames Corpus, compiled by Card et al. [11], studies three topics
(Immigration, Smoking, and same-sex marriages), and identifies fifteen framing
dimensions in each. We identify two major limitations of their work. Firstly, Card et al.
study framing as a static detection problem, identifying which dimensions appear in a
given news article. However, research in sociology [10] shows that most news topics
feature a dominant frame (or dominant dimension in the terminology of [11]). Further,
for a generic news topic, the dominant frame is not necessarily one of fifteen previously
chosen dimensions, but can instead be an unknown arbitrary frame specific to the topic
under consideration. For example, in the example given in the Introduction and
Contributions section, the dominant frame related to the privacy of individuals, which is
not one of the fifteen dimensions described in Card et al. [11].
Secondly, Sheshadri and Singh [12] showed that public and legislative reaction tend
to occur only after changes in the dominant frame. That finding motivates an approach
to framing that focuses on identifying and detecting changes in the dominant frame of a
news domain.
Sheshadri and Singh further propose two simple metrics that they motivate as
measures of domain framing: framing polarity and density. They define framing polarity
as the average frequency of occurrence in a domain corpus of terms from a benchmark
sentiment lexicon. Framing density is measured using an entropic approach that counts
the number of terms per article required to distinguish a current corpus from an earlier
one.
We identify the following limitations of the aforementioned measures (introduced
in [12]). Firstly, both measures make no effort to associate a given news article with a
particular frame. Prior work does not support the inherent assumption that all articles
in a given domain belong to a particular frame [10, 11]. We enhance understanding by
analyzing each domain using several distinct frames.
Contributions
This paper contributes a fully unsupervised and data-driven natural language based
approach to detecting framing change trends over several years in domain news
publishing. To the best of our knowledge, this paper is the first to address framing
change detection, a problem of significant public and legislative import. Our approach
agrees with and extends the results of earlier manual surveys, which required human
data collection and were consequently limited in scope. Our approach removes this
restriction by being fully automated. Our method can thus be run simultaneously over
all news domains, limited only by the availability of real-time news data. Further, we
show that our approach yields results that foreshadow periods of legislative activity.
This motivates the predictive utility of our method for legislative activity, a problem of
significant import.
Further, we contribute a Framing Changes Dataset, which is a collection of over
12,000 news articles from seven news topics or domains. In four of these domains,
surveys carried out in earlier research have shown framing to change. In two domains,
periods with significant legislative activity are considered. Our individual domain
datasets within the framing changes dataset cover the years in which earlier research
found framing changes, as well as periods ranging up to ten years before and after the
change. Our dataset is the first to enable computational modeling of framing change
trends. We plan to release the dataset with our paper. We note that a fraction of the
articles in this dataset were used earlier for the analysis in [12].
Data Sources
We use two Application Programming Interfaces (APIs) to create our datasets.
Benchmark Datasets
We identified three open source benchmark review datasets from which to create our
adjective probability distribution. Together, these datasets provide about 150 million
reviews of various restaurants, services and products, with each review rated from one
to five. Given the large volume of reviews from different sources made available by these
datasets, we assume that they provide a sufficiently realistic representation of all
adjectives in the English language.
We rely primarily on the Trip Advisor dataset to create our adjective probability
distribution. We identified two other benchmark datasets, namely, the Yelp Challenge
dataset and the Amazon review dataset. Due to the fact that these datasets together
comprise about 150 million reviews, it is computationally infeasible for us to include
them in our learning procedure. Instead, we learned distributions from these datasets
for sample adjectives, to serve as a comparison with and as verification of our overall
learned distribution. The resulting distributions for these adjectives appeared
substantially similar to those of the corresponding adjectives in our learned distribution.
We therefore conclude that our learned distribution provides a valid representation of all
adjectives in the English language. We describe each dataset below.
Trip Advisor
The Trip Advisor dataset consists of 236,000 hotel reviews. Each review provides text,
an overall rating, and aspect specific ratings for the following seven aspects: Rooms,
Cleanliness, Value, Service, Location, Checkin, and Business. We limit ourselves to
using the overall rating of each review.
Amazon
The Amazon dataset provides approximately 143 million reviews from 24 product
categories such as Books, Electronics, Movies, and so on. The dataset uses the JSON
format and includes reviews comprising a rating, review text, and helpfulness votes.
Additionally, the JSON string encodes product metadata such as a product description,
category information, price, brand, and image features.
Polarity of Adjectives
For each adjective in the English language, we are interested in producing a probability
distribution that describes the relative likelihood of the adjective appearing in a review
whose rating is r. For our data, r ranges from one to five.
We began by compiling a set of reviews from the Trip Advisor dataset for each
rating from one to five. We used the Stanford CoreNLP parser [22] to parse each of the
five sets of reviews so obtained. We thus obtained sets of parses corresponding to each
review set. From the set of resultant parses, we extracted all words that were assigned a
part-of-speech of ‘JJ’ (adjective). Our search identified 454,281 unique adjectives.
For each unique adjective a, we counted the number of times it occurred in our set of
parses corresponding to review ratings one to five. We denote this by Ni , with
N1 N2 N5
1 ≤ i ≤ 5. Our probability vector for adjective a is then { Saa , Saa , . . . , Saa } where
Sa = Na1 + Na2 + Na3 + Na4 + Na5 .
Additionally, we recorded the rarity of each adjective as S1a . This estimates a
probability distribution P , with 454,281 rows and six columns.
Table 2 shows example entries from our learned probability distribution. As can be
seen from the table, our learned distribution not only correctly encodes probabilities
(the adjective ‘great’ has nearly 80% of its probability mass in the classes four and five,
whereas the adjective ‘horrible’ has nearly 80% of its mass in classes one and two), but
also implicitly learns an adjective ranking such as the one described in De Melo et
al. [23]. To illustrate this ranking, consider that the adjective ‘excellent’ has 60% of its
probability mass in class five, whereas the corresponding mass for the adjective ‘good’ is
only 38%.
For visual illustration, we depict our learned probability distribution as a heatmap in
Table 3.
Motivated by our learned probability distribution, we posit that classes 1 represents
negativity, class 2 to 4 represent neutrality, and class 5 represents positivity.
For a majority of our domains (five out of seven), we use a threshold of q > −∞,
that is, no adjectives are excluded. For the remaining two domains, (drones and LGBT
rights), we employ a threshold of q > 10−4 .
The trends in our results appeared to be fairly consistent across a reasonable range
of threshold values.
Corpus-Specific Representations
A domain corpus is a set of news articles from a given domain. Let a given domain have
m years in its period of interest with annual domain corpora T1 , T2 , . . . , Tm .
Corpus Clustering
An overall domain corpus is therefore T = T1 ∪ T2 ∪ . . . ∪ Tm .
We assume that a corpus has k unique frames. We adopt a standard topic modeling
approach to estimate frames. We use the benchmark Latent Dirichlet Allocation
(LDA) [24] approach to model k = 5 topics (that is, frames) in each domain corpus. We
extract the top l = 20 terms v from each frame. We also extract the set of all unique
nouns in T . We define a cluster as the set of nouns v ∩ T . We thus generate k clusters,
each representing a unique frame.
8/34
columns (see the Polarity of Adjectives section). We estimate the annual cluster polarity
of c as the vector of column-wise averages of Ai . Let Pc = {P1 , P2 , . . . , Pm } be the set
of annual cluster polarities so obtained.
Annual polarities for representative clusters from each of our domains are shown in
figures 11 to 15.
4
Drones
2
2003 2005 2007 2009 2011
1.00 Immigration
0.80
0.60
0.40
2000 2003 2005 2007 2009 2011 2013 2015 2017
4.00
2.00
LGBT Rights
0.00
1996 1999 2002 2005 2008 2011 2014
Fig 1. The average number of adjectives per article, shown for our domains over their
respective periods of interest. This metric serves as a measure of the subjectivity of
news in a domain. Notice that in the domain LGBT rights, the peak in this measure
immediately precedes a framing change identified in an earlier study [25].
10.00
Obesity
0.00
1990 1993 1996 1999 2002 2005 2008
40.00
30.00 Smoking
20.00
10.00
5.00
Surveillance
4.00
3.00
2.00
C1j
To measure the correlation of subset Ti–j , we compute its matrix of correlation
coefficients [27] K. We reshape K into a vector of size f × 1 where f = i ∗ j, and
evaluate its median, l. We find the maximum value of l, lmax , over all possible values of
i and j. We denote the values of i and j corresponding to lmax as imax and jmax . We
return Timax –jmax as our period of maximum correlation (PMC).
We note that the smaller the duration of a PMC, the greater the possibility that our
class vectors may have a high correlation in the period due to random chance. To
compensate for this effect, we employ a threshold whereby a period is not considered as
a candidate for the domain PMC unless it lasts at least y years. We uniformly employ a
value of y = 3 in this paper.
Our approach thus identifies polarity drifts that are both correlated (quantitatively
measured by correlations between different measures of polarity) and sustained (by the
imposition of a threshold of duration). We point out that our approach filters out
isolated drifts in individual polarity measures, since such drifts are uncorrelated across
multiple measures. Further, we note that the magnitude of individual drifts matters
only indirectly to our approach, to the extent that a larger drift, if consistent across
multiple polarity measures, may have higher correlation than a smaller drift that is also
correlated.
A block diagram depicting our overall approach is shown in figure 3.
Quantitative Evaluation
We now discuss a partial quantitative evaluation of our approach using a
Precision-Recall analysis. Our analysis relies on ground truth annotation of framing
changes, as detailed in the section below.
We are unable to conduct a full precision-recall analysis over all domains due to the
limitations we discuss in the following sections, as well as in the Qualitative Analysis
and Discussion section. However, we expect that our partial analysis is representative of
the general performance of the approach.
LDA
Precision-Recall Analysis
To gain confidence that our approach successfully identifies framing changes, we
conduct a precision-recall analysis on our data. We consider each year in each domain
as a data point in our analysis. We calculate overall precision and recall over all data
points in our domains. We consider a data point a true positive or true negative if both
a ground truth study and our approach labeled it as corresponding to a framing change,
or otherwise, respectively. We refer to a data point that was labeled as a positive (or
negative) by our approach, but which is a negative (or positive) according to the
relevant ground truth survey as a false positive or false negative, respectively.
Fig 4. Our estimated clusters for the domain abortion. Each cluster is said to
represent a unique frame. The frame discussed in cluster 1 (characterized by the terms
‘abortion’ and ‘ban’) concerns a proposed ban on abortion. We analyze this cluster, and
find that our estimated PMC (Figure 15) coincides with the period immediately
preceding the Partial Birth Abortion Act of 2003.
Fig 5. Our estimated clusters for the domain drones. Each cluster is said to represent
a unique frame. The frame discussed in cluster 1 concerns the use of drones against
terrorist targets. Our analysis of this cluster returns a PMC of 2009 to 2011 (Figure 17).
Our PMC immediately foreshadows the Federal Aviation Administration’s
Modernization and Reform Act of 2012.
Fig 7. Our estimated clusters for the domain obesity. Each cluster is said to represent
a unique frame. We posit that cluster 2 (characterized by the terms ‘food’, ‘diet’, and
‘make’) represents societal causes of obesity (see the Obesity section). We analyze this
cluster and estimate a PMC of 2005 to 2007 (Figure 13). Our PMC agrees with the
findings of an earlier human survey [2].
Fig 9. Our estimated clusters for the domain surveillance. Each cluster is said to
represent a unique frame. The frame of cluster 3, characterized by the terms ‘national‘,
‘security’, and ‘agency’, discusses the Snowden revelations of 2013. We analyze this
cluster and estimate a PMC of 2013 to 2014 (Figure 12). Our PMC coincides exactly
with the period following the Snowden revelations. Additionally, we note that the
Columbia Journalism Review [29] found that following the Snowden revelations, news
coverage of Surveillance changed to a narrative focusing on individual rights and digital
privacy [12].
Results
We find that our periods of maximum correlation correlate substantially with framing
changes described in earlier surveys [2, 29, 31, 32], and also foreshadow legislation.
Our computed class vectors are depicted in figures 11 to 15. We discuss each domain
below.
Smoking
The NCI published a monograph discussing the influence of the news media on tobacco
use [28]. On page 337, the monograph describes how, during the period 2001 to 2003,
American news media had progressed towards tobacco control frames. It states that
55% of articles in this period reported progress on tobacco control, whereas only 23%
reported setbacks.
In contrast, the monograph finds (also on page 337) that between 1985 to 1996,
tobacco control frames (11) were fairly well balanced with pro-tobacco frames (10). We
extracted a dataset of over 2,000 articles from 1990 to 2007.
Our approach returns a PMC of 2001 to 2003 (see figure 11) for this domain. Since
no studies cover the period 1997 to 2000 [28], we interpret the findings described in the
monograph to imply that the change towards tobacco control frames predominantly
began in 2000, and ended in 2003. This domain therefore contributes three true
positives (2001 to 2003) and one false negative (2000), with no false positives, to our
precision-recall analysis.
Surveillance
The CJR [29] found that following the Snowden revelations, news coverage of
Surveillance in the US changed to a narrative focusing on individual rights and digital
privacy [12]. We compiled a dataset consisting of approximately 2,000 surveillance
articles from the New York Times for the period 2010 to 2016.
0.240 Class 1
0.220
0.200
0.180
0.110
Class 2
0.100
0.150
0.140
Class 3
0.130
0.120
0.260
0.240
0.220
Class 4
Class 5
0.340
0.320
0.300
0.280
0.260
1990 1993 1996 1999 2002 2005
Fig 11. Annual polarities for cluster 3, (characterized by the terms ‘cancer’ and
‘smoke’), from Figure 8 from the domain smoking for the classes 1 to 5. The PMC is
shown with solid lines in square markers, and coincides exactly with a framing change
described in an earlier NCI monograph.
Obesity
Kim and Willis [2] found that the framing of obesity news underwent changes between
the years 1997 and 2004. During this period, Kim and Willis found that the fraction of
news frames attributing responsibility for obesity to social causes increased significantly.
Prior to this period, obesity tended to be framed as an issue of individual responsibility.
For example, obesity news after the year 2000 has often criticized food chains for their
excessive use of sugar in fast food, as shown in the NYT snippet in the Introduction and
Contributions section. We compiled a dataset of over 3,000 articles from the New York
Times (since Kim and Willis [2] restrict their study to Americans) from 1990 to 2009.
The clusters we estimate for this domain are shown in Figure 7. Cluster 2 addresses
possible causes of obesity, with a particular focus on dietary habits. We posit that this
cluster represents societal causes more than individual ones (since individual causes, as
shown in the NYT snippet of the Introduction and Contributions section tend to discuss
topics such as fitness and sedentary lifestyles, as opposed to food content). We observe
that the PMC for this domain (2005 to 2007) is characterized by increased positivity,
shown by classes 4 and 5, and decreased negativity (class 1). Our results for this
domain thus agree with the findings of Kim and Willis [2].
We were unable to use this domain in our precision-recall analysis, since Kim and
Willis, to the best of our knowledge, do not specify a precise period during which the
framing change took place.
However, since Figures 2 and 3 of Kim and Willis [2] show a dramatic increase of
social causes in 2004, and a corresponding marked decline of individual causes, we
conclude a substantial agreement between their findings and our results.
LGBT Rights
We compiled a dataset of over 3,000 articles from the period 1996 to 2015 in this domain.
Figure 6 depicts our estimated clusters. Cluster 3 represents a frame that discusses the
subject of same-sex marriage and its legality. We note that the Supreme Court ruled to
legalize same-sex marriages in the US in the year 2015. Our class vectors for this domain
are shown in figure 14. We obtained two PMCs with nearly identical correlation scores
(0.999 for the period 2006 to 2008, and 0.989 for the period 2013 to 2015). Figure 14
highlights the period 2013 to 2015 immediately preceding the judicial interest of 2015.
We were unable to identify a prior study that discusses the framing of LGBT news
over our entire period of interest. However, we use the findings reported in Gainous et
al. [32] as our ground truth for this domain. Gainous et al. studied the framing of
LGBT related publishing in the New York Times over the period 1988 to 2012, and
found a dramatic increase in equality frames between approximately 25 in 2008 and
approximately 110 in 2012. Correspondingly, our findings of Figure 14 show that
between 2008 and 2012, there was a dramatic increase in the measures of classes 4 and 5
0.20
0.15
0.14
0.12 Class 2
0.1
0.14
0.12 Class 3
0.1
0.24
0.22
Class 4
0.2
0.18
Class 5
0.35
0.3
2010 2011 2012 2013 2014 2015 2016
Fig 12. Annual polarities for a representative cluster (characterized by the terms
‘national‘, ‘security’, and ‘agency’) from the domain surveillance for the classes 1 to 5.
The PMC is shown with solid lines in square markers.
Class 1
0.25
0.2
0.120 Class 2
0.100
0.15
0.14
Class 3
0.13
0.12
0.24
Class 4
0.22
0.2
0.36 Class 5
0.34
0.32
0.3
0.28
1990 1993 1996 1999 2002 2005 2008
Fig 13. Annual polarities for cluster 2 (characterized by the terms ‘diet’, ‘food’, and
‘make’) from Figure 7 from the domain obesity for the classes 1 to 5. The PMC is shown
with solid lines in square markers. We posit that this cluster represents societal causes
of obesity (see the Obesity section). We observe that the PMC for this cluster (2005 to
2007) agrees with the findings of Kim and Willis [2].
Abortion
The Partial-Birth Abortion Ban Act was enacted in 2003. We obtained 248 articles for
the period 2000 to 2003, for this domain. We obtain a PMC of 2001 to 2003 for this
domain, as shown in figure 15.
Immigration
We study the framing of immigration news in the United Kingdom. We obtained about
3,600 articles on the subject of Immigration from the Guardian API for the period 2000
to 2017. For this domain, we carried out our analysis on the article titles (rather than
the full text). Since the Guardian returns full length articles, we found that this design
choice allows us to produce a more focused domain corpus than the one generated by
the full article text. We depict our estimated class vectors and PMC in figure 16.
We analyze the frame of cluster 2 in Figure 10. This cluster deals with the issue of
asylum seekers to the United Kingdom. In the period beginning immediately before the
year 2000,a new peak in asylum claims to the United Kingdom of 76,040 had been
reached [33]. This event coincided with a high-profile terrorist act by a set of Afghan
asylum seekers [33].
These events resulted in increased border refusals and the final 2002 white paper on
“Secure Borders, Safe Haven.” We estimate a PMC of 2000 to 2002 (Figure 16). Our
PMC coincides exactly with the period immediately foreshadowing the government
white paper.
Drones
We obtained nearly 4,000 articles on this domain for the period 2003 to 2012. We
obtain a PMC of 2009 to 2011 for this domain, as shown in Figure 17.
Our PMC immediately foreshadows the Federal Aviation Administration’s
Modernization and Reform Act of 2012.
Predictive Utility
The aforementioned two domains (immigration and drones) highlight the predictive
utility of news framing. Whereas we did not find earlier surveys that coincide with our
PMCs for these domains, we note that these PMCs foreshadowed substantial legislative
activity. This observation suggests that PMCs estimated through real-time monitoring
of domain news may yield predictive utility for legislative and commercial activity.
0.180
0.160
0.120
Class 2
0.115
0.110
0.105
0.150 Class 3
0.140
0.130
0.250
0.240 Class 4
0.230
0.220
0.350
Class 5
0.340
0.330
1996 1998 2000 2002 2004 2006 2008 2010 2012 2014
Fig 14. Annual polarities for cluster 3, characterized by the terms ‘gay’, ‘rights’, and
‘marriage’, in Figure 6 from the domain LGBT Rights for the classes 1 to 5. We obtain
two PMCs with nearly identical correlation scores, namely, 2006 to 2008 and 2013 to
2015. The PMC of 2013 to 2015 is shown with solid lines in square markers,
immediately preceding the judicial interest of 2015.
0.26
0.24
0.12
0.11
Class 2
0.10
0.09
0.14
Class 3
0.13
0.12
Class 4
0.22
0.20
0.18
0.30
Class 5
0.28
0.26
0.400
0.300
0.200 Class 1
0.15
0.10
0.05 Class 2
0.15
0.10
Class 3
0.25
0.20
0.15 Class 4
0.50
0.40
0.30
0.20 Class 5
0.160
0.140
0.100
Class 2
0.090
Class 3
0.140
0.130
Class 4
0.260
0.240
0.400
0.380
Class 5
0.360
0.340
Conclusion
We highlight a problem of significant public and legislative importance, framing change
detection. We contribute an unsupervised natural language based approach that detects
framing change trends over several years in domain news publishing. We identify a key
characteristic of such changes, namely, that during frame changes, the polarity of
adjectives describing cooccurring nouns changes cumulatively over multiple years. Our
approach agrees with and extends the results of earlier manual surveys. Whereas such
surveys depend on human effort and are therefore limited in scope, our approach is fully
automated and can simultaneously run over all news domains. We contribute the
Framing Changes Dataset, a collection of over 12,000 news articles from seven domains
in which framing has been shown to change by earlier surveys. We will release the
dataset with our paper. Our work suggests the predictive utility of automated news
monitoring, as a means to foreshadow events of commercial and legislative import.
Our work represents one of the first attempts at a computational modeling of
framing and framing changes. We therefore claim that our approach produces promising
results, and that it will serve as a baseline for more sophisticated analysis over wider
temporal and geographical data.
Ethics Statement
Our study involved no human or animal subjects.
Funding Statement
CS has a commercial affiliation to Amazon. The funder provided support in the form of
salaries for this author, but did not have any additional role in the study design, data
collection and analysis, decision to publish, or preparation of the manuscript. The
specific roles of these authors are articulated in the ‘author contributions’ section.
Author Contributions
KS and CS conceived the research and designed the method. KS prepared the datasets
and performed the analysis. KS and MPS designed the evaluation approach. KS, CS,
and MPS wrote the paper.
References
1. Gunnars K. Ten Causes of Weight Gain in America; 2015.
https://www.healthline.com/nutrition/10-causes-of-weight-gain#section12.
2. Kim SH, Willis A. Talking about Obesity: News Framing of Who Is Responsible
for Causing and Fixing the Problem. Journal of Health Communication.
2007;12(4):359–376.
3. Flegal K, Carroll M, Kit B, Ogden C. Prevalence of Obesity and Trends in the
Distribution of Body Mass Index Among US Adults, 1999–2010. Journal of the
American Medical Association. 2012;307(5):491–497.
* kshesha@ncsu.edu
Abstract
Changes in the framing of topical news have been shown to foreshadow significant
public, legislative, and commercial events. Automated detection of framing changes is
therefore an important problem, which existing research has not considered. Previous
approaches are manual surveys, which rely on human effort and are consequently
limited in scope. We make the following contributions. We systematize discovery of
framing changes through a fully unsupervised computational method that seeks to
isolate framing change trends over several years. We demonstrate our approach by
isolating framing change periods that correlate with previously known framing changes.
We have prepared a new dataset, consisting of over 12,000 articles from seven news
topics or domains in which earlier surveys have found framing changes. Finally, our
work highlights the predictive utility of framing change detection, by identifying two
domains in which framing changes foreshadowed substantial legislative activity, or
preceded judicial interest.
Related Work
The Media Frames Corpus, compiled by Card et al. [11], studies three topics
(Immigration, Smoking, and same-sex marriages), and identifies fifteen framing
dimensions in each. We identify two major limitations of their work. Firstly, Card et al.
study framing as a static detection problem, identifying which dimensions appear in a
given news article. However, research in sociology [10] shows that most news topics
feature a dominant frame (or dominant dimension in the terminology of [11]). Further,
for a generic news topic, the dominant frame is not necessarily one of fifteen previously
chosen dimensions, but can instead be an unknown arbitrary frame specific to the topic
under consideration. For example, in the example given in the Introduction and
Contributions section, the dominant frame related to the privacy of individuals, which is
not one of the fifteen dimensions described in Card et al. [11].
Secondly, Sheshadri and Singh [12] showed that public and legislative reaction tend
to occur only after changes in the dominant frame. That finding motivates an approach
to framing that focuses on identifying and detecting changes in the dominant frame of a
news domain.
Sheshadri and Singh further propose two simple metrics that they motivate as
measures of domain framing: framing polarity and density. They define framing polarity
as the average frequency of occurrence in a domain corpus of terms from a benchmark
sentiment lexicon. Framing density is measured using an entropic approach that counts
the number of terms per article required to distinguish a current corpus from an earlier
one.
We identify the following limitations of the aforementioned measures (introduced
in [12]). Firstly, both measures make no effort to associate a given news article with a
particular frame. Prior work does not support the inherent assumption that all articles
in a given domain belong to a particular frame [10, 11]. We enhance understanding by
analyzing each domain using several distinct frames.
Contributions
This paper contributes a fully unsupervised and data-driven natural language based
approach to detecting framing change trends over several years in domain news
publishing. To the best of our knowledge, this paper is the first to address framing
change detection, a problem of significant public and legislative import. Our approach
agrees with and extends the results of earlier manual surveys, which required human
data collection and were consequently limited in scope. Our approach removes this
restriction by being fully automated. Our method can thus be run simultaneously over
all news domains, limited only by the availability of real-time news data. Further, we
show that our approach yields results that foreshadow periods of legislative activity.
This motivates the predictive utility of our method for legislative activity, a problem of
significant import.
Further, we contribute a Framing Changes Dataset, which is a collection of over
12,000 news articles from seven news topics or domains. In four of these domains,
surveys carried out in earlier research have shown framing to change. In two domains,
periods with significant legislative activity are considered. Our individual domain
datasets within the framing changes dataset cover the years in which earlier research
found framing changes, as well as periods ranging up to ten years before and after the
change. Our dataset is the first to enable computational modeling of framing change
trends. We plan to release the dataset with our paper. We note that a fraction of the
articles in this dataset were used earlier for the analysis in [12].
Data Sources
We use two Application Programming Interfaces (APIs) to create our datasets.
Benchmark Datasets
We identified three open source benchmark review datasets from which to create our
adjective probability distribution. Together, these datasets provide about 150 million
reviews of various restaurants, services and products, with each review rated from one
to five. Given the large volume of reviews from different sources made available by these
datasets, we assume that they provide a sufficiently realistic representation of all
adjectives in the English language.
We rely primarily on the Trip Advisor dataset to create our adjective probability
distribution. We identified two other benchmark datasets, namely, the Yelp Challenge
dataset and the Amazon review dataset. Due to the fact that these datasets together
comprise about 150 million reviews, it is computationally infeasible for us to include
them in our learning procedure. Instead, we learned distributions from these datasets
for sample adjectives, to serve as a comparison with and as verification of our overall
learned distribution. The resulting distributions for these adjectives appeared
substantially similar to those of the corresponding adjectives in our learned distribution.
We therefore conclude that our learned distribution provides a valid representation of all
adjectives in the English language. We describe each dataset below.
Trip Advisor
The Trip Advisor dataset consists of 236,000 hotel reviews. Each review provides text,
an overall rating, and aspect specific ratings for the following seven aspects: Rooms,
Cleanliness, Value, Service, Location, Checkin, and Business. We limit ourselves to
using the overall rating of each review.
Amazon
The Amazon dataset provides approximately 143 million reviews from 24 product
categories such as Books, Electronics, Movies, and so on. The dataset uses the JSON
format and includes reviews comprising a rating, review text, and helpfulness votes.
Additionally, the JSON string encodes product metadata such as a product description,
category information, price, brand, and image features.
Polarity of Adjectives
For each adjective in the English language, we are interested in producing a probability
distribution that describes the relative likelihood of the adjective appearing in a review
whose rating is r. For our data, r ranges from one to five.
We began by compiling a set of reviews from the Trip Advisor dataset for each
rating from one to five. We used the Stanford CoreNLP parser [22] to parse each of the
five sets of reviews so obtained. We thus obtained sets of parses corresponding to each
review set. From the set of resultant parses, we extracted all words that were assigned a
part-of-speech of ‘JJ’ (adjective). Our search identified 454,281 unique adjectives.
For each unique adjective a, we counted the number of times it occurred in our set of
parses corresponding to review ratings one to five. We denote this by Ni , with
N1 N2 N5
1 ≤ i ≤ 5. Our probability vector for adjective a is then { Saa , Saa , . . . , Saa } where
Sa = Na1 + Na2 + Na3 + Na4 + Na5 .
Additionally, we recorded the rarity of each adjective as S1a . This estimates a
probability distribution P , with 454,281 rows and six columns.
Table 2 shows example entries from our learned probability distribution. As can be
seen from the table, our learned distribution not only correctly encodes probabilities
(the adjective ‘great’ has nearly 80% of its probability mass in the classes four and five,
whereas the adjective ‘horrible’ has nearly 80% of its mass in classes one and two), but
also implicitly learns an adjective ranking such as the one described in De Melo et
al. [23]. To illustrate this ranking, consider that the adjective ‘excellent’ has 60% of its
probability mass in class five, whereas the corresponding mass for the adjective ‘good’ is
only 38%.
For visual illustration, we depict our learned probability distribution as a heatmap in
Table 3.
Motivated by our learned probability distribution, we posit that classes 1 represents
negativity, class 2 to 4 represent neutrality, and class 5 represents positivity.
For a majority of our domains (five out of seven), we use a threshold of q > −∞,
that is, no adjectives are excluded. For the remaining two domains, (drones and LGBT
rights), we employ a threshold of q > 10−4 .
The trends in our results appeared to be fairly consistent across a reasonable range
of threshold values.
Corpus-Specific Representations
A domain corpus is a set of news articles from a given domain. Let a given domain have
m years in its period of interest with annual domain corpora T1 , T2 , . . . , Tm .
Corpus Clustering
An overall domain corpus is therefore T = T1 ∪ T2 ∪ . . . ∪ Tm .
We assume that a corpus has k unique frames. We adopt a standard topic modeling
approach to estimate frames. We use the benchmark Latent Dirichlet Allocation
(LDA) [24] approach to model k = 5 topics (that is, frames) in each domain corpus. We
extract the top l = 20 terms v from each frame. We also extract the set of all unique
nouns in T . We define a cluster as the set of nouns v ∩ T . We thus generate k clusters,
each representing a unique frame.
8/34
columns (see the Polarity of Adjectives section). We estimate the annual cluster polarity
of c as the vector of column-wise averages of Ai . Let Pc = {P1 , P2 , . . . , Pm } be the set
of annual cluster polarities so obtained.
Annual polarities for representative clusters from each of our domains are shown in
figures 11 to 15.
4
Drones
2
2003 2005 2007 2009 2011
1.00 Immigration
0.80
0.60
0.40
2000 2003 2005 2007 2009 2011 2013 2015 2017
4.00
2.00
LGBT Rights
0.00
1996 1999 2002 2005 2008 2011 2014
Fig 1. The average number of adjectives per article, shown for our domains over their
respective periods of interest. This metric serves as a measure of the subjectivity of
news in a domain. Notice that in the domain LGBT rights, the peak in this measure
immediately precedes a framing change identified in an earlier study [25].
10.00
Obesity
0.00
1990 1993 1996 1999 2002 2005 2008
40.00
30.00 Smoking
20.00
10.00
5.00
Surveillance
4.00
3.00
2.00
C1j
To measure the correlation of subset Ti–j , we compute its matrix of correlation
coefficients [27] K. We reshape K into a vector of size f × 1 where f = i ∗ j, and
evaluate its median, l. We find the maximum value of l, lmax , over all possible values of
i and j. We denote the values of i and j corresponding to lmax as imax and jmax . We
return Timax –jmax as our period of maximum correlation (PMC).
We note that the smaller the duration of a PMC, the greater the possibility that our
class vectors may have a high correlation in the period due to random chance. To
compensate for this effect, we employ a threshold whereby a period is not considered as
a candidate for the domain PMC unless it lasts at least y years. We uniformly employ a
value of y = 3 in this paper.
Our approach thus identifies polarity drifts that are both correlated (quantitatively
measured by correlations between different measures of polarity) and sustained (by the
imposition of a threshold of duration). We point out that our approach filters out
isolated drifts in individual polarity measures, since such drifts are uncorrelated across
multiple measures. Further, we note that the magnitude of individual drifts matters
only indirectly to our approach, to the extent that a larger drift, if consistent across
multiple polarity measures, may have higher correlation than a smaller drift that is also
correlated.
A block diagram depicting our overall approach is shown in figure 3.
Quantitative Evaluation
We now discuss a partial quantitative evaluation of our approach using a
Precision-Recall analysis. Our analysis relies on ground truth annotation of framing
changes, as detailed in the section below.
We are unable to conduct a full precision-recall analysis over all domains due to the
limitations we discuss in the following sections, as well as in the Qualitative Analysis
and Discussion section. However, we expect that our partial analysis is representative of
the general performance of the approach.
LDA
Precision-Recall Analysis
To gain confidence that our approach successfully identifies framing changes, we
conduct a precision-recall analysis on our data. We consider each year in each domain
as a data point in our analysis. We calculate overall precision and recall over all data
points in our domains. We consider a data point a true positive or true negative if both
a ground truth study and our approach labeled it as corresponding to a framing change,
or otherwise, respectively. We refer to a data point that was labeled as a positive (or
negative) by our approach, but which is a negative (or positive) according to the
relevant ground truth survey as a false positive or false negative, respectively.
Fig 4. Our estimated clusters for the domain abortion. Each cluster is said to
represent a unique frame. The frame discussed in cluster 1 (characterized by the terms
‘abortion’ and ‘ban’) concerns a proposed ban on abortion. We analyze this cluster, and
find that our estimated PMC (Figure 15) coincides with the period immediately
preceding the Partial Birth Abortion Act of 2003.
Fig 5. Our estimated clusters for the domain drones. Each cluster is said to represent
a unique frame. The frame discussed in cluster 1 concerns the use of drones against
terrorist targets. Our analysis of this cluster returns a PMC of 2009 to 2011 (Figure 17).
Our PMC immediately foreshadows the Federal Aviation Administration’s
Modernization and Reform Act of 2012.
Fig 7. Our estimated clusters for the domain obesity. Each cluster is said to represent
a unique frame. We posit that cluster 2 (characterized by the terms ‘food’, ‘diet’, and
‘make’) represents societal causes of obesity (see the Obesity section). We analyze this
cluster and estimate a PMC of 2005 to 2007 (Figure 13). Our PMC agrees with the
findings of an earlier human survey [2].
Fig 9. Our estimated clusters for the domain surveillance. Each cluster is said to
represent a unique frame. The frame of cluster 3, characterized by the terms ‘national‘,
‘security’, and ‘agency’, discusses the Snowden revelations of 2013. We analyze this
cluster and estimate a PMC of 2013 to 2014 (Figure 12). Our PMC coincides exactly
with the period following the Snowden revelations. Additionally, we note that the
Columbia Journalism Review [29] found that following the Snowden revelations, news
coverage of Surveillance changed to a narrative focusing on individual rights and digital
privacy [12].
Results
We find that our periods of maximum correlation correlate substantially with framing
changes described in earlier surveys [2, 29, 31, 32], and also foreshadow legislation.
Our computed class vectors are depicted in figures 11 to 15. We discuss each domain
below.
Smoking
The NCI published a monograph discussing the influence of the news media on tobacco
use [28]. On page 337, the monograph describes how, during the period 2001 to 2003,
American news media had progressed towards tobacco control frames. It states that
55% of articles in this period reported progress on tobacco control, whereas only 23%
reported setbacks.
In contrast, the monograph finds (also on page 337) that between 1985 to 1996,
tobacco control frames (11) were fairly well balanced with pro-tobacco frames (10). We
extracted a dataset of over 2,000 articles from 1990 to 2007.
Our approach returns a PMC of 2001 to 2003 (see figure 11) for this domain. Since
no studies cover the period 1997 to 2000 [28], we interpret the findings described in the
monograph to imply that the change towards tobacco control frames predominantly
began in 2000, and ended in 2003. This domain therefore contributes three true
positives (2001 to 2003) and one false negative (2000), with no false positives, to our
precision-recall analysis.
Surveillance
The CJR [29] found that following the Snowden revelations, news coverage of
Surveillance in the US changed to a narrative focusing on individual rights and digital
privacy [12]. We compiled a dataset consisting of approximately 2,000 surveillance
articles from the New York Times for the period 2010 to 2016.
0.240 Class 1
0.220
0.200
0.180
0.110
Class 2
0.100
0.150
0.140
Class 3
0.130
0.120
0.260
0.240
0.220
Class 4
Class 5
0.340
0.320
0.300
0.280
0.260
1990 1993 1996 1999 2002 2005
Fig 11. Annual polarities for cluster 3, (characterized by the terms ‘cancer’ and
‘smoke’), from Figure 8 from the domain smoking for the classes 1 to 5. The PMC is
shown with solid lines in square markers, and coincides exactly with a framing change
described in an earlier NCI monograph.
Obesity
Kim and Willis [2] found that the framing of obesity news underwent changes between
the years 1997 and 2004. During this period, Kim and Willis found that the fraction of
news frames attributing responsibility for obesity to social causes increased significantly.
Prior to this period, obesity tended to be framed as an issue of individual responsibility.
For example, obesity news after the year 2000 has often criticized food chains for their
excessive use of sugar in fast food, as shown in the NYT snippet in the Introduction and
Contributions section. We compiled a dataset of over 3,000 articles from the New York
Times (since Kim and Willis [2] restrict their study to Americans) from 1990 to 2009.
The clusters we estimate for this domain are shown in Figure 7. Cluster 2 addresses
possible causes of obesity, with a particular focus on dietary habits. We posit that this
cluster represents societal causes more than individual ones (since individual causes, as
shown in the NYT snippet of the Introduction and Contributions section tend to discuss
topics such as fitness and sedentary lifestyles, as opposed to food content). We observe
that the PMC for this domain (2005 to 2007) is characterized by increased positivity,
shown by classes 4 and 5, and decreased negativity (class 1). Our results for this
domain thus agree with the findings of Kim and Willis [2].
We were unable to use this domain in our precision-recall analysis, since Kim and
Willis, to the best of our knowledge, do not specify a precise period during which the
framing change took place.
However, since Figures 2 and 3 of Kim and Willis [2] show a dramatic increase of
social causes in 2004, and a corresponding marked decline of individual causes, we
conclude a substantial agreement between their findings and our results.
LGBT Rights
We compiled a dataset of over 3,000 articles from the period 1996 to 2015 in this domain.
Figure 6 depicts our estimated clusters. Cluster 3 represents a frame that discusses the
subject of same-sex marriage and its legality. We note that the Supreme Court ruled to
legalize same-sex marriages in the US in the year 2015. Our class vectors for this domain
are shown in figure 14. We obtained two PMCs with nearly identical correlation scores
(0.999 for the period 2006 to 2008, and 0.989 for the period 2013 to 2015). Figure 14
highlights the period 2013 to 2015 immediately preceding the judicial interest of 2015.
We were unable to identify a prior study that discusses the framing of LGBT news
over our entire period of interest. However, we use the findings reported in Gainous et
al. [32] as our ground truth for this domain. Gainous et al. studied the framing of
LGBT related publishing in the New York Times over the period 1988 to 2012, and
found a dramatic increase in equality frames between approximately 25 in 2008 and
0.20
0.15
0.14
0.12 Class 2
0.1
0.14
0.12 Class 3
0.1
0.24
0.22
Class 4
0.2
0.18
Class 5
0.35
0.3
2010 2011 2012 2013 2014 2015 2016
Fig 12. Annual polarities for a representative cluster (characterized by the terms
‘national‘, ‘security’, and ‘agency’) from the domain surveillance for the classes 1 to 5.
The PMC is shown with solid lines in square markers.
Class 1
0.25
0.2
0.120 Class 2
0.100
0.15
0.14
Class 3
0.13
0.12
0.24
Class 4
0.22
0.2
0.36 Class 5
0.34
0.32
0.3
0.28
1990 1993 1996 1999 2002 2005 2008
Fig 13. Annual polarities for cluster 2 (characterized by the terms ‘diet’, ‘food’, and
‘make’) from Figure 7 from the domain obesity for the classes 1 to 5. The PMC is shown
with solid lines in square markers. We posit that this cluster represents societal causes
of obesity (see the Obesity section). We observe that the PMC for this cluster (2005 to
2007) agrees with the findings of Kim and Willis [2].
Abortion
The Partial-Birth Abortion Ban Act was enacted in 2003. We obtained 248 articles for
the period 2000 to 2003, for this domain. We obtain a PMC of 2001 to 2003 for this
domain, as shown in figure 15.
Immigration
We study the framing of immigration news in the United Kingdom. We obtained about
3,600 articles on the subject of Immigration from the Guardian API for the period 2000
to 2017. For this domain, we carried out our analysis on the article titles (rather than
the full text). Since the Guardian returns full length articles, we found that this design
choice allows us to produce a more focused domain corpus than the one generated by
the full article text. We depict our estimated class vectors and PMC in figure 16.
We analyze the frame of cluster 2 in Figure 10. This cluster deals with the issue of
asylum seekers to the United Kingdom. In the period beginning immediately before the
year 2000,a new peak in asylum claims to the United Kingdom of 76,040 had been
reached [33]. This event coincided with a high-profile terrorist act by a set of Afghan
asylum seekers [33].
These events resulted in increased border refusals and the final 2002 white paper on
“Secure Borders, Safe Haven.” We estimate a PMC of 2000 to 2002 (Figure 16). Our
PMC coincides exactly with the period immediately foreshadowing the government
white paper.
Drones
We obtained nearly 4,000 articles on this domain for the period 2003 to 2012. We
obtain a PMC of 2009 to 2011 for this domain, as shown in Figure 17.
Our PMC immediately foreshadows the Federal Aviation Administration’s
Modernization and Reform Act of 2012.
Predictive Utility
The aforementioned two domains (immigration and drones) highlight the predictive
utility of news framing. Whereas we did not find earlier surveys that coincide with our
PMCs for these domains, we note that these PMCs foreshadowed substantial legislative
0.180
0.160
0.120
Class 2
0.115
0.110
0.105
0.150 Class 3
0.140
0.130
0.250
0.240 Class 4
0.230
0.220
0.350
Class 5
0.340
0.330
1996 1998 2000 2002 2004 2006 2008 2010 2012 2014
Fig 14. Annual polarities for cluster 3, characterized by the terms ‘gay’, ‘rights’, and
‘marriage’, in Figure 6 from the domain LGBT Rights for the classes 1 to 5. We obtain
two PMCs with nearly identical correlation scores, namely, 2006 to 2008 and 2013 to
2015. The PMC of 2013 to 2015 is shown with solid lines in square markers,
immediately preceding the judicial interest of 2015.
0.26
0.24
0.12
0.11
Class 2
0.10
0.09
0.14
Class 3
0.13
0.12
Class 4
0.22
0.20
0.18
0.30
Class 5
0.28
0.26
0.400
0.300
0.200 Class 1
0.15
0.10
0.05 Class 2
0.15
0.10
Class 3
0.25
0.20
0.15 Class 4
0.50
0.40
0.30
0.20 Class 5
0.160
0.140
0.100
Class 2
0.090
Class 3
0.140
0.130
Class 4
0.260
0.240
0.400
0.380
Class 5
0.360
0.340
Conclusion
We highlight a problem of significant public and legislative importance, framing change
detection. We contribute an unsupervised natural language based approach that detects
framing change trends over several years in domain news publishing. We identify a key
characteristic of such changes, namely, that during frame changes, the polarity of
adjectives describing cooccurring nouns changes cumulatively over multiple years. Our
approach agrees with and extends the results of earlier manual surveys. Whereas such
surveys depend on human effort and are therefore limited in scope, our approach is fully
automated and can simultaneously run over all news domains. We contribute the
Framing Changes Dataset, a collection of over 12,000 news articles from seven domains
in which framing has been shown to change by earlier surveys. We will release the
dataset with our paper. Our work suggests the predictive utility of automated news
monitoring, as a means to foreshadow events of commercial and legislative import.
Our work represents one of the first attempts at a computational modeling of
framing and framing changes. We therefore claim that our approach produces promising
results, and that it will serve as a baseline for more sophisticated analysis over wider
temporal and geographical data.
Ethics Statement
Our study involved no human or animal subjects.
Funding Statement
CS has a commercial affiliation to Amazon. The funder provided support in the form of
salaries for this author, but did not have any additional role in the study design, data
collection and analysis, decision to publish, or preparation of the manuscript. The
specific roles of these authors are articulated in the ‘author contributions’ section.
Author Contributions
KS and CS conceived the research and designed the method. KS prepared the datasets
and performed the analysis. KS and MPS designed the evaluation approach. KS, CS,
and MPS wrote the paper.
References
1. Gunnars K. Ten Causes of Weight Gain in America; 2015.
https://www.healthline.com/nutrition/10-causes-of-weight-gain#section12.
2. Kim SH, Willis A. Talking about Obesity: News Framing of Who Is Responsible
for Causing and Fixing the Problem. Journal of Health Communication.
2007;12(4):359–376.
3. Flegal K, Carroll M, Kit B, Ogden C. Prevalence of Obesity and Trends in the
Distribution of Body Mass Index Among US Adults, 1999–2010. Journal of the
American Medical Association. 2012;307(5):491–497.
Responses:
Comment on Motivation: “The research lacks motivation as it is not clear what benefits
can be achieved if frame changes are detected.”
Framing changes have been shown to have commercial and legislative consequences,
and have also been shown to foreshadow public attention changes. We cite five
example articles here [1-5] and can readily provide more as necessary. A large body of
literature in the fields of Political Science and Communication addresses the manual
identification of framing changes in specific domains. Whereas we cite two examples
here [6-7], additional examples are available – please let us know. However, existing
work does not attempt to address the problem of computationally detecting framing
changes. Our work is the first attempt at this problem, which has significant commercial,
public, and legislative import. Our results substantially agree with the results of earlier
human surveys, and further have shown predictive utility for legislative and public
response. Our work therefore has significant scientific and potential commercial value.
Comment on Contribution: “Moreover, the problem is already discussed and presented
in articles [4,5]. This paper seems to provide more empirical evidence in support to the
existing research [4,5]. Hence, the research contribution is unclear.”
We emphasize that our work is the first attempt at computationally modeling changes in
framing. The closest previous efforts in this area are those of [10] and [11]. We describe
our novel contributions over these efforts in detail in the Related Work section. We are
unaware of any other relevant related work and would be happy to learn of any such
work from the Editor.
This statement is incorrect. We have clearly stated in our submission that all data and
code will be made available, and are available online at the following link:
https://drive.google.com/open?id=1zAH__Y1lcdriuwUcjZsKmvaqYtzAjyZ9
All our results are reproducible from the data and code in the above mentioned
repository. We will provide a guide to run our code.
3) “The comparison of research with state of the art approaches and manual techniques
has not been conducted.”
Please refer to our responses above to the comment on motivation and comment #1.
4) “An overview diagram for the proposed approach would help the reader understand the
flow of the proposed approach.”
We are grateful for this suggestion and will incorporate an overview diagram illustrating
our approach. However, this is a simple suggestion for presentation that may easily be
addressed in a revision.
5) “The results are presented but not discussed. The section should be renamed to "Results
and Discussion" and appropriate discussion should be added with each pair of graphs.”
The Results section discusses our results for each domain, using both a qualitative
comparison with manual surveys (by other authors) and by highlighting the predictive utility of
the returned result. We show that our results both agree with previous manual surveys, and are
also able to predict significant public and legislative response in each domain. We will rename
this section to “Results and Discussion”.
References:
1. A. C. Gunther, The persuasive press inference: Effects of mass media on perceived
public opinion. Commun. Res. 25, 486–504 (1998).
3.G. King, B. Schneer, A. White, How the news media activate public expression and
influence national agendas. Science 358, 776–780 (2017).
7 S. M. Engel, Frame spillover: Media framing and public opinion of a multifaceted LGBT
rights agenda. Law Soc. Inq. 38, 403–441 (2013).
Response to Reviewers
We thank the editors for their valuable feedback. We address the main points below.
Editor Comment:
My main concern with this paper deals with the evaluation of the approach. More precisely, the
experimental section illustrates a series of case studies or scenarios where the frame change is
identified through a sudden polarity drift (Figures 10.16) that is shown to correlate with some
well-known fact or event studied in the literature. The point is: how much is this evaluation
anecdotal, and to what extent can it be quantitatively measured? In all Figures from 10 through
16, several peaks and sudden changes can be observed in the polarity distribution (e.g., Figure
12, class 5, years 2000 through 2003, or Figure 13, class 1, year 2006, to mention just a few):
do they all correspond to frame changes? If not, how can they be detected/studied? The paper
states that the dataset was annotated by experts: how? Can such annotation be used for
quantitative evaluation of the approach?
Response:
This comment concerns two primary aspects of the paper: (i) defining a framing change, and in
particular, isolating framing changes, and filtering out polarity drifts that do not correspond to
framing changes (ii) quantitative evaluation of the approach. We address each aspect below.
We have added a section entitled Defining Framing Changes. We summarize the main points
here.
Since language and human behavior are not strictly deterministic, the measurement of any
temporally disparate pair of news corpora using adjective polarity (or any other numerical
metric) would result in different representative values of the two corpora. Therefore, in this
sense, any pair of news corpora can be said to have undergone a framing change.
Further, individual metrics are susceptible to noisy readings due to imprecise data and
measurement. In particular, such an effect may cause sudden isolated spikes between
successive measurements.
This motivates the question of how a framing change is defined, in the context of our
computational measurements. The usual social science definition is that a framing change is a
shift in the way that a specific topic is presented to an audience. To isolate such changes
computationally, we use the following key observations from ground truth framing changes: (i)
framing changes take place as trends that are consistent over at least $k$ years (ii) framing
changes must be consistent across multiple measurements.
Our aim in this paper is to begin from a set of time series such as the ones in figures 10 to 16,
and isolate such trends. The requirement motivated by our first condition, namely, that framing
changes must last at least k years, is easy to satisfy by imposing such a numerical threshold.
Our approach thus identifies polarity drifts that are both correlated (quantitatively measured by
correlations between different measures of polarity) and sustained (by the imposition of a
threshold of duration). We point out that our approach filters out isolated drifts in individual
polarity measures, since such drifts are uncorrelated across multiple measures. Further, we
note that the magnitude of individual drifts matters only indirectly to our approach, to the extent
that a larger drift, if consistent across multiple polarity measures, may have higher correlation
than a smaller drift that is also correlated.
In order to do so, we study the literature pertaining to framing changes in the domains we
examine. We identify large-scale studies conducted by reputed organizations such as the
National Cancer Institute, the Columbia Journalism Review, Pew Research, and so on. These
studies examine news and media publishing in a particular domain over a period of time, as we
do, and manually identify changes in the framing of domain news during these periods.
The studies we rely on for ground truth sometimes provide quantitative justification for their
findings. These studies therefore provide an expert annotation of framing changes in our
domains, for the periods we examine..
By demonstrating substantial agreement between the results of our approach and those of
earlier ground truth surveys, we establish our claim that our approach may be used to
automatically identify framing changes in domain news publishing.
Given that the data sources and coverage between our analysis and that of prior surveys are
usually quite different, the correlations we obtain appear quite substantial. However, quantitative
evaluation remains challenging for the reasons we point out.
This paper follows the spirit of recent work in seeking to develop the study of framing into a
computational science. We acknowledge that our methods may undergo refinement to tackle
broader ground truth data, of a wider temporal and geographical scope. Nonetheless, we posit
that our methods and results have scientific value, and hope that future work will provide greater
coverage of ground truth.
Please note that the underlying data preparation requires social science expertise and cannot
be effectively crowdsourced via a platform such as Mechanical Turk. We therefore hope that
our approach piques the interest of social scientists and leads them to pursue more
comprehensive studies of framing in news media that would enable improvements in
computational methods.
Editor Comment:
* What do the annotators tag in the dataset? The paper just state that two raters code a random
sample of articles from each domain, reporting Cohen's kappa. But what do they code? Frames,
or frame changes? If so, how is a frame change defined?
Response:
To ensure that the articles returned by our term search procedure are indeed relevant to each
domain, a random sample of articles from each domain dataset was coded for relevance by two
raters.
Please refer to the section on Defining Framing Changes for a discussion on how we treat the
problem of identifying changes in framing.
Editor Comment:
* With regards to Figures 1 and 2, the authors state that the peak in the LGBT domain
immediately precedes a frame change. But, does this hold also for other peaks of other
domains? Such as, drones in 2004, obesity in 2005, or smoking in 2005?
Response:
Whereas we do not claim that this correlation is true for all domains, we posit that it motivates
the utility of adjective polarity in the study of framing changes.
Editor Comment:
* What are the differences of the proposed approach with respect to an approach that detects
just frames (instead of frame changes), but then look at changes in the detected frames...? See,
e.g., the following references:
** Alashri et al., "Climate Change" Frames Detection and Categorization Based on Generalized
Concepts", International Journal of Semantic Computing, 2016
** Tsur et al., "A Frame of Mind: Using Statistical Models for Detection of Framing and Agenda
Setting Campaigns", ACL 2015
Response:
We have added paragraphs to the Related Work section, detailing the novel contributions of our
work and drawing distinctions between this paper and earlier approaches. We summarize this
discussion here.
We note that our approach is similar in spirit to Tsur et al’s [9] work, in that both that work and
this paper apply a topic modeling strategy to analyze framing as a time series. However, we
highlight the following key differences and contributions of our work. Firstly, as both Sheshadri
and Singh [8] and Tsur et al point out, framing is a subjective aspect of communication.
Therefore, a computational analysis of framing should ideally differentiate subjective aspects
from fact-based and objective components of communication. Since adjectives in and of
themselves are incapable of communicating factual information, we take them to be artifacts of
how an event or topic is framed. In contrast, generic n-grams (as used by Tsur et al) do not
provide this distinction.
Further, Tsur et al rely upon estimating ``changes in framing'' using changes in the relative
frequencies of n-grams associated with various topics or frames. Whereas such an approach is
useful in evaluating which of a set of frames may be dominant at any given time, it does not
measure ``framing changes'' in the sense originally described in [5]. In contrast, our work
estimates changes in framing using consistent polarity drifts of adjectives associated with
individual frames. Our approach may also be applied to each of a number of frames
independently of the others, as opposed to Tsur et al.
Editor Comment:
I have consulted with another academic editor, Dr. Marco Lippi, and we agree that a desk
rejection was premature. Nevertheless, experience tells us that all reviewers provide a
perspective similar to at least some other readers. For this reason I am requesting that you
revise the manuscript to address issues raised in the desk rejection. Perhaps you made some
revisions in your appeal. However, these were not apparent to me with track changes. Please
revise the manuscript itself, including a rebuttal that identifies the specific location of the
modifications you made in response to the original decision.
Response:
We have submitted a revised manuscript in which we highlight our responses to each point
made in the original decision. We summarize our responses to each point in the original
decision below.
Editor Comment:
Response:
Our updated related work section discusses alternative approaches in detail, and describes our
novel contributions over these approaches.
Editor Comment:
Response:
The dataset and code are available online at the following link:
https://drive.google.com/open?id=1zAH__Y1lcdriuwUcjZsKmvaqYtzAjyZ9
All our results are reproducible from the data and code in the above mentioned
repository. We will provide a guide to run our code.
Editor Comment:
4. An overview diagram for the proposed approach would help the reader understand the flow of
the proposed approach.
Response:
Editor Comment:
5. The results are presented but not discussed. The section should be renamed to "Results and
Discussion" and appropriate discussion should be added with each pair of graphs.
Response:
We have expanded our analysis of the results for each domain, including adding a quantitative
precision-recall analysis based on ground truth data.
In addition to these responses, we have appended a copy of our original response to reviews
below.
Editor Comments:
This research is focused on detecting framing changes in topical news. The authors argue that
the public opinion varies with the way the news is framed. The research lacks motivation as it is
not clear what benefits can be achieved if frame changes are detected. Moreover, the problem
is already discussed and presented in articles [4,5]. This paper seems to provide more empirical
evidence in support to the existing research [4,5]. Hence, the research contribution is unclear.
Furthermore, following points are worth considering:-
1. The related work should be discussed in detail highlighting the advantages/limitations of
existing approaches.
2. The dataset and codes are not available online.
3. The comparison of research with state of the art approaches and manual techniques has not
been conducted.
4. An overview diagram for the proposed approach would help the reader understand the flow of
the proposed approach.
5. The results are presented but not discussed. The section should be renamed to "Results and
Discussion" and appropriate discussion should be added with each pair of graphs.
Responses:
Framing changes have been shown to have commercial and legislative consequences,
and have also been shown to foreshadow public attention changes. We cite five
example articles here [1-5] and can readily provide more as necessary. A large body of
literature in the fields of Political Science and Communication addresses the manual
identification of framing changes in specific domains. Whereas we cite two examples
here [6-7], additional examples are available – please let us know. However, existing
work does not attempt to address the problem of computationally detecting framing
changes. Our work is the first attempt at this problem, which has significant commercial,
public, and legislative import. Our results substantially agree with the results of earlier
human surveys, and further have shown predictive utility for legislative and public
response. Our work therefore has significant scientific and potential commercial value.
Revised sections:
We emphasize that our work is the first attempt at computationally modeling changes in
framing. The closest previous efforts in this area are those of [10] and [11]. We describe
our novel contributions over these efforts in detail in the Related Work section. We are
unaware of any other relevant related work and would be happy to learn of any such
work from the Editor.
2) “The dataset and codes are not available online.”
This statement is incorrect. We have clearly stated in our submission that all data and
code will be made available, and are available online at the following link:
https://drive.google.com/open?id=1zAH__Y1lcdriuwUcjZsKmvaqYtzAjyZ9
All our results are reproducible from the data and code in the above mentioned
repository. We will provide a guide to run our code.
3) “The comparison of research with state of the art approaches and manual techniques
has not been conducted.”
Please refer to our responses above to the comment on motivation and comment #1.
4) “An overview diagram for the proposed approach would help the reader understand the
flow of the proposed approach.”
We are grateful for this suggestion and will incorporate an overview diagram illustrating
our approach. However, this is a simple suggestion for presentation that may easily be
addressed in a revision.
5) “The results are presented but not discussed. The section should be renamed to "Results
and Discussion" and appropriate discussion should be added with each pair of graphs.”
The Results section discusses our results for each domain, using both a qualitative
comparison with manual surveys (by other authors) and by highlighting the predictive utility of
the returned result. We show that our results both agree with previous manual surveys, and are
also able to predict significant public and legislative response in each domain. We will rename
this section to “Results and Discussion”.
References:
1. A. C. Gunther, The persuasive press inference: Effects of mass media on perceived
public opinion. Commun. Res. 25, 486–504 (1998).
3.G. King, B. Schneer, A. White, How the news media activate public expression and
influence national agendas. Science 358, 776–780 (2017).
7 S. M. Engel, Frame spillover: Media framing and public opinion of a multifaceted LGBT
rights agenda. Law Soc. Inq. 38, 403–441 (2013).
9 O. Tsur, D. Calacci, and D. Lazer, A Frame of Mind: Using Statistical Models for
Detection of Framing and Agenda Setting Campaigns. Proceedings of the 53rd Annual
Meeting of the Association for Computational Linguistics and the 7th International Joint
Conference on Natural Language Processing (Volume 1: Long Papers).