IEEE TRANSACTIONS ON SMART GRID, VOL. 7, NO.
5, SEPTEMBER 2016
Energy Big Data Analytics and Security:
Challenges and Opportunities
Jiankun Hu, Member, IEEE, and Athanasios V. Vasilakos, Senior Member, IEEE
AbstractThe limited available fossil fuels and the call for sus-
tainable environment have brought about new technologies for
the high efficiency in the use of fossil fuels and introduction of
renewable energy. Smart grid is an emerging technology that can
fulfill such demands by incorporating advanced information and
communications technology (ICT). The pervasive deployment of
the advanced ICT, especially the smart metering, will generate
big energy data in terms of volume, velocity, and variety. The
generated big data can bring huge benefits to the better energy
planning, efficient energy generation, and distribution. As such
data involve end users privacy and secure operation of the criti-
cal infrastructure, there will be new security issues. This paper is
to survey and discuss new findings and developments in the exist-
ing big energy data analytics and security. Several taxonomies
have been proposed to express the intriguing relationships of
various variables in the field.
Index TermsEnergy, big data, analytics, cyber security, smart
grid, anomaly detection, SCOPF.
I. I NTRODUCTION
OSSIL fuel reserves are finite and it is predicted that
F the known oil reserves will be exhausted by 2050 [1].
Renewable energy such as wind energy and solar energy can
provide a solution to both energy shortage and sustainable
environment. Smart and efficient energy usage will also be
effective in energy saving and reducing carbon emissions.
The emerging smart grid (SG) technology provides an effec-
tive means in incorporating various renewable energy sources
into the existing energy system and also making smart energy
a reality. It is considered as a technological paradigm shift.
In a smart grid, networking, intelligent communications
technology and information processing functions are immersed
into every facet of the energy system ranging from power
generation, power transmission, power distribution and con-
sumer appliances [2][4]. A large-scale smart grid can consist
of thousands of microgrids that are operating in both intercon-
nected and isolated modes [2]. Smart metering is an integral
component of a smart grid where smart meters are being
installed in homes and other parts of the system in a large
scale. Reports indicate that the global smart meter installation
Manuscript received August 29, 2015; revised December 11, 2015,
March 9, 2016, and April 11, 2016; accepted April 24, 2016. Date
of publication May 9, 2016; date of current version August 19, 2016.
Paper no. TSG-01019-2015.
J. Hu is with the University of New South Wales, Canberra, ACT 2610,
Australia (e-mail: J.hu@adfa.edu.au).
A. V. Vasilakos is with the Department of Computer, Electrical and Space
Engineering, Luea University of Technology, Lulea 97187, Sweden (e-mail:
vasilako@ath.forthnet.gr).
Digital Object Identifier 10.1109/TSG.2016.2563461
1949-3053 
c 2016 EU
2423
numbers will triple from 10.3 million in 2011 to 29.9 million
by 2017 [4]. There are many potential advantages to be derived
from smart grid data [4]: automated and real-time moni-
toring users energy consumption, automated processing of
billing, detection of energy losses (possible fault and/or fraud),
early warning of blackouts, fast detection of disturbances in
energy supply, intelligent and real-time energy planning and
pricing. Smart energy data exhibit characteristics of 3Vs
model describing big data, which are Volume, Velocity and
Variety.
A. Big Data Characteristics of Smart Energy Data
1) Volume and Velocity: Smart meters are usually deployed
at the scale of multimillion units. The E-Sketch project showed
that hourly power consumption data will miss much vital infor-
mation on how homes consume energy, which is needed for
real-time pricing plans. Also overlapping of peak demand from
individual homes may cause blackouts at some substations
within several seconds [5]. Therefore data generated at the
minute-level and even second-level will be desirable [5][7].
According to 2012 housing units in New York State,
127.1TB is needed to store each days power consumption
data [5].
2) Variety: There are many structured and unstructured
data from multiple sources and categories that are relevant to
the energy generation, energy planning, and distribution etc. In
addition to the household appliance-level energy consumption
data generated by the Advanced Metering Infrastructure (AMI)
and smart meters, other data such as different renewable
energy, weather, and market etc. are collected for the optimal
operation of the system.
B. Motivation
The big data nature of smart energy poses new challenges
in data analytics and security where conventional technolo-
gies cannot deal with. Recently much research effort has been
made to address the challenges of energy big data analyt-
ics and security. Energy big data analytics and security is
a very broad area involving large distributed infrastructure, big
data generation, transmission, storage, sharing and processing,
and security and privacy. In addition to the common challenges
of big data analytics and security, energy big data analyt-
ics will add another dimension of difficulty in dealing with
the unique factor of tight cyber-physical couplings. A sys-
2424
Fig. 1. Taxonomy of energy big data challenges.
tematical and comprehensive survey on the new findings will
help track the latest developments in the field, which will be
the main motivation of this paper. Although there exist sev-
eral surveys on energy big data analytics, no survey has been
found that has incorporated the security and privacy issues
of energy big data analytics. Our survey will attempt to fill
this gap.
C. Main Contributions
In this paper, we will present a survey reflecting the latest
developments in the field. Our contributions are manifold:
It provides a first comprehensive survey covering both
energy big data analytics and its security in an integrated
framework.
It proposes several well-designed taxonomies for energy
big data analytics and security, which can help under-
stand intriguing relationships among many variables and
concepts in the field.
We have discussed several important lessons learned from
previous research activities. For example, power sys-
tem community has a different security concept from
cyber security community, while conventional cryp-
tography in the cyber security community does not
consider the factors of real-time and the tight cyber-
physical couplings in the advanced power systems
such as smart grids. Such parallel and un-converged
security research activities will leave severe security
loopholes in the gap. We also suggest the adop-
tion of expanded dependability framework to close
the gap.
We have provided and discussed open research questions
for future research directions.
The rest of this paper is organized as follows: In Section II,
an overview is given on the challenges of energy big data ana-
lytics and security/privacy. In Section III, an overall taxonomy
of energy big data analytics is presented. Various algorithms
and schemes on energy big data analytics are classified under
this proposed taxonomy. In Section IV, we discuss energy
big data infrastructure security from cyber-physical aspect. In
Section V, latest developments on energy big data security and
privacy are surveyed and summarized from data-driven aspect.
Section VI will discuss data-driven schemes for resilient smart
grid operations. Open research questions for future research
directions are discussed in Section VII. Section VIII is devoted
to the conclusions.
IEEE TRANSACTIONS ON SMART GRID, VOL. 7, NO. 5, SEPTEMBER 2016
II. C HALLENGES OF E NERGY B IG DATA
A NALYTICS AND S ECURITY
In this section, an overview is presented on the challenges of
energy big data analytics and security/privacy. A taxonomy of
challenges of energy big data analytics and security is shown
in Fig. 1.
A. Scalable and Interoperable Computing Infrastructure
A SG is a highly distributed system. The huge amount of
data is collected from every corner including energy genera-
tion, distribution, renewal energy powered vehicles and smart
meters etc. It includes dynamic streaming and non-streaming
data, structure and un-structured data. Also there is a constant
flow of the data, e.g., machine to machine, machine to human
etc. It is very challenging to store, share and process such
data. A scalable and interoperable computing infrastructure
is needed.
B. Real-Time Big Data Intelligence
A report indicates that overlapping of peak demand from
individual homes may cause blackouts at some substa-
tions within several seconds [5]. Therefore real-time deci-
sion is essential for both system operation and real-time
pricing. An intelligent decision making needs to process
current data and historical data. Given the huge volume
and high variety of the data, it is challenging enough
to process such data. With the constraint of real-time
demand, it will be extremely challenging to design new
algorithms that can provide real-time intelligence from such
big data.
C. Big Data Knowledge Representation and Processing
Big data analytics requires new machine learning the-
ory and artificial intelligence. It is well known that that
the process and outputs from machine learning and artifi-
cial intelligence lack of intuitive physical interpretation [8].
Therefore it is important to fill this gap by provid-
ing suitable knowledge interpretation in order to make
a sound decision based on the intelligence derived from
machine learning and artificial intelligence. This task is
challenging due to the big data nature of the smart
energy data.
HU AND VASILAKOS: ENERGY BIG DATA ANALYTICS AND SECURITY: CHALLENGES AND OPPORTUNITIES 2425

TABLE I
S UMMARY OF S MART G RID DATA

D. Big Data Security and Privacy III. E NERGY B IG DATA A NALYTICS :

Smart energy data contain individual users private infor-
mation which is required to be protected under vari-
ous legal regulations [9], [10]. The data can also contain
sensitive information of an organization. More impor-
tantly such data can be used to make decisions affect-
ing the safe operation of the critical infrastructure.
Therefore security and privacy will be an important
issue. However, this is also very challenging due to the
big data nature of the smart energy data, tight cyber-
physical couplings, distributed and open environment of the
infrastructure.
OVERALL TAXONOMY
A. Overview of Energy Big Data Analytics
Load modeling and forecasting are two major energy big
data intelligence applications. Load modeling is essential to the
understanding of the behavior of the individual and system in
achieving efficient energy management. Load forecasting has
generic load forecasting (long term and medium term) and
short term forecasting. The former is useful for system capac-
ity planning and planning etc. while the latter is useful for
many aspects including power distribution, demand-response
and pricing [11], [12].
2426
Fig. 2. Overall taxonomy of energy big data analytics.
Due to its real-time nature, short-term forecasting is most
challenging. Several models have been developed for energy
load forecasting over big data [12][22]. A cost-effective
method is the penalized linear regression-based MapReduce
algorithms [22], where the MapReduce algorithm can reduce
the training dataset significantly for the penalized linear
regression. The framework of the similar day approach,
being used in conventional short-term forecasting, seems to
be useful for the short-term forecasting over energy big
data [17], [18]. Energy big data analytics is a very broad area,
covering big data generation infrastructure, big data comput-
ing platform/architecture, big data intelligence applications,
big data intelligence algorithms and tools, and big data secu-
rity and privacy. To deal with such complicated environment,
a well-structured analysis framework is desirable. In the next
section, taxonomy will be designed in order to provide such
a structured analysis framework.
B. Overall Taxonomy
In order to understand the issues related to energy big
data analytics, it will be helpful to know what types of
data a SG can encounter. There are many different data related
to energy data analytics and it is impossible to provide
a complete list here. Based on the power system application
data dictionary [23], we provide a summary of various smart
grid relevant energy data as shown in Table I. Basically the
classification is based on the data generation source.
IEEE TRANSACTIONS ON SMART GRID, VOL. 7, NO. 5, SEPTEMBER 2016
In order to provide a systematical analysis of the tech-
niques related to energy big data analytics, we need to provide
a structure or classification to these vast amounts of tech-
niques. Taxonomy is an effective tool in providing a structure
for the distributed and complicated environment. Designing
a good taxonomy is non-trivial as it needs to meet the seven
principles of taxonomy design [2]. By following the seven tax-
onomy design principles [2], a taxonomy is designed for the
overall energy big data analytics as shown in Fig. 2.
In Fig. 2, energy big data analytics are categorized into three
distinct categories: (i) energy big data architecture/platform,
(ii) energy big data intelligence, and (iii) energy big data secu-
rity and privacy. Different from a common perception that
energy big analytics is on energy big data intelligence, our
proposed taxonomy has placed the above three mentioned
components under an integrated framework.
This is because an optimal energy big data intelligence
scheme is tightly coupled with how energy big data is stored,
accessed, and communicated. Also as energy data involve
human privacy and system security, energy big data intel-
ligence schemes have to address the issue of privacy and
security. For example, the best platform of energy big
data intelligence is cloud computing where data are encrypted.
Data intelligence over these encrypted data has to tailor its
design to complying with the underlying privacy and security
protection mechanisms.
This overall taxonomy shows the intriguing relations among
various components under energy big data analytics. For
HU AND VASILAKOS: ENERGY BIG DATA ANALYTICS AND SECURITY: CHALLENGES AND OPPORTUNITIES
example, components of Control System Aspect, Cyber
Security Aspect, and energy big data oriented anomaly detec-
tion will be tightly coupled with Data-driven Resilient SG
Management. Such knowledge will be helpful in the under-
standing of the issues in the energy big data security and
privacy, which will be the focus in the remaining sections.
IV. E NERGY B IG DATA I NFRASTRUCTURE S YSTEM
S ECURITY: C YBER P HYSICAL A SPECT
Smart energy data contain commercial information of com-
panies, and involve end users privacy. A SG, as an open
critical infrastructure, is vulnerable to cyber-attacks which
could lead to catastrophic disasters. In the energy sector, the
concept of system security differs from the one in the cyber
security community. Although energy system security and
cyber security are both integral parts of the energy infrastruc-
ture security, little research effort has been made to consider
them in a coherent framework. In this section, we will attempt
to discuss energy infrastructure security methods in a coher-
ence framework and reveal lessons we have learned from the
prior ignorance of such coherence.
In the energy sector, security is defined as the ability of
a power system to withstand sudden disturbances [24], [25].
Traditionally such security is concerned with designing a sys-
tem that can provide acceptable service under sudden distur-
bances from equipment faults and/or nature disasters. Such
design relies heavily on the technique called contingency
analysis [25]. The contingency analysis refers to a system-
atical system security assessment, which is often dependent
on performing evaluation of the state estimator against possi-
ble occurrence of credible contingencies to determine whether
steady-state operating limits would be violated. Traditionally
the possible occurrence of credible contingencies is most
likely from equipment faults and nature disasters. Therefore
most relevant theories and models are based on the parts
random fault and nature disaster occurrence probability distri-
bution. Unfortunately such energy systems are most unlikely
to withstand cyber-attacks. Instead of following the nature
random equipment fault/failure probability distribution, cyber-
attacks will usually target most critical components of the
energy system and conduct coordinated attacks, which will
generate concatenated failures leading to far more severe dam-
ages than what can be coped with by the traditional security
system design. Security incorporating tight cyber-physical cou-
pling has become a new exciting research paradigm. With the
emerging smart grid, it will become a pressing research issue.
Recently some developments have been made to extend
power system security into covering cyber-attacks as
disturbances [26][29]. These works focus on two important
aspects of smart grid security: (i) Security framework covering
cyber aspect, and (ii) Cyber-physical coupling modeling.
Security framework covering cyber aspect: Traditional
power system security design is based on the properties
of pure power system physical components such as relay
devices, and switches etc. The random fault/failure proper-
ties have played a pivotal role in system security assessment
and resilient mechanism design. For the emerging smart grid
2427
systems, it is essential to develop a new security framework
that can cover other factors due to the introduction of the
ICT technology. In [26], a distributed agent based security
framework is proposed to deal with possible system dam-
age caused by the cyber-attacks. The proposed framework
utilized peer-to-peer communication architecture, reputation-
based trust management scheme and a data retransmission
scheme to detect possible cyber-attacks. Although such frame-
works have incorporated ICT components into the security,
they are mostly based on the high-level abstraction of the cyber
impact, e., reputation, in detecting possible cyber-attacks.
Such coarse generalization tends to generate too conservative
results, leading to inefficiency. A more accurate description of
cyber-physical interaction will help produce optimal results.
Cyber-physical coupling modeling: The most unconven-
tional security feature in smart grids is the tight cyber-physical
coupling. We have following two interesting observations:
Real-time factor: Traditional power system security
involves real-time factor while traditional cryptography
does not involve this factor.
Cyber-physical interaction: Both traditional power sys-
tem security and cryptography do not consider such
interactions. Yet, in reality, cyber activities can impact
physical processes significantly and the characteristics
of a physical process can also be used for detecting
cyber-attacks [2].
Therefore, a good understanding of the tight cyber-physical
coupling will be essential for the smart grid infrastructure
security. Some efforts have been made on this aspect. In [28],
cause-effect relationships of cyber-physical system have been
used to estimate the impact of cyber-attacks. A graph the-
ory approach is used to model such relationships. Although
cause-effect relationships of cyber-physical systems are very
important for the understanding the cyber-physical coupling
characteristics, such relationships are from qualitative aspect
which is not sufficient. This is because the physical opera-
tion process of a power system depends also on the real-time
quantitative relationships of the cyber-physical coupling.
In [27], a security-oriented stochastic risk index CPINDEX
is proposed to measure the security level of the underlying
cyber-physical settings in the smart grid. This is the first com-
prehensive attempt in describing the quantitative relationships
of the cyber-physical coupling and their impact to a smart
grid system security. The proposed CPINDEX utilizes the
information of system logs and other information to estimate
the security index by using belief propagation algorithms. An
interesting security measure named cyber-physical index is
proposed as follows:
  
CP(s) = CIPC () Pr root(CT ) = 0s , (1)
A(s)
where s is system security state, represents a critical asset,
CIPC () is the physical contingency ranking centrality index,
and CT represents Boolean expressions consequence trees
related to low level detectable incidents. A(s) refers to all the
assets in the control of a physical equipment and the probabil-
ity that the asset is affected due to the attack at the current
network security state is represented by Pr(rootCT ) = 0|s).
2428
This index can effectively measure the severity of the dam-
age/risk when a specific physical incident happens due to
cyber-attacks. We can have following observations:
System security state is not a well-defined concept, which
is hard to describe quantitatively. Security risk could be
a better mechanism.
Mathematically eq.(1) represents the average or expecta-
tion of the impact to the assets being controlled by the
same physical equipment. In practical applications, mea-
surement of the maximum impact to the system will be
more concerned.
In summary, we can have following observations:
Most of system security works are within the control
engineering community where the major research focus is
on physical process and many conventional cyber security
issues, e.g., confidentiality etc., are seldom considered.
The system security definition is very similar to the con-
cept of reliability in the domain of cyber security: Under
the impact of cyber-attacks on command signals, and sys-
tem state etc., the system can maintain the service quality
within a tolerable error range [30]. The dependability
framework can integrate these two security definitions
seamlessly together [30].
V. E NERGY B IG DATA S ECURITY AND P RIVACY:
DATA -D RIVEN A SPECT
In the domain of cyber security, security is defined and
characterized as follows [30]:
Reliability: Even under the disturbances of faults and
cyber-attacks, the system correct service can be main-
tained within a certain level.
Confidentiality: Property that data or information is not
made available to unauthorized persons or processes. In
the proposed dependability framework [30], it refers to
the property that unauthorized persons or processes will
not be able to observe the values/contents of the sensitive
variables of the relevant systems.
Availability: Readiness for correct service. The correct
service is defined as delivered system behavior that is
within the error tolerance boundary. .
Integrity: Absence of malicious external disturbance,
which makes the system output off its desired service.
Maintainability: Ability to undergo modifications and
repairs.
Authenticity: Ability to provide services with provable
origin.
Non-repudiation: Services provided cannot be disclaimed
later.
Although many security solutions have been made for
SGs, most of them are not big data oriented. Recently new
developments have been made for energy big data security
and privacy [31][36]. These new achievements cover three
important aspects of energy big data security: (i) big data ori-
ented cryptosystems [31], [32], (ii) big data oriented privacy-
preserving intelligent financial applications [33], and (iii) big
data oriented anomaly detection [33][35], [37].
IEEE TRANSACTIONS ON SMART GRID, VOL. 7, NO. 5, SEPTEMBER 2016
A. Big Data Oriented Cryptosystems
Cryptography is the most fundamental component in a secu-
rity system. As a SG is of a hierarchical structure, its relevant
big data oriented cryptosystems will normally have a focus on
individual hierarchical levels. Existing energy big data oriented
cryptosystems can be classified into following two categories:
(i) AMI and smart meters oriented big data privacy and
security, and (ii) scalable privacy-preserving data-sharing and
authentication schemes for overall smart grids, and (iii) Smart
grid big data oriented scalable public-key certificate revocation
schemes.
1) AMI and Smart Meters Oriented Big Data Privacy and
Security: AMI and smart meters are major components of
big data generation and they provide the major source for
intelligent applications. Data aggregation mechanisms can
effectively reduce the data size for further processing and
address the privacy issue. Although much research effort has
been made on how to perform data aggregation, less effort has
been made on how to make methods of performing data aggre-
gation and encryption scalable. Recently some progress has
been made on this through reducing the computation load of
real-time smart meter data encryption [38], and providing scal-
able encryption [39], [40]. In [38], an online/offline attribute
based encryption (ABE) scheme is proposed which is based
on the J.Hurs ABE scheme with hidden policy. The involved
data privacy and policy privacy are achieved due to the prop-
erties of the J.Hurs ABE scheme with hidden policy. On the
other hand, the computational load of the encryption is signifi-
cantly reduced by splitting the encryption algorithm into online
and offline phases where the tedious decryption operations can
be delegated to the offline phases. This can effectively reduce
the real-time computing demand which is related to the big
data velocity aspect.
In [39], a privacy-enhanced data aggregation scheme is
proposed against internal attackers in smart grid. In this
scheme, electricity suppliers can learn about the current energy
usage of each neighborhood to arrange energy supply and
distribution without knowing the individual electricity con-
sumption of each user. Secure batch verification and formal
proofs are provided. This is the first scheme against internal
attackers.
In this proposed scheme, the aggregator and users generate
their relevant public-private key pairs, then the offline trusted
third party (TTP) sends the blinding factors to the aggrega-
tor and each user. The end user Ui collects his/her power
consumption reading mi from own smart meter and computes

a ciphertext CTi = gm ri i
0 (H2 (t)h ) and a signature i . Upon
i
receiving the signature and cipertext pairs from all users, the
aggregator cancompute the aggregated energy usagesof n
the
neighborhood ni=1 mi by taking the discrete log of (g) i=1 mi .
This is feasible as the total power usage within a neighbor-
hood is not a large number within a regular interview, and
the computation can take polynomial time using the Pollards
lambda method.
Although such privacy-preserving data aggregation solu-
tions are strong from cryptographic aspect, they are vulnerable
to human factor attack [41], [42]. For example, an adversary
HU AND VASILAKOS: ENERGY BIG DATA ANALYTICS AND SECURITY: CHALLENGES AND OPPORTUNITIES
can infer a persons meter reading from the knowledge of the
persons presence or absence in the property. In [42], a new
attack has been identified and formalized. The attack could
exploit about the presence or absence of a specific person
to infer his meter readings via human-factor-aware differen-
tial aggregation (HAD). This attack could not be addressed
by existing privacy-preserving aggregation protocols proposed
for smart grids. In order to address this problem, two new
protocols, including basic scheme and advanced scheme, have
been proposed to resist the HAD attack [42].
Wan et al. [40] show that a recent proposed key management
scheme for AMI suffers from the desynchronization attack
and is not scalable. To address these issues, a new scalable
key management scheme (SKM) based on the combination of
identity-based cryptosystem and efficient key tree technique
is proposed [40]. The complexity of the proposed SKM is
O(log n) in each aspect of computation and communication,
which is scalable in terms of the number of smart meters n.
Although it provides a solution for scalable key management
for AMI and associated smart meters, it does not address the
issue of secure communication and /or secure data sharing
with other parts of the smart grid, e.g., control centers etc. It
is desirable to develop a framework that can cover the whole
smart grid operation ranges. This issue can be addressed by
cryptosystems designed for the overall system.
2) Scalable Privacy-Preserving Data-Sharing and
Authentication Schemes for Overall Smart Grids: A smart
grid is a large distributed system with hierarchical structure.
Most existing research works consider only the lowest level
components, e.g., smart meters, which will be exposed
to new attacks when coming across different hierarchical
levels [43]. Recently some developments have been made
to address this issue by covering higher hierarchical levels.
In [44], a SmartAnalyzer security analysis tool is proposed
for analyzing threats in AMI. It can provide formal modeling
of AMI configuration and systematic diagnosing of smart
meter unusual traces. The accuracy and scalability of the tool
are evaluated on an AMI testbed and various synthetic test
networks. In [42], the scope and functionalities of a smart
grid have been introduced covering its automation and
control system, and communications. It presents a general
SCADA cyberattack process and proposed a conceptual
layered framework for protecting power grid automation
systems against cyberattacks without compromising the
timely availability of control and signal data. The on-site
test of the developed security prototype system has been
discussed.
In addition to the work on directly handling the smart
meter data, research efforts have been made on designing
scalable cryptosystems for the overall smart grids [45][49].
The homomorphic encryption mechanism has been used to
design a distributed smart grid dataset access control scheme
achieving both privacy-preserving data aggregation and access
control [49], and scalable privacy-preserving demand response
scheme with adaptive key evolution in smart grids [47].
Recently more general SG oriented scalable cryptosystems
have been developed [48]. In [48], an identity-based ring sig-
nature scheme with forward security has been proposed. The
2429
property of the identify-based ring signature can eliminate
the process of certificate verification. The property of for-
ward security can enjoy the following advantage: if a secrete
key of a user has been compromised, all previous gener-
ated signatures including this user still remain valid. This
property is very useful for large scale data sharing SG sys-
tem because it is infeasible to request all data owners to
re-authenticate their data when one user has been compro-
mised. In [31], a secure cloud computing based framework for
big data information management in smart grids is proposed.
The proposed framework is a hierarchical structure consist-
ing of multiple interconnected regional clouds. Four types of
services are provided via this cloud computing architecture:
(i) Information storages. They store all smart grid informa-
tion received from intelligent devices such as smart meters.
(ii) General user services. They supply all services that an
energy consumer will need. (iii) Control and management ser-
vices, and (iv) Electricity distribution services. Based on this
architecture, an identity-based cryptography scheme is pro-
posed for authentication of data and nodes. Confidentiality
and nonrepudiation security features have also been supported.
The proposed security scheme provides two related but differ-
ent constructions for confidentiality service and authentication
service.
a) Confidentiality service: Key generation phase: The
trusted external party PKG generates two groups G1 , G2
of prime order q and an computational efficient pairing:
e : G1 G2 G2 , generator g and hash functions Hi .
It picks up a random key s Zq and computes its pub-
lic key u = gs . Then it sets up the master key mk = s
for the top cloud, and distribute the set of public parameters
params = (G1 , G2 , e, g, u, H1 , H2 ) to top and regional clouds
and end-users. Upon receiving identities TC, IS, A, and EU
from the top cloud, information storage unit in the regional
cloud, service in the regional cloud, and the end user, the
PKG will calculate their corresponding private keys KTC =
H1 (TC)s , KIS = H1 (IS)s , KSerA = H1 (SerA)s , KEU = H1 (EU)s
and returns to them.
Encryption phase: For the encryption of the message M
intended for the top cloud, an entity in the regional cloud
can select a random key r Zq and compute the ciphertext
C1 = gr , C2 = M e(u, H1 (TC))r . Then it sends the cipher-
text to the top cloud. Once receiving the above ciphertext, the
top cloud TC can recover the message as M = C2 /e(C1 , KTC ).
Similarly encryption and decryption can be performed between
an end user and information storage unit in the regional cloud;
proxy and the service etc.
b) Authentication service: The key generation process is
similar to the identity based encryption (IBE) scheme used
in the confidentiality service oriented scheme. A signature
component is added for the authentication service as follows.
For a message M, a cloud can pick a random number r Zq
and compute its signature as 1 = grM KTC , 2 = gr0 where
g1 = H1 (TC) G1 , gM = H1 (TC, M) G1 . Any party
can verify this signature by using the clouds pubic key by
checking e(g0 , 1 ) = e(u, g1 )e(2 , gM ).
3) Smart Grid Big Data Oriented Scalable Public-Key
Certificate Revocation Schemes: Most of cryptosystems for
2430
energy big data are public key infrastructure (PKI) based due
to the advantages of access control, and nonrepudiation etc.
However PKI schemes will require public key certificates in
binding the user and the associated public key. There exists
a need for certificate revocation when the certificate expires,
security policy changes and/or a key or a node has been
compromised. In addressing this issue, scalable and efficient
SG oriented PKI certificate revocation schemes have been
proposed [50][52]. In [52], a Bloom filter based scheme is
proposed for the efficient certificate revocation for large-scale
AMI networks. The proposed scheme is constructed based on
the Merkle tree to enable the gateway to provide proof for cer-
tificate revocation without contacting the certificate authority
in achieving significant overhead savings. In [50], a scheme
based on the compressed Certificate Revocation Lists (CRLs)
is proposed for pseudonymous public key infrastructure. It
is shown the proposed scheme is secure and the size of
the certificate revocation list is linear with the number of
revoked certificate series. This scheme has been applied to
the vehicle-to-grid communication application [51].
The cryptosystems discussed above are mainly concerned
with data security and node authentication. They are not
suitable for other important energy big data security related
applications such as privacy-preserving intelligent applica-
tions, and energy big data oriented anomaly detection. For
these applications, different security mechanisms are needed.
4) Big Data Oriented Privacy-Preserving Intelligent
Applications: For privacy-preserving intelligent applications,
the big challenge is how to find useful information from
potential protected/encrypted big data. Efficiency/scalability is
another hard constraint. Privacy-preserving pricing is a typical
application of big energy analytics. In [53], a usage-based
dynamic pricing (UDP) scheme is proposed for smart grid
which is privacy-preserving. The proposed scheme has
4 phases: (i) The utility company first defines the usage
threshold em of the local power grid, defines the dynamic
price function f(), and selects random secretes for each
community and its gateway C. (ii) Customers will report their
electricity usage per time slot in the encrypted mode to their
community gateway C where decryption will be conducted
to obtain these data. Then it sends back a price indicator to
the customer. (iii) The community gateway C will forward
the received electricity usage to the utility company. (iv) The
utility company will recover the electricity usage and compute
the actual electricity price for each customer. The above
scheme is constructed through the homomorphic encryption
technique where privacy is guaranteed. Scalability and
efficiency have been achieved via this distributed encryption
process.
The above scheme is targeting a specific application of the
SG in a community environment. It is desirable to develop
more generic energy big data intelligent schemes to accommo-
date variant applications in a more general environment. One
solution is to retrieve general features over encrypted energy
big data [32], [54][56]. In [54], a privacy-preserving deep
computation model is proposed. The proposed model uses
Brakerski-Gentry-Vaikuntanathan (BGV) encryption scheme
to encrypt the private data and then employs cloud servers
IEEE TRANSACTIONS ON SMART GRID, VOL. 7, NO. 5, SEPTEMBER 2016
to perform the high-order back-propagation algorithm on the
encrypted data for deep computation model training. The
proposed scheme is highly scalable with privacy-preserving.
Direct feature retrieval over encrypted big data is very
difficult and costly. A more cost-effective way is to first
retrieve all relevant records (raw features) from the big data
and then perform relevant intelligent applications based on
these records. A range query has been a popular opera-
tion in database in retrieving all records where some value
is within a range, which will very useful for the efficient
application over big data. For example, in the smart grid
financial application, energy buyers can filter out the energy
with reasonable price with the help of the keyword specify-
ing the range of price. For privacy-preserving query, query
over encrypted database is the solution. Unfortunately exist-
ing encrypted keyword search schemes in smart grid auction
market cannot achieve range query of keywords [56]. In order
to address this issue, some recent research efforts have been
made [32], [55], [56]. In [56], a SG auction market oriented
scheme is proposed which can support both range query and
ranked search over encrypted big data. Based on the homo-
morphic Pailier encryption, it can aggregate multidimensional
keywords of the buyer and the seller where the comparison
between the keywords of all sellers and one buyer can be
conducted with only one calculation. In [32], a more generic
privacy-preserving range query (PaRQ) scheme is proposed to
perform query over encrypted metering data for the intelligent
financial auditing applications. The PaRq constructs a hidden
vector encryption based range query predicate to encrypt the
searchable attributes and session keys of the encrypted data,
The requesters range can be transferred into two query tokens
to be used to find the matched query results. More specifi-
cally, for a range query p = (a1 x1 b1 ) (a2 x2
b2 ) (an xn bn ), the control center will divide P into
two parts:
p = (a1 x1 ) (a2 x2 ) (an xn ),
p = (x1 b1 ) (x2 b2 ) (xn bn ).
By using Boneh and Waters Hidden Vector Encryption (HVE)
predicate encryption [57], privacy-preserving tokens with con-
stant size can be generated which can reduce communication
overhead, computation overhead and response time signifi-
cantly. Security analysis demonstrates that data confidentiality
and query privacy are preserved. Also simulation results show
that the proposed scheme is scalable which is suitable for big
data analytics. A major limitation of this scheme is the specific
environment setting where a central control center is connected
to two cloud servers and acts as a proxy for the users and
the searcher. A practical SG would require more general and
complex environment such as more distributed cloud servers,
multiple control centers, and direct communication between
users and clouds etc.
5) Summary of Energy Big Data Oriented Cryptosystems:
In Table II, we provide summarized comparisons of var-
ious energy big data oriented cryptosystems. The com-
parison is made from the aspects of functionality includ-
ing confidentiality, integrity, authenticity, non-repudiation,
HU AND VASILAKOS: ENERGY BIG DATA ANALYTICS AND SECURITY: CHALLENGES AND OPPORTUNITIES
TABLE II
S UMMARY OF E NERGY B IG DATA O RIENTED C RYPTOSYSTEM
and privacy-preserving intelligence; scalability, cost and
data source.
B. Big Data Oriented Anomaly Detection
Although traditional cryptographic methods can provide
strong solutions for message based security, they are inef-
fective in addressing issues related to real-time tight cyber-
physical couplings because conventional cryptography does
not consider real-time factor and tight cyber-physical cou-
plings. On the other hand, a power system obeys relevant
physical laws. Therefore we can utilize such physical process
knowledge to help detect anomaly behavior which could be
related to cyber-attacks and/or energy theft. In [29], a reputa-
tion based trust management system has been introduced. It
aims to address the risks of cyber-attacks originated from mis-
using information generated within the smart grid. The risks
of cyber-attacks are broad [29]:
Customer profiling attack: smart meter data contain pri-
vate information of users usage. They can be used to
determine, among many other things, when residential
customers are and are not at home. Such information will
be detrimental to individuals if it is used by thieves.
IP spoofing attack: It can be used to redirect smart grid
information to an adversarys own computer for further
analysis. It can also cause a smart grid communica-
tion node appear non-responsive and faulty, which will
2431
reduce its assigned trust value in trust based security
mechanisms.
Man-in-the-middle attack: Such attacks can either
covertly monitor or manipulate information between two
smart grid communication nodes.
Denial of service attack: It can prevent smart grid from
providing services.
System hijacking attack: It can obtain unauthorized
remote access to a smart grid node.
Anomaly detection is a powerful mechanism for smart grid
fault detection [58], power theft and fraud detection [36], [58],
and cyber intrusion detection [37], [58]. Unfortunately most
of the existing solutions are not big data oriented. Recently
several algorithms have been developed in addressing this
issue [33][35], [58]. These algorithms can be classified into
following two categories: (i) general energy big data oriented
anomaly detection algorithms applicable to electricity theft,
fault and cyber intrusions, (ii) energy theft based algorithm,
and (iii) physical process based anomaly detection scheme
against false data-injection attacks.
1) Generic Energy Big Data Anomaly Detection: In a smart
grid, anomaly based intrusion detection system (IDS) design
is far more challenging and also critical due to the real-time
and cyber-physical interaction factors. For example, a delayed
command signal or a compromised command signal can cause
the physical power system to become unstable. As there
could exist multiple sources, e.g., fault, cyber intrusion and
2432
electricity theft, leading to the observed abnormal data of
a smart grid, it is not always easy to determine which source
is behind the observed abnormal data. Therefore research
efforts have been made to design general energy big data ori-
ented anomaly detection algorithms applicable to electricity
theft, fault and cyber intrusions without identifying the source
causing the anomaly [33], [34]. In [33], a scalable method is
proposed to observe anomaly behavior over energy big data.
Supervised learning classifiers have been applied to finding
anomaly data. It can be used to detect electricity theft, fault,
and cyber intrusions. This approach has utilized some ad hoc
intuitions in addressing the big data challenge.
A more rigorous algorithm is proposed in [34] for detect-
ing energy big data anomaly algorithm. In [34], a data-driven
scheme is proposed for monitoring the abnormal events hap-
pening in a smart grid. In order to address the issue of
high-dimensional analysis, a big data architecture is designed
based on the random matrix theory. A statistic named mean
spectral radius (MSR) is proposed to reflect the correlations
of system data in different dimensions. The smart grid state
can be monitored from a distributed MSR estimated locally.
It is shown that anomaly events of the smart grid can be
detected with the proposed scheme under reasonable cost.
More specifically suppose data collected at the energy system
at time ti is expressed as a vector x ti . Use x to repre-
sent all the collections of the raw data accumulated in the
database. The size (xti ) represents the dimension of the data at
time ti whose upper bound will be very big due to the large
number of variables in the system. By using the probabil-
ity Ring Law, the smart grid system state can be estimated
as the mean of the radius of all eigenvalues of Z that is
formed by the interested data area specified by a split-window
(e.g., N rows, T columns). If this value deviates significantly
across two consecutive time windows, an anomaly will be
announced.
2) Energy Theft Detection: Energy theft has been a seri-
ous problem in the power service industry. It is estimated
that the annual losses in the United States alone can reach
$6 billion [59]. In [59], investigation has been conducted on
adversary means of defrauding by manipulating AMI systems.
An energy theft attack tree has been constructed to explore
various vulnerabilities of energy theft. It is found that manip-
ulating the demand information at AMI is the most effective
way for energy theft [59].
Energy theft is another important issue which can be
detected via machine learning based anomaly detection.
In [60], a multi-sensor energy detection framework for the
Ami is proposed. The proposed scheme, named as AMIDS, is
an AMI intrusion detection system. It fuses meter audit logs of
physical and cyber events with consumption data to increase
the accuracy in detecting theft-related behavior. One limita-
tion of this approach is the difficulty in collecting reliable
physical activity logs for the large-scale smart grid. In [61],
a game-theoretic framework is proposed to model the adver-
sarial nature of the electricity theft problem. It considers two
environments, i.e., unregulated monopoly and perfect compe-
tition. This framework can help detect electricity theft through
the observation of the power usage behavior. Suppose we use
IEEE TRANSACTIONS ON SMART GRID, VOL. 7, NO. 5, SEPTEMBER 2016
qg , qBf to represent expected energy usage from genuine type
of customers and fraudulent type of customers respectively.
The framework considers following two hypotheses:
iid  
Hg : Y1i , . . . , YKi
= fg , E Yti = qg ,
iid  
Hf : Y1i , . . . , YKi
= ff , E Yti = qBf < qg , (2)
where yit is a meter measurement collected at the ith customers
meter at time t which is a realization of the random variable Yti .
fg , ff denote the probability density function of the meter mea-
surements of the genuine customer and fraudulent customer
respectively. Setting a threshold , for a customer i, then we
have following fraudulent detection rule:
K i
yt < , fraudulent
t=1
K (3)
t=1 yt , genuine.
i
It is interesting to observe that, like many other similar
approaches, the above mentioned game-theoretic framework
for energy theft detection is based on behaviors of individ-
ual meter reading. They are not applicable when dealing with
coordinated attacks. In [62], two types of data attacks have
been investigated. One attack is that an adversary has compro-
mised a sufficient number of smart meters so that the network
state becomes unobservable by the control center. A graph the-
oretic approach has been proposed to characterize the smallest
set of attacked smart meters that can cause network unobserv-
ability. Another type of attack is that an adversary controls
only a small number of smart meters. Such attack can be
examined from a decision theoretic perspective for both the
control center and the adversary.
3) Physical Process Based Anomaly Detection Scheme
Against False Data-Injection Attacks: With the emerging
smart grid, a new type of cyberintrusion is looming which
attempts to alter the power load through the Internet via auto-
matic and distributed software intruding agents [63]. Such
attacks can compromise direct load control command signals,
demand side management price signals etc. In [63], a system-
atical investigation has been conducted to identify a variety
of practical loads that can be vulnerable to Internet-based
load altering attacks. It also provides an overview of potential
defense mechanisms including:
Protection of Command and Price Signal using (i) pri-
vate key encryption and message authentication code,
(ii) efficient group key management
Protection of smart meters and data centers
Load anomaly detection
In a smart grid, state estimation is used to estimate the
power grid state based on the meter measurement data. This
information will be used in the reliable operation of the
smart grid, contingency analysis, optimal power flow, and pric-
ing etc. [64], [65]. Recently false data-injection attacks have
been reported which can bypass existing bad data detection
techniques [65]. This discovery has sparked a strong interest
in exploring various strategies of false data-injection attacks
and countering measures [64][80].
Suppose the state x = (x1 , x2 , . . . , xn )T and the measure-
ment data is expressed as z = Hx + e where H is related to
HU AND VASILAKOS: ENERGY BIG DATA ANALYTICS AND SECURITY: CHALLENGES AND OPPORTUNITIES
network configurations and the measurement error, and e is
assumed normally distributed with zero mean. Conventional
bad data measurement detection is based the observation
whether the measurement residual zHx is larger than a prede-
fined threshold or not [65]. In the basic false data-injection
attacks, the measurement data received at the control center
will be za = z + a where a is the attack vector. When the
adversary chooses a = Hc as the attack vector, we will have
z = H(x + c) + e where such attacks are unobservable as the
measurement residual will be within the bad data detection
threshold [65]. In practice, an adversary can only compromise
a small set of meters, and hence a sparse structure a is gen-
erated where most of as entries are zeros. These k nonzero
entries will correspond to the k compromised meters.
In [67], it is shown that smallest size unobservable attack,
i.e., an observable attack with smallest number of compro-
mised meters k, can be identified in polynomial time via the
graph theory. If the adversary can compromise a much smaller
set of meters, i.e., in the weal attack regime, it is possible
to detect the presence of such attacks via the generalized
likelihood ratio test (GLRT). A similar work can be found
in [81] where the problem of finding the minimum num-
ber of measurement points to be attacked undetectably can
be reduced to minimum cut problems on hypergraphs. Most
recently two data-driven strategies for unobservable attacks
are presented where detailed knowledge of system parameters
are not needed [82]. The first strategy is to affect the system
state by hiding directly the attack vector in the system sub-
space. The second strategy misleads the bad data detection
mechanism so that data not under attack are removed.
Most recent developments along this line are: (i) the
centralized sparse attack construction case is extended to
a distributed framework for both the estimation and attack
problems. Corresponding optimization is formulated together
with a proposed solution [73], (ii) an optimal data-injection
attacking strategy is designed to selecting a set of meters which
can cause maximum damage. A temporal-based detection
using online nonparametric cusum change detection mecha-
nism is proposed to detect such attacks [64], and (iii) a new
CUSUM-type algorithms based on the generalized likelihood
ration (GLR) is proposed for centralized and distributed cyber-
attack detection. The proposed scheme is efficient in terms of
low communication overhead and small detection delay [68].
The above schemes are based on the principle of power
system physical process. The fundamental cause of false
data-injection attacks is the modification of sensed meter
data. At the data level, cryptographic authentication mech-
anisms such as message authentication code can address
effectively the issue of data integrity. Recently some effort
has been made in this field. In [72], an anomaly detection
mechanism is designed with an integrated watermarking com-
ponent to address the false data injection attacks in smart
grids. This scheme can help detect more stealthy attacks
that involve subtle manipulation of the measurement data.
In [70], a polynomial-based compromise-resilient en-route fil-
tering scheme (PREF) is proposed to filtering false injected
data effectively and achieve a high resilience to the number of
compromised nodes without relying on static routes and node
2433
localization. The limitation is the specific application environ-
ment of wireless sensor networks where the filtering relies on
the cooperation of the sensor nodes during the message for-
warding process. It is not applicable for the meter that does
not go through this forwarding process, e.g., the meter is just
one hop to the smart grid. Also the pre-stored master key
mechanism will make it costly in key update. As a matter of
fact, hardware attacks can reveal data stored in the hardware.
Most recently a short-term state forecasting-aided method is
proposed to detect smart grid general false data injection
attacks [83]. This idea is to extend the approximate DC model
to a more general linear model that can handle both supervi-
sory control and data acquisition and phasor measurement unit
measurements.
VI. DATA -D RIVEN S CHEMES FOR R ESILIENT SG
O PERATION : S ECURITY-C ONSTRAINED
O PTIMAL P OWER F LOW (SCOPF)
The network state estimated from the state estimator
can be used for resilient SG operations including planning,
operational planning, pricing and real-time operation of the
system [84]. In some cases, a severe incident, e.g., natural
disaster or cyber-attack can trigger contingency actions and
it is very important that the SG can maintain resilient opera-
tions. The SCOPF is a useful tool for this purpose which can
make correct actions/decisions against a set of predefined con-
tingencies. Due to the big data nature of the SG, the SCOPF
problem will become a nonlinear, non-convex, and large-scale
optimization problem with both continues and discrete vari-
ables, which is very challenging [84]. SCOPF problem usually
has two aspects (i) SCOPF problem and (ii) SCOPF problem
solution including reducing the size of the SCOPF problem,
efficient algorithms for handling challenges brought about by
the mixed continuous variables and discrete variables.
SCOPF problem formulation includes the modelling of
a limited number of post-contingency control actions, deter-
mining the minimum number of control actions, determining
the sequence of control actions, and handling voltage and tran-
sient stability in SCOPF [84]. In [85], an optimization-based
approach is proposed for identifying constraints that are nec-
essary and sufficient to the description of the feasible set of an
SCOPF problem. It is shown that the resulting sizes of SCOPF
problems are much smaller and can be solved much faster.
Recently some efforts have been made to incorporate cyber-
attack issue into the SCOPF problem formulation. In [86],
investigation has been conducted in analyzing the behavior
of the Optimal Power Flow (OPF) algorithm in the pres-
ence of false data-injection and the resulting consequences
to the system operator. It characterizes the set of attacks
that may lead the operator to apply the erroneous OPF
recommendation. In [87], it considers the problem of charac-
terizing impacts of bad data on real-time locational marginal
price (LMP). Numeric simulations are provided to illustrate
the worst performance for IEEE-14 and IEEE-118 networks.
SCOPF problem solution includes reducing the size of
the SCOPF problem, efficient algorithms for handling chal-
lenges brought about by the mixed continuous variables and
2434
discrete variables [84]. For large-scale smart grids, it is impos-
sible to obtain the direct solution of the SCOPF problems
due to the memory limitation and/or prohibitive computation
times [88]. As a result, many indirect solutions are proposed
such as iterative contingency selection schemes; decomposi-
tion method; network compression and hybrid methods that
combine contingency selection and network compression [88].
VII. O PEN R ESEARCH I SSUES
Smart energy big data analytics is a very complex and chal-
lenging topic. In addition to sharing many common issues with
the generic big data analytics, smart energy big data involve
extensively with physical processes where data intelligence
can have a huge impact to the safe operation of the systems
in real-time. This tight cyber-physical dimension has brought
about many exciting research problems. Following is the list
of open research problems:
Holistic and modular architectures: Existing architectures
for smart energy big data analytics are based on generic
cloud computing architectures. They are either too gen-
eral to be implemented or too specific without covering
sufficient functionalities. A holistic and modular energy
big data analytics architecture based platform is needed
to ensure the widest coverage of issues related to smart
energy big data analytics and interoperability among
various modules including new modules.
Platform incorporating real-time control: Existing plat-
forms for smart energy big data analytics is mainly for
data sharing and intelligence, or precisely system mon-
itoring and financial related intelligence. It is desirable
to incorporate a real-time control module that can pro-
duce real-time physical control signals based on the smart
energy big data analytics.
Co-design of smart energy big data analytics and secu-
rity mechanism: Existing smart energy big data analytics
schemes and the security schemes are designed sepa-
rately. Security functions are mostly aftermath thoughts.
A co-design of smart energy big data analytics and
security mechanisms can produce a seamless integrated
framework which can reduce the security risk to the
minimum.
Distributed and parallel intelligence: Smart energy big
data analytics is experiencing data explosion collected
from distributed sources. A distributed and parallel intel-
ligence can effectively address this problem which can
also reduce the raw data accumulation and communica-
tion significantly. Existing aggregation or summarization
methods can achieve the same aim of reducing large raw
data. However as aggregation or summarization meth-
ods are targeting local raw data without considering
overall system target, they can lose very useful infor-
mation which would be needed for specific applications.
A good distributed intelligence algorithm should be built
upon a solid theoretical basis to approximate the relevant
overall performance indicator. For example, for anomaly
detection in a wireless sensor network, distributed local
intelligence based on the observations from a single node
IEEE TRANSACTIONS ON SMART GRID, VOL. 7, NO. 5, SEPTEMBER 2016
or several neighboring nodes should be able to esti-
mate the overall sensor network data probability density
distribution with a solid theoretical basis [74], [75].
VIII. C ONCLUSION
In this paper, we have surveyed the latest developments in
the field of energy big data analytics and security/privacy.
We have provided a comprehensive coverage of energy
big data analytics and security/privacy ranging from energy
big data architecture, intelligence applications, cryptosystem
design, system security assessment, cyber intrusion detection,
secure financial intelligent application, and electricity theft.
In addition to the usual 3Vs challenges of energy big data,
we have also covered the real-time and tight cyber-physical
coupling which are salient features in a smart grid. We have
also proposed an energy big data oriented taxonomy for better
understanding the complicated and intriguing relations among
various components, security issues and associated solutions.
Finally we have provided and discussed various open research
questions and future research directions.
R EFERENCES
[1] Ecotricity. The End of Fossil Fuels. Accessed on Jul. 21, 2015.
[Online]. Available: https://www.ecotricity.co.uk/our-green-energy/
energy-independence/the-end-of-fossil-fuels
[2] J. Hu, H. R. Pota, and S. Guo, Taxonomy of attacks for agent-
based smart grids, IEEE Trans. Parallel Distrib. Syst., vol. 25, no. 7,
pp. 18861895, Jul. 2014.
[3] N. Bui, A. P. Castellani, P. Casari, and M. Zorzi, The Internet of
energy: A Web-enabled smart grid system, IEEE Netw., vol. 26, no. 4,
pp. 3945, Jul./Aug. 2012.
[4] D. Alahakoon and X. Yu, Smart electricity meter data intelligence for
future energy systems: A survey, IEEE Trans. Ind. Informat., vol. 12,
no. 1, pp. 425436, Feb. 2016, doi: 10.1109/TII.2015.2414355.
[5] Z. Huang, H. Luo, D. Skoda, T. Zhu, and Y. Gu, E-Sketch: Gathering
large-scale energy consumption data based on consumption patterns,
in Proc. IEEE Int. Conf. Big Data (Big Data), Washington, DC, USA,
2014, pp. 656665.
[6] J. Yin, P. Sharma, I. Gorton, and B. Akyoli, Large-scale data challenges
in future power grids, in Proc. IEEE 7th Int. Symp. Service Orient. Syst.
Eng. (SOSE), Redwood City, CA, USA, 2013, pp. 324328.
[7] M. Aiello and G. A. Pagani, The smart grids data generating poten-
tials, in Proc. Federated Conf. Comput. Sci. Inf. Syst. (FedCSIS),
Warsaw, Poland, 2014, pp. 916.
[8] K. L. Wagstaff, Machine learning that matters, in Proc. 29th Int. Conf.
Mach. Learn. (ICML), Pasadena, CA, USA, 2012, pp. 529536.
[9] D. A. Powner, Electricity grid modernization: Progress being made on
cybersecurity guidelines, but key challenges remain to be addressed,
GAO report, United States Gov. Account. Office, Washington,
DC, USA, Tech. Rep. GAO-11-117, Jan. 2011. [Online]. Available:
http://www.gao.gov/new.items/d11117.pdf
[10] S. Simitis, From the market to the polis: The EU directive on the
protection of personal data, Iowa Law. Rev., vol. 80, no. 3, p. 445,
1994.
[11] S. Bera, S. Misra, and J. J. P. C. Rodrigues, Cloud computing appli-
cations for smart grid: A survey, IEEE Trans. Parallel Distrib. Syst.,
vol. 26, no. 5, pp. 14771494, May 2015.
[12] J. W. Taylor and P. E. McSharry, Short-term load forecasting methods:
An evaluation based on european data, IEEE Trans. Power Syst., vol. 22,
no. 4, pp. 22132219, Nov. 2007.
[13] M. Frincu, C. Chelmis, M. U. Noor, and V. Prasanna, Accurate and
efficient selection of the best consumption prediction method in smart
grids, in Proc. IEEE Int. Conf. Big Data (Big Data), Washington, DC,
USA, 2014, pp. 721729.
[14] S. Aman, Y. Simmhan, and V. K. Prasanna, Holistic measures for eval-
uating prediction models in smart grids, IEEE Trans. Knowl. Data Eng.,
vol. 27, no. 2, pp. 475488, Feb. 2015.
HU AND VASILAKOS: ENERGY BIG DATA ANALYTICS AND SECURITY: CHALLENGES AND OPPORTUNITIES
[15] K. Nose-Filho, A. D. P. Lotufo, and C. R. Minussi, Short-term
multinodal load forecasting using a modified general regression neu-
ral network, IEEE Trans. Power Del., vol. 26, no. 4, pp. 28622869,
Oct. 2011.
[16] L. Ghelardoni, A. Ghio, and D. Anguita, Energy load forecasting using
empirical mode decomposition and support vector regression, IEEE
Trans. Smart Grid, vol. 4, no. 1, pp. 549556, Mar. 2013.
[17] T. Senjyu, P. Mandal, K. Uezato, and T. Funabashi, Next day load
curve forecasting using hybrid correction method, IEEE Trans. Power
Syst., vol. 20, no. 1, pp. 102109, Feb. 2005.
[18] Y. Chen et al., Short-term load forecasting: Similar day-based wavelet
neural networks, IEEE Trans. Power Syst., vol. 25, no. 1, pp. 322330,
Feb. 2010.
[19] R. E. Abdel-Aal, Short-term hourly load forecasting using abduc-
tive networks, IEEE Trans. Power Syst., vol. 19, no. 1, pp. 164173,
Feb. 2004.
[20] S. Li, P. Wang, and L. Goel, A novel wavelet-based ensemble method
for short-term load forecasting with hybrid neural networks and fea-
ture selection, IEEE Trans. Power Syst., vol. 31, no. 3, pp. 17881798,
May 2016.
[21] A. Bracale, P. Caramia, G. Carpinelli, A. R. Di Fazio, and P. Varilone,
A bayesian-based approach for a short-term steady-state forecast of
a smart grid, IEEE Trans. Smart Grid, vol. 4, no. 4, pp. 17601771,
Dec. 2013.
[22] W. Lee, B.-W. On, I. Lee, and J. Choi, A big data management sys-
tem for energy consumption prediction models, in Proc. 9th Int. Conf.
Digital Inf. Manag. (ICDIM), 2014, pp. 156161.
[23] R. Christie, Power systems test case archive, Dept. Elect. Eng.,
Univ. Washington, Seattle, WA, USA, 2000. [Online]. Available:
http://www.ee.washington.edu/research/pstca/
[24] M. Shahidehpour, F. Tinney, and Y. Fu, Impact of security on
power systems operation, Proc. IEEE, vol. 93, no. 11, pp. 20132025,
Nov. 2005.
[25] N. Balu et al., On-line power system security analysis, Proc. IEEE,
vol. 80, no. 2, pp. 262282, Feb. 1992.
[26] K. J. Ross, K. M. Hopkinson, and M. Pachter, Using a dis-
tributed agent-based communication enabled special protection system
to enhance smart grid security, IEEE Trans. Smart Grid, vol. 4, no. 2,
pp. 12161224, Jun. 2013.
[27] C. Vellaithurai, A. Srivastava, S. Zonouz, and R. Berthier, CPINDEX:
Cyber-physical vulnerability assessment for power-grid infrastructures,
IEEE Trans. Smart Grid, vol. 6, no. 2, pp. 566575, Mar. 2015.
[28] D. Kundur, X. Feng, S. Liu, T. Zourntos, and K. L. Butler-Purry,
Towards a framework for cyber attack impact analysis of the elec-
tric smart grid, in Proc. 1st IEEE Int. Conf. Smart Grid Commun.
(SmartGridComm), Gaithersburg, MD, USA, 2010, pp. 244249.
[29] J. Fadul, K. Hopkinson, C. Sheffield, J. Moore, and T. Andel, Trust
management and security in the future communication-based smart
electric power grid, in Proc. 44th Hawaii Int. Conf. Syst. Sci. (HICSS),
2011, pp. 110.
[30] J. Hu, I. Khalil, S. Han, and A. Mahmood, Seamless integration of
dependability and security concepts in SOA: A feedback control system
based framework and taxonomy, J. Netw. Comput. Appl., vol. 34, no. 4,
pp. 11501159, 2011.
[31] J. Baek, Q. H. Vu, J. K. Liu, X. Huang, and Y. Xiang, A secure cloud
computing based framework for big data information management of
smart grid, IEEE Trans. Cloud Comput., vol. 3, no. 2, pp. 233244,
Apr./Jun. 2014.
[32] M. Wen et al., PaRQ: A privacy-preserving range query scheme over
encrypted metering data for smart grid, IEEE Trans. Emerg. Topics
Comput., vol. 1, no. 1, pp. 178191, Jun. 2013.
[33] W. Hurst, M. Merabti, and P. Fergus, Big data analysis techniques for
cyber-threat detection in critical infrastructures, in Proc. 28th Int. Conf.
Adv. Inf. Netw. Appl. Workshops (WAINA), Victoria, BC, Canada, 2014,
pp. 916921.
[34] X. He et al., A big data architecture design for smart grids
based on random matrix theory, IEEE Trans. Smart Grid, 2015,
doi: 10.1109/TSG.2015.2445828.
[35] S. Pan, T. Morris, and U. Adhikari, Developing a hybrid intrusion
detection system using data mining for power systems, IEEE Trans.
Smart Grid, vol. 6, no. 6, pp. 31043113, Nov. 2015.
[36] C. C. O. Ramos, A. N. de Sousa, J. P. Papa, and A. X. Falco,
A new approach for nontechnical losses detection based on optimum-
path forest, IEEE Trans. Power Syst., vol. 26, no. 1, pp. 181189,
Feb. 2011.
2435
[37] R. Mitchell and I.-R. Chen, Behavior-rule based intrusion detection
systems for safety critical smart grid applications, IEEE Trans. Smart
Grid, vol. 4, no. 3, pp. 12541263, Sep. 2013.
[38] Z. Wang, F. Chen, and A. Xia, Attribute-based online/offline encryp-
tion in smart grid, in Proc. 24th Int. Conf. Comput. Commun.
Netw. (ICCCN), Las Vegas, NV, USA, 2015, pp. 15.
[39] C.-I. Fan, S.-Y. Huang, and Y.-L. Lai, Privacy-enhanced data aggrega-
tion scheme against internal attackers in smart grid, IEEE Trans. Ind.
Informat., vol. 10, no. 1, pp. 666675, Feb. 2014.
[40] Z. Wan, G. Wang, Y. Yang, and S. Shi, SKM: Scalable key management
for advanced metering infrastructure in smart grids, IEEE Trans. Ind.
Electron., vol. 61, no. 12, pp. 70557066, Dec. 2014.
[41] W. Jia, H. Zhu, Z. Cao, X. Dong, and C. Xiao, Human-factor-aware
privacy-preserving aggregation in smart grid, IEEE Syst. J., vol. 8, no. 2,
pp. 598607, Jun. 2014.
[42] D. Wei, Y. Lu, M. Jafari, P. M. Skare, and K. Rohde, Protecting smart
grid automation systems against cyberattacks, IEEE Trans. Smart Grid,
vol. 2, no. 4, pp. 782795, Dec. 2011.
[43] X. Dong, J. Zhou, and Z. Cao, Efficient privacy-preserving tem-
poral and spacial data aggregation for smart grid communications,
Concurrency Comput. Pract. Exp., vol. 28, no. 4, pp. 11451160, 2016.
[44] M. A. Rahman, E. Al-Shaer, and P. Bera, A noninvasive threat analyzer
for advanced metering infrastructure in smart grid, IEEE Trans. Smart
Grid, vol. 4, no. 1, pp. 273287, Mar. 2013.
[45] Q. Li et al., ESO: An efficient and secure outsourcing scheme for smart
grid, in Proc. Int. Conf. Wireless Commun. Signal Process. (WCSP),
Hangzhou, China, 2013, pp. 16.
[46] E. Gonzalez, L. B. Kish, R. S. Balog, and P. Enjeti, Information
theoretically secure, enhanced Johnson noise based key distribution over
the smart grid with switched filters, PLoS One, vol. 8, no. 7, 2013,
Art. no. e70206.
[47] H. Li et al., EPPDR: An efficient privacy-preserving demand response
scheme with adaptive key evolution in smart grid, IEEE Trans. Parallel
Distrib. Syst., vol. 25, no. 8, pp. 20532064, Aug. 2014.
[48] X. Huang et al., Cost-effective authentic and anonymous data shar-
ing with forward security, IEEE Trans. Comput., vol. 64, no. 4,
pp. 971983, Apr. 2015.
[49] S. Ruj and A. Nayak, A decentralized security framework for
data aggregation and access control in smart grids, IEEE Trans. Smart
Grid, vol. 4, no. 1, pp. 196205, Mar. 2013.
[50] M. M. E. A. Mahmoud, J. Misic, and X. Shen, Efficient public-key
certificate revocation schemes for smart grid, in Proc. IEEE Glob.
Commun. Conf. (GLOBECOM), Atlanta, GA, USA, 2013, pp. 778783.
[51] M. M. E. A. Mahmoud, J. Miic, K. Akkaya, and X. Shen, Investigating
public-key certificate revocation in smart grid, IEEE Internet Things,
vol. 2, no. 6, pp. 490503, Dec. 2015.
[52] M. M. E. A. Mahmoud, K. Akkaya, K. Rabieh, and S. Tonyali, An
efficient certificate revocation scheme for large-scale AMI networks,
in Proc. IEEE Int. Perform. Comput. Commun. Conf. (IPCCC), Austin,
TX, USA, 2014, pp. 18.
[53] X. Liang, X. Li, R. Lu, X. Lin, and X. Shen, UDP: Usage-based
dynamic pricing with privacy preservation for smart grid, IEEE Trans.
Smart Grid, vol. 4, no. 1, pp. 141150, Mar. 2013.
[54] Q. Zhang, L. Yang, and Z. Chen, Privacy preserving deep com-
putation model on cloud for big data feature learning, IEEE
Trans. Comput., vol. 65, no. 5, pp. 13511362, May 2015,
doi: 10.1109/TC.2015.2470255.
[55] M. Wen et al., ECQ: An efficient conjunctive query scheme over
encrypted multidimensional data in smart grid, in Proc. IEEE Glob.
Commun. Conf. (GLOBECOM), Atlanta, GA, USA, 2013, pp. 796801.
[56] Y. Yang, H. Li, M. Wen, H. Luo, and R. Lu, Achieving ranked
range query in smart grid auction market, in Proc. IEEE Int. Conf.
Commun. (ICC), Sydney, NSW, Australia, 2014, pp. 951956.
[57] D. Boneh and B. Waters, Conjunctive, subset, and range queries on
encrypted data, in Theory of Cryptography. Heidelberg, Germany:
Springer, 2007, pp. 535554.
[58] S. S. S. Rawat, V. A. Polavarapu, V. Kumar, E. Aruna, and V. Sumathi,
Anomaly detection in smart grid using rough set theory and K cross val-
idation, in Proc. Int. Conf. Circuit Power Comput. Technol. (ICCPCT),
Nagercoil, India, 2014, pp. 479483.
[59] S. McLaughlin, D. Podkuiko, and P. McDaniel, Energy theft
in the advanced metering infrastructure, in Critical Information
Infrastructures Security. Heidelberg, Germany: Springer, 2010,
pp. 176187.
2436
[60] S. McLaughlin, B. Holbert, A.-Q. Fawaz, R. Berthier, and S. Zonouz,
A multi-sensor energy theft detection framework for advanced meter-
ing infrastructures, IEEE J. Sel. Areas Commun., vol. 31, no. 7,
pp. 13191330, Jul. 2013.
[61] S. Amin, G. A. Schwartz, A. A. Cardenas, and S. S. Sastry, Game-
theoretic models of electricity theft detection in smart utility networks:
Providing new capabilities with advanced metering infrastructure, IEEE
Control Syst., vol. 35, no. 1, pp. 6681, Feb. 2015.
[62] O. Kosut, L. Jia, R. J. Thomas, and L. Tong, Malicious data attacks
on the smart grid, IEEE Trans. Smart Grid, vol. 2, no. 4, pp. 645658,
Dec. 2011.
[63] A.-H. Mohsenian-Rad and A. Leon-Garcia, Distributed Internet-based
load altering attacks against smart power grids, IEEE Trans. Smart
Grid, vol. 2, no. 4, pp. 667674, Dec. 2011.
[64] Q. Yang et al., On false data-injection attacks against power system
state estimation: Modeling and countermeasures, IEEE Trans. Parallel
Distrib. Syst., vol. 25, no. 3, pp. 717729, Mar. 2014.
[65] Y. Liu, P. Ning, and M. K. Reiter, False data injection attacks against
state estimation in electric power grids, ACM Trans. Inf. Syst. Security,
vol. 14, no. 1, 2011, Art. no. 13.
[66] O. Kosut, L. Jia, R. J. Thomas, and L. Tong, Malicious data attacks
on smart grid state estimation: Attack strategies and countermeasures,
in Proc. 1st IEEE Int. Conf. Smart Grid Commun. (SmartGridComm),
Gaithersburg, MD, USA, 2010, pp. 220225.
[67] O. Kosut, L. Jia, R. J. Thomas, and L. Tong, Malicious data attacks
on the smart grid, IEEE Trans. Smart Grid, vol. 2, no. 4, pp. 645658,
Dec. 2011.
[68] S. Li, Y. Yilmaz, and X. Wang, Quickest detection of false data injection
attack in wide-area smart grids, IEEE Trans. Smart Grid, vol. 6, no. 6,
pp. 27152735, Nov. 2015.
[69] L. Liu, M. Esmalifalak, and Z. Han, Detection of false data injection in
power grid exploiting low rank and sparsity, in Proc. IEEE Int. Conf.
Commun. (ICC), Budapest, Hungary, 2013, pp. 44614465.
[70] X. Yang et al., A novel en-route filtering scheme against false
data injection attacks in cyber-physical networked systems, IEEE Trans.
Comput., vol. 64, no. 1, pp. 418, Jan. 2015.
[71] Y. W. Law, T. Alpcan, and M. Palaniswami, Security games for risk
minimization in automatic generation control, IEEE Trans. Power Syst.,
vol. 30, no. 1, pp. 223232, Jan. 2015.
[72] W. Yu, D. Griffith, L. Ge, S. Bhattarai, and N. Golmie, An integrated
detection system against false data injection attacks in the smart grid,
Security Commun. Netw., vol. 8, no. 2, pp. 91109, 2015.
[73] M. Ozay, I. Esnaola, F. T. Vural, S. R. Kulkarni, and H. V. Poor, Sparse
attack construction and state estimation in the smart grid: Centralized
and distributed models, IEEE J. Sel. Areas Commun., vol. 31, no. 7,
pp. 13061318, Jul. 2013.
[74] S. Cui et al., Coordinated data-injection attack and detection in the
smart grid: A detailed look at enriching detection solutions, IEEE Signal
Process. Mag., vol. 29, no. 5, pp. 106115, Sep. 2012.
[75] Y. Yuan, Z. Li, and K. Ren, Modeling load redistribution attacks in
power systems, IEEE Trans. Smart Grid, vol. 2, no. 2, pp. 382390,
Jun. 2011.
[76] A. Ashok and M. Govindarasu, Cyber attacks on power system state
estimation through topology errors, in Proc. IEEE Power Energy Soc.
Gen. Meeting, San Diego, CA, USA, 2012, pp. 18.
[77] O. Vukovi and G. Dn, On the security of distributed power system
state estimation under targeted attacks, in Proc. 28th Annu. ACM Symp.
Appl. Comput., Coimbra, Portugal, 2013, pp. 666672.
[78] A. Tajer, S. Kar, H. V. Poor, and S. Cui, Distributed joint cyber
attack detection and state recovery in smart grids, in Proc. IEEE
Int. Conf. Smart Grid Commun. (SmartGridComm), Brussels, Belgium,
2011, pp. 202207.
[79] L. Liu, Z. Han, H. V. Poor, and S. Cui, Big data processing for smart
grid security, in Big Data Over Networks. Cambridge, U.K.: Cambridge
Univ. Press, 2016, pp. 217243.
[80] J. Zhao, G. Zhang, Z. Y. Dong, and K. P. Wong, Forecasting-aided
imperfect false data injection attacks against power system nonlin-
ear state estimation, IEEE Trans. Smart Grid, vol. 7, no. 1, pp. 68,
Jan. 2016.
[81] Y. Yamaguchi, A. Ogawa, A. Takeda, and S. Iwata, Cyber security
analysis of power networks by hypergraph cut algorithms, IEEE Trans.
Smart Grid, vol. 6, no. 5, pp. 21892199, Sep. 2015.
[82] J. Kim, L. Tong, and R. J. Thomas, Subspace methods for data attack on
state estimation: A data driven approach, IEEE Trans. Signal Process.,
vol. 63, no. 5, pp. 11021114, Mar. 2015.
IEEE TRANSACTIONS ON SMART GRID, VOL. 7, NO. 5, SEPTEMBER 2016
[83] J. Zhao et al., Short-term state forecasting-aided method for detection
of smart grid general false data injection attacks, IEEE Trans. Smart
Grid, vol. pp, issue 99, 2015, pp. 111, doi: 10.1109/TSG.2015.2492827.
[84] F. Capitanescu et al., State-of-the-art, challenges, and future trends
in security constrained optimal power flow, Elect. Power Syst. Res.,
vol. 81, no. 8, pp. 17311741, 2011.
[85] A. J. Ardakani and F. Bouffard, Identification of umbrella constraints in
DC-based security-constrained optimal power flow, IEEE Trans. Power
Syst., vol. 28, no. 4, pp. 39243934, Nov. 2013.
[86] A. Teixeira, H. Sandberg, G. Dn, and K. H. Johansson, Optimal power
flow: Closing the loop over corrupted data, in Proc. Amer. Control
Conf. (ACC), Montral, QC, Canada, 2012, pp. 35343540.
[87] L. Jia, J. Kim, R. J. Thomas, and L. Tong, Impact of data quality on
real-time locational marginal price, IEEE Trans. Power Syst., vol. 29,
no. 2, pp. 627636, Mar. 2014.
[88] L. Platbrood, F. Capitanescu, C. Merckx, H. Crisciu, and L. Wehenkel,
A generic approach for solving nonlinear-discrete security-constrained
optimal power flow problems in large-scale systems, IEEE Trans. Power
Syst., vol. 29, no. 3, pp. 11941203, May 2014.
Jiankun Hu received the B.E. degree from Hunan
University, Changsha, China, in 1983, the Ph.D.
degree in control engineering from the Harbin
Institute of Technology, China, in 1993, and the
Masters by Research degree in computer science
and software engineering from Monash University,
Australia, in 2000. He is a Full Professor and the
Research Director of Cyber Security Laboratory,
School of Engineering and IT, University of
New South Wales, Canberra, Australia. He has been
with Ruhr University Bochum, Germany, on the
prestigious German Alexander von Humboldt Fellowship from 1995 to 1996
and a Research Fellow with the Delft University of the Netherlands, from
1997 to 1998, and a Research Fellow with Melbourne University, Australia,
from 1998 to 1999.
He is a Guest Professor with the College of Mathematics, Shandong
University, China, and the Guest Professor with the State Key Laboratory of
Information Security, the Institute of Information Engineering, the Chinese
Academy of Sciences. His current research interests include the field of
cyber security including biometric security, bio-cryptography, intrusion detec-
tion, and applied cryptography, where he has published many papers in top
journals, including the IEEE T RANSACTIONS ON PATTERN A NALYSIS AND
M ACHINE I NTELLIGENCE, the IEEE T RANSACTIONS ON C OMPUTERS, the
IEEE T RANSACTIONS ON PARALLEL AND D ISTRIBUTED S YSTEMS, and
the IEEE T RANSACTIONS ON I NFORMATION F ORENSICS AND S ECURITY.
He has served in the Editorial Board of up to seven international journals,
including the IEEE T RANSACTIONS ON I NFORMATION F ORENSICS AND
S ECURITY and served as the Security Symposium Chair of the IEEE flag-
ship conferences of the IEEE ICC and IEEE Globecom. He has obtained
seven Australian Research Council (ARC) Grants and has served at the pres-
tigious Panel of Mathematics, Information, and Computing Sciences, ARC
the Excellence in Research for Australia (ERA) Evaluation Committee 2012.
He is the invited expert of Australia Attorney-Generals Office.
Athanasios V. Vasilakos is currently a Professor
with the Lulea University of Technology, Sweden.
He served or is serving as an Editor for
many technical journals, such as the IEEE
T RANSACTIONS ON N ETWORK AND S ERVICE
M ANAGEMENT, the IEEE T RANSACTIONS ON
C LOUD C OMPUTING, the IEEE T RANSACTIONS
ON I NFORMATION F ORENSICS AND S ECURITY , the
IEEE T RANSACTIONS ON C YBERNETICS, the IEEE
T RANSACTIONS ON NANOBIOSCIENCE, the IEEE
T RANSACTIONS ON I NFORMATION T ECHNOLOGY
IN B IOMEDICINE , ACM Transactions on Autonomous and Adaptive Systems,
and the IEEE J OURNAL ON S ELECTED A REAS IN C OMMUNICATIONS. He is
also the General Chair of the European Alliances for Innovation.