IEEE INTERNET OF THINGS JOURNAL, VOL. 5, NO.
2, APRIL 2018
Toward Unified Control of Networks of
Switches and Sensors Through a
Network Operating System
Angelos-Christos G. Anadiotis, Member, IEEE, Sebastiano Milardo , Giacomo Morabito ,
and Sergio Palazzo, Senior Member, IEEE
Abstract—Wireless sensor networks (WSNs) are an integral
part of an Internet of Things (IoT) ecosystem. In fact, a typi-
cal IoT service deployment considers an integrated network of
switches and sensors (as well as actuators). In order to manage
this heterogeneous network infrastructure, the use of software
defined networking (SDN) technologies is anticipated. However,
existing SDN-enabling technologies consider only isolated net-
works and there has been no concrete solution that treats the
whole network in a unified way. In this paper, a network oper-
ating system is proposed as a way to address the above issue.
Starting from the open network operating system, which cur-
rently supports only OpenFlow networks, the proposed solution
leverages SDN-WISE to integrate WSNs and achieves unified sys-
tem representation and treatment. The proposed system has been
prototyped and is publicly available in open source. Experiments
have been executed demonstrating that, by exploiting the pro-
posed solution, it is also possible to improve efficiency of the
overall network.
Index Terms—Internet of Things (IoT), software defined
networking (SDN), wireless sensor networks (WSNs).
I. I NTRODUCTION
ARKET forecasts released by different institutions
M anticipate the deployment of larger and larger num-
bers of Internet of Things (IoT) devices in the near future.
For example, the most recent study conducted by IDC for
Intel and the United Nations forecasts that there will be
around 200 billion IoT devices by 2020.1 As a result of
these high expectations industry is investing huge resources
in the development and deployment of IoT technologies and
Manuscript received January 31, 2017; revised August 31, 2017; accepted
January 26, 2018. Date of publication February 12, 2018; date of cur-
rent version April 10, 2018. This work was supported by the European
Commission in the framework of the In-Network Programmability for Next-
Generation Personal Cloud Service Support Project through the Horizon
2020 Program (Call H2020-ICT-2014-1) under Grant 644672. (Corresponding
author: Giacomo Morabito.)
A.-C. G. Anadiotis was with CNIT UdR Catania, 95125 Catania, Italy. He
is now with the Data-Intensive Applications and Systems Laboratory, École
Polytechnique Fédérale de Lausanne, 1015 Lausanne, Switzerland (e-mail:
angelos.anadiotis@epfl.ch).
S. Milardo is with the Dipartimento Energia, Ingegneria dell’Informazione
e Modelli Matematici, University of Palermo, 90133 Palermo, Italy (e-mail:
sebastiano.milardo@unipa.it).
G. Morabito and S. Palazzo are with the Dipartimento di Ingegneria
Elettrica Elettronica e Informatica, University of Catania, 95125 Catania, Italy
(e-mail: giacomo.morabito@dieei.unict.it; sergio.palazzo@dieei.unict.it).
Digital Object Identifier 10.1109/JIOT.2018.2805191
1 [Online]. Available: http://www.intel.com/content/www/us/en/internet-of-
things/infographics/guide-to-iot.html
2327-4662 c 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
895
platforms. The academic research community is following
(and in certain cases riding) this wave and today IoT is one
of the hottest research topic in computer science, whereas it
is also spreading to other domains, such as transportation,
production and manufacturing, as well as environmental and
social sciences.
Fig. 1 depicts a standard IoT ecosystem, where cloud ser-
vices, such as SVC-1, SVC-2, and SVC-N, receive data from
networks of sensors through a transport network of switches.
More specifically, typical IoT deployments are either per-
formed vertically, i.e., along with the supporting infrastructure,
or leverage data originally generated by wireless sensor net-
works (WSNs) and finally stored in a data center. In the first
case, the service provider has full control over the devices,
but with increased cost and operational complexity. The cost
is significantly lower in the second case which, furthermore,
perfectly fits the most widely used WSN protocol stack, such
as 6LoWPAN and ZigBee. In fact, the above protocol stacks
are perfectly designed for the case in which sensor nodes for-
ward data toward their sink(s). However, processing in the
cloud causes loss of context, as it cannot be unambiguously
described in metadata, and real-time processing cannot be
efficiently supported.
The fragmentation of the IoT landscape, due to the
deployment of many Intranets of Things, which cannot
cooperate effectively and efficiently, instead of a unique
IoT [1], sets interoperability constraints despite the integration
efforts [2], [3].
Software defined networking (SDN) solutions for WSNs
come to question this way of operation, though. Indeed, by
leveraging SDN technologies, it has now become feasible to
control the infrastructure without having to deploy specific
solutions.
Even though the SDN-driven approach is definitely a step
toward the vision of an easily (re-)configurable IoT environ-
ment, as also outlined in [4] and [5], there is still a major issue
that must be addressed. More specifically, traditional SDN
solutions focus on device-level protocols, thereby providing
isolated views of the individual network segments consisting
an IoT ecosystem. In fact, considering for example that the net-
work of switches of Fig. 1 supports OpenFlow [6] and that the
WSN supports some specialized SDN solution such as SDN-
WISE [7] or Sensor OpenFlow [8], there will be two different
SDN controllers managing each network segment. Bearing in
896
Fig. 1. Typical IoT ecosystem.
mind the large number of network segments involved in a
standard IoT ecosystem, it is obvious that, compared to the
existing application-level integration solutions, this approach
introduces very high complexity and therefore, does not scale.
Actually, a similar problem arose in the first days of SDN
even for fixed networks, where several controllers existed, each
one optimized for different scenarios. In order to tackle with
this issue, the traditional SDN controllers evolved to become
network operating systems (NOSs), which provide high-level
abstractions for several heterogeneous devices and protocols
and can be used to manage even large networks. The open
NOS (ONOS), depicted in Fig. 1, is an example of such a sys-
tem. Unfortunately, NOSs have not been sufficiently exploited
in the context of WSNs, furthermore, no attention has been
paid to the network-level integration of IoT devices.
In this paper, we leverage the developments in NOSs and
we specify a unified system, which controls an IoT ecosystem
at the network layer in a holistic way. More specifically, our
contributions are as follows.
1) We provide generic abstractions for sensor nodes as well
as encapsulations of non-IP packets, in order to enable
IoT service providers to deploy their network configu-
ration over any allocated infrastructure, transparently.
2) We propose an architecture enabling the IoT service
providers to create networks of sensors and switches
on the fly, while supporting packet forwarding through
different network segments.
3) We develop a framework which supports a shift in the
IoT communication paradigm from device-to-cloud to
device-to-device, allowing the direct communication of
sensor nodes over networks of switches. In this way,
clouds of sensors can be created in an ad-hoc manner,
exploiting subsets of the already deployed WSN devices,
thereby fostering granularity and reusability.
4) We introduce the software sensor nodes (SSNs) con-
cept which acts as an integration element for different
IoT protocol stacks so enabling the communication of
devices adopting different standards. This is achieved by
IEEE INTERNET OF THINGS JOURNAL, VOL. 5, NO. 2, APRIL 2018
exploiting both the extensible nature of the NOS, which
allows the implementation of any device-level proto-
col, and the common abstractions for sensors and rules,
which are enforced, irrespectively of the underlying
protocol.
Note that in this paper we are not proposing a new proto-
col (or set of protocols) which should replace existing ones,
like ZigBee and 6LoWPAN. Instead, we are proposing a gen-
eral approach that leverages the existence of SDN solutions
in both the infrastructured and infrastructureless segments of
the IoT to provide application developers with abstractions of
the whole network and the related services that can be easily
utilized in efficient and flexible manner.
In this context, we have focused on SDN-WISE for the
infrastructureless segment because it is stateful and, as a con-
sequence, Turing complete. Therefore, it can implement any
protocol if correctly instructed by the corresponding network
application. For example, we have implemented a network
application that programs the sensor nodes so that they exe-
cute geographical routing [9] and we are implementing another
network application that enables nodes to interpret and gen-
erate typical 6LoWPAN signaling messages, such as router
solicitation, router advertisement, node registration, and node
confirmation so that interaction with standard 6LoWPAN
nodes is possible.
As a whole the proposed solution has several advantages for
all actors involved in the IoT ecosystem. In fact, by exploit-
ing the overall view of the network topology it is possible
to establish the optimal communication path according to the
application requirements. Considering for example the net-
work of Fig. 1, it is typically more efficient for nodes n2 and n6
to communicate through the switches than through the WSN,
which would be the standard solution. This is valuable for final
users (they can obtain higher levels of QoS) and network oper-
ators (they can increase resource efficiency). Furthermore, by
leveraging the virtualization/abstractions of physical resources
it is possible for application developers to focus on the imple-
mentation of the application, ignoring the specific low level
details of the physical objects and their interconnections. This
would involve significant reduction in the development cost
and time to market. Finally, by leveraging the possibility to
program the behavior of nodes through network applications
running on top of the NOS it becomes possible to have IoT
nodes that, in principle, can execute any existing or novel IoT
protocol. It is even possible to have IoT nodes that handle
packets belonging to different applications according to dif-
ferent protocols. This very high level of flexibility can be
exploited.
1) Integrate IoT platforms running different IoT solutions,
so contributing to solve the IoT fragmentation problem,
and making it possible to implement and deploy appli-
cations that exploit the services offered by a larger pool
of IoT resources.
2) Improve the quality of service offered to applications
and increase efficiency. This is possible because it is
well known fact that there is no one-fits-all solution in
the IoT context, and by exploiting the proposed approach
any application can rely on the most appropriate set
ANADIOTIS et al.: TOWARD UNIFIED CONTROL OF NETWORKS OF SWITCHES AND SENSORS THROUGH NOS
of protocols (which should be implemented as network
application on top of the NOS).
The individual components of the proposed architecture
have been previously described in [10]. The main goal of this
paper is to propose an architecture which accommodates the
above components and explain how they interact in order to
achieve the aforementioned contributions. Also, in this paper,
the overall system is assessed in an application scenario involv-
ing the use of MapReduce to process the data produced by IoT
devices.
In order to maximize compatibility with relevant standards,
major requirement we considered in this paper has been
to minimize the number of new abstractions needed in the
ONOS framework and avoid any need for modifications of
the OpenFlow (and other) standard(s).
The rest of this paper is organized as follows. Section II out-
lines related work in the area of SDN protocols and existing
NOS-based solutions for the IoT. Then, Section III provides
a general view of the overall system architecture and sum-
marizes the motivation for the proposed system. Section IV
introduces the concept of the SSN, which is used to integrate
WSNs and networks of switches. The fundamental opera-
tions offered by the proposed system, namely sensor node
registration and integrated packet forwarding are explained in
Sections V and VI, respectively. A MapReduce-based use case
for the system is presented in Section VII and it is used to
extract performance metrics, which highlight measurable ben-
efits of the proposed approach. Finally, the conclusions are
drawn in Section VIII.
II. BACKGROUND AND R ELATED W ORK
A. SDN Protocols
SDN has been envisioned as a way to reduce complexity
and increase flexibility of network configuration and manage-
ment [6]. In SDN networks management operations (control
plane) are logically centralized and physically decoupled from
forwarding operations (data plane), so allowing to easily
update the behavior of the network.
The most popular and widely used SDN solution for wired
networks is OpenFlow [6]. OpenFlow success mainly stems
from its standardized interfaces, which can work on networks
made of heterogeneous switches.
Despite the vast adoption of OpenFlow, and SDN in gen-
eral, in the wired domain, there has been no such widely
accepted solution in wireless networks and, especially, WSNs.
Early solutions such as [11] and [12] have been outperformed
by recent ones, either based on TinyOS [13] or Contiki [7].
In the context of this paper, we consider SDN-WISE [7],
since it is open source and provides a software prototype
of an SDN-enabled sensor node, which has been extensively
used in practice [14], which can be easily integrated into any
emulation framework.
B. SDN Controllers
At the control plane, research has evolved from sim-
ple services, called controllers, to more sophisticated solu-
tions that can efficiently manage large scale systems, the
897
NOSs. Recently, two open source NOSs have been pro-
posed: 1) OpenDaylight [15] and 2) ONOS [16]. OpenDaylight
provides a project, IoTDM, which focuses on the IoT, but
it follows the data-level integration approach, rather than a
network-oriented solution, which would actually complement
existing solutions for networks of switches.2 More specif-
ically, IoTDM does not enable the direct communication
among devices, whereas it relies on application-level pro-
tocols, such as CoAP,3 in order to integrate heterogeneous
devices. Apart from being a hybrid, rather than a pure network-
level approach, IoTDM considers only one of the several
existing standards and as has been well documented [2], [4],
much effort is still required in order to settle to a particular
standard for the IoT. Even though the architectural princi-
ples between ONOS and OpenDaylight are similar, in this
paper, we build our prototype leveraging ONOS, currently sup-
porting only OpenFlow networks, by extending it to support
SDN-WISE enabled WSNs.
III. M OTIVATION AND A RCHITECTURE
The key motivation of this paper is the realization of a
unified system which enables the communication among het-
erogeneous devices in order to create IoT ecosystems on the
fly, exploiting the infrastructure already in place. Following
a pure network-level approach, the proposed system does not
depend on any particular IoT architecture. This is achieved by
the following.
1) Employing SDN technologies in both the transport and
the sensor networks.
2) Specifying generic, protocol-agnostic abstractions for
sensor nodes and network packets.
3) Introducing the concept of the SSN, an SDN-enabled
sensor node, which can be easily integrated with any
existing WSN deployment, by implementing the corre-
sponding protocol procedures inside the NOS.
Fig. 2 depicts the extended ONOS architecture, where
the dark-colored components are required to integrate soft-
ware defined sensor nodes [10]; whereas the light colored
components have been kept back in their original ONOS
implementation.
The proposed architecture follows the logical separation
of ONOS into subsystems. The SensorNode Subsystem pro-
vides the abstractions as well as an up-to-date registry of the
sensor nodes connected to the system. More specifically, the
SensorNode API is a generic, extensible, API, which specifies
a generic sensor node, so that it can be efficiently accessed
from the other layers of the NOS. The SensorNode API is used
for both sending information, such as forwarding rules, to the
sensor nodes as well as receiving information, like the battery
level and their neighbors, from them. The API is extensible,
so that application-specific sensors can also be easily inte-
grated. The SensorNode Manager implements the SensorNode
API functionality. Essentially, it is a registry for all the sensor
nodes managed by the NOS by maintaining up-to-date infor-
mation either for logging reasons (e.g., forwarding rules) or
2 [Online]. Available: https://wiki.opendaylight.org/view/IoTDM_Overview
3 [Online]. Available: https://tools.ietf.org/html/rfc7252
898
Fig. 2. ONOS integrated architecture.
for status maintenance (e.g., battery level). The SensorNode
Provider translates protocol-specific data to data that can be
accessed through the SensorNode API and vice-versa. More
specifically, incoming data are decoded into ONOS-specific
APIs, so that they can be effectively managed by the higher
layers. Outgoing data are encoded into protocol-specific APIs,
by leveraging the Protocols layer.
The SensorFlowRule Subsystem is used to create and install
flow rules, by leveraging the protocol supported by the tar-
get device. The SensorFlowRule API specifies the high-level
semantics for creating rules that are sensor-node specific and
can look at any portion of the packets to classify them into
flows. This is supported by SDN-WISE and enables a high
degree of flexibility which goes well beyond what OpenFlow
can obtain. The SensorFlowRule Provider is used to encode
flow rules, by leveraging the corresponding protocol, so that
they can be sent to the sensor nodes. Let us note here that the
FlowRule Manager and the FlowRuleProvider API have been
kept unchanged with respect to their original ONOS imple-
mentation. This has been achieved by exploiting the high-level
semantics of ONOS combined with the generic design of the
FlowRule API.
The SensorPacket Subsystem handles incoming and out-
going packets from/to the sensor nodes, by using a non-
IP, generic packet encapsulation API. In particular, the
SensorPacket API provides access to incoming and outgoing
packets from/to sensor nodes. Even though the network-level
packet format depends on the corresponding underlying net-
work protocol, the SensorPacket Provider encapsulates all
this information into ONOS high-level abstractions. The cur-
rent API has been designed by considering requirements of
generic sensor nodes and provides access to information such
as host and destination address, protocol version and other
low-level details. However, this API can be extended to sup-
port application-specific protocols by building on top of the
existing system.
In the protocols layer, the specific semantics of the under-
lying network protocol are implemented. More specifically,
IEEE INTERNET OF THINGS JOURNAL, VOL. 5, NO. 2, APRIL 2018
the proposed architecture includes the server-side SDN-WISE
implementation, which is responsible for communicating with
the sensor nodes and provides access to low-level proto-
col semantics. This is achieved through a protocol-specific
API, with respect to the already existing one in ONOS
for OpenFlow, which is able to encode/decode information
coming from the WSN/NOS.
Observe that in the APP layer, ONOS supports the imple-
mentation of network applications. In fact, these applications
are responsible for creating the rules, which are then installed
in the nodes by leveraging the corresponding subsystems.
When a sensor node, which is connected to ONOS, receives a
packet that it cannot handle, it attaches this packet to a request
and sends it to ONOS. ONOS decapsulates the various packet
layers as described previously in the context of the subsys-
tems and, in case it cannot directly be handled by them (e.g.,
it is not a trivial beacon packet), it is delivered to the net-
work application associated with this node. Then, the network
application decides how to handle this packet. Note here that
a network application essentially implements a network pro-
tocol. Therefore, even existing protocols, such as 6LoWPAN
or ZigBee can, in principle, be implemented in ONOS. For
instance, if an SDN-WISE node receives a 6LoWPAN packet,
it will encapsulate it in an SDN-WISE packet by inserting
it in the payload. Then, the network application will decode
that payload, it will understand that it is a 6LoWPAN packet
and will send the corresponding reply packet to the sensor
node.
Given the heterogeneity of the supported devices (sensor
nodes and switches), ONOS has to associate each device type
with a set of subsystems in order to handle them accordingly.
This is achieved by adopting system-wide naming schemes
in the form of uniform resource identifiers for every com-
ponent. For example, an OpenFlow switch is represented as
of:000000001 and an SDN-WISE node as sdnwise:00000101.
In this way, every device connected to the system as well
as every device-specific software module are registered using
the same prefix. As a result, ONOS can deliver packets and
ANADIOTIS et al.: TOWARD UNIFIED CONTROL OF NETWORKS OF SWITCHES AND SENSORS THROUGH NOS
Fig. 3. Heterogeneous network integration with SSN.
events coming from the devices to the respective subsystems
and viceversa, transparently.
Despite the complexity and the variety of operations pro-
vided by the NOS, in this paper, we focus on a fundamental
subset, in order to illustrate how the proposed solution achieves
the contributions outlined at the beginning of this section are
accomplished. More specifically, in the following, we first
introduce the concept of the SSN and then, we explain the
procedure implemented for registering and representing a sen-
sor node in the system. Then, we illustrate how different sensor
network applications can register their packet types to the sys-
tem, so that they can efficiently make use of the corresponding
flow rules capabilities. Then, we present the integrated for-
warding network application, which essentially bridges the gap
between networks of switches and sensors by leveraging both
existing and newly introduced NOS components.
IV. S OFTWARE S ENSOR N ODE
The fundamental component introduced at the device level
for network-layer integration is the SSN. SSNs are essentially
SDN-WISE-enabled nodes, which do not necessarily corre-
spond to a physical device, with respect to the concept of
OVSwitch for OpenFlow networks. In fact, such nodes may
as well be deployed as services in the network, e.g., using
network function virtualization, by leveraging the SDN-WISE
node implementation,4 which has already been used for WSNs
emulation.
Due to their programmability stemming from their SDN
support, SSNs can be easily integrated with any IoT proto-
col, by just implementing the corresponding APIs as an NOS
application. This way, an SSN can be virtually attached to
any WSN by just sending the corresponding signaling mes-
sages to the nodes it has to connect to. For example, in Fig. 3,
suppose that nodes n1 and n2 belong to a 6LoWPAN net-
work, whereas nodes n3 and n4 belong to a ZigBee network.
4 [Online]. Available: https://github.com/sdnwiselab
899
Then, the SSN, can send the appropriate neighborhood dis-
covery packets to n1 and n4 , through the respective sinks n2
and n3 , and become neighbor with both them. Furthermore, it
can also exchange neighborhood information with both these
nodes, thereby creating a symmetric link n1 −SSN−n4 , which
in fact hinders the path of the intermediate sinks and switches,
giving the impression to the sensor nodes that they have two
hops distance.
This way, every SSN actually represents the set of sensor
nodes of a particular WSN that are connected to it. Therefore,
the overall network topology can be abstracted as network of
SDN-enabled switches and sensors. Moreover, dynamic ad-
hoc networks can be created by connecting individual nodes
over an heterogeneous infrastructure, consisting of switches
and SDN-WISE sensor nodes, either physical or virtual (soft-
ware) ones. As a result, transferable network configurations
can be enforced on the fly to already existing WSNs, with
respect to the IoT vision presented in [4], by controlling the
whole infrastructure in a holistic way.
The proposed NOS performs holistic management of the
networks of switches and sensors by introducing generic
abstractions for sensor nodes and packets and by performing
integrated packet forwarding transparently across these net-
works. Note that the implementation of 6LoWPAN and other
WSN protocols in the NOS has not been prototyped in the
context of this paper, since it mostly involves development
work, whereas it does not add any complexity in the overall
architecture; eventually, the challenge is for SDN-WISE nodes
to communicate over OpenFlow switches.
V. S ENSOR N ODE R EGISTRATION AND R EPRESENTATION
A key requirement of the proposed system is the unified
representation of the whole network, consisting of both sensor
nodes and switches. This is achieved by introducing new and
extending existing core ONOS abstractions, which provide fine
grained access to the network components, namely the devices
and the links. More specifically, two access layers are consid-
ered for devices and links, the base and the enhanced layer.
The base layer is used to create an internal graph representa-
tion of the network topology, where every node is represented
by a Device and every edge by a Link. The enhanced layer
for the sensor nodes, SensorNode, extends the Device
and it is used to access implementation-specific information,
such as the energy level of a sensor, its geographic coordi-
nates and its neighborhood. Whereas, the enhanced layer for
the sensor links, SensorLink, extends the Link to provide
information on the received signal strength indicator (RSSI)
of a wireless link, which is typically used as a metric for the
weight of an edge representing a wireless link.
Information provided at the enhanced layer can be further
extended through annotations, implemented as key-value pairs.
Such annotations are typically useful for passing application-
specific parameters to ONOS, which are required to make
network-layer decisions. For example, consider a WSN that
contains two types of sensor nodes, temperature and pressure
and assigns a different aggregator to each type. In order to
forward the packets to the corresponding aggregator, ONOS
900
(a)
(b)
Fig. 4. Representation of the (a) sink node in ONOS and (b) its protocol
stack.
needs to be aware of the sensor type each node carries. This
information cannot be contained in any network packet header,
though. Therefore, through the annotations, such information
can be passed to the system from an external service, rather
than the network, so that it can be later used for taking
networking decisions, such as packet forwarding.
The registration of SDN-WISE nodes is quite straight-
forward and is achieved through the SDN-WISE protocol
mechanisms by leveraging the SensorLink abstraction pro-
vided by the proposed system. Links between sensors are
periodically reported to ONOS through the REPORT packets
issued by every sensor and containing the list of its neighbors
and the associated RSSI. Links between sensors (sinks) and
switches are reported through the DP_CONNECTION packets,
which contain the MAC address of the sink nodes.
In fact, as shown in Fig. 4(a) our extension of ONOS repre-
sents sinks both as sensors (for the wireless link) and as edge
hosts (for the wired link), because they typically carry two
different types of network interfaces. This design approach is
the consequence of the two major requirements: 1) to avoid
any change in the standard OpenFlow and 2) minimize the
number of novel abstractions to be introduced in ONOS. To
this purpose sink nodes will run the protocol stack represented
in Fig. 4(b).
At the physical and link layers two sets of protocols will be
implemented that are specific of the infrastructured and WSN
segments, such as IEEE 802.3 and IEEE 802.15.4, respec-
tively. SDN-WISE is executed on top of the IEEE 802.15.4
interface; whereas a relay layer is also executed which is
responsible of the operations needed to bridge the two different
network segments; one of its major roles is packet adaptation,
for example.
Finally, links between SDN-WISE nodes and others imple-
menting any other stack (e.g., 6LoWPAN), are established by
leveraging the communication protocol of the standard nodes.
More specifically, SDN-WISE node can be instructed by the
NOS to send the corresponding protocol-specific packets and
IEEE INTERNET OF THINGS JOURNAL, VOL. 5, NO. 2, APRIL 2018
Fig. 5. Sensor node registration flowchart.
then, use the corresponding response to identify the link, in the
same respect as with links between two SDN-WISE nodes. For
example, SDN-WISE nodes can generate and interpret typi-
cal 6LoWPAN signaling messages, such as router solicitation,
router advertisement, node registration, and node confirma-
tion. In this way it is possible for SDN-WISE nodes to build
a network topology overlayed at the 6LoWPAN nodes.
This is a fundamental feature of the proposed solu-
tion as there is already a large number of WSN devices
already deployed worldwide (e.g., according to an ABI market
research 850 million IEEE 802.15.4 devices have been shipped
in 2016).
On the ONOS side, there are two subsystems involved
in the sensor node registration process: 1) the SensorPacket
Subsystem and 2) the SensorNode Subsystem. The flowchart
given in Fig. 5 depicts the role of the SensorPacket Subsystem,
for example. When a REPORT packet arrives, a direct
SensorLink is created between that node and all its neighbors.
In the special case of the DP_CONNECTION, a standard edge
Link is created, so that ONOS treats the sink as the edge of the
network of switches. Links between SDN-WISE and standard
sensor nodes are created in the context of the ONOS appli-
cation implementing the corresponding protocol, based on the
SensorLink abstraction. Note that these are also edge Links, as
the standard nodes cannot accept SDN rules and, thus, they are
treated as regular hosts. After the links have been established,
the network topology is updated accordingly.
ONOS keeps an updated view of the WSN using the
SensorNode Subsystem. More specifically, the SensorNode
Provider checks for sensor nodes connectivity, based on the
reports received by them, whereas it creates a new device
upon a sensor node arrival to the system. At the high level,
the SensorNodeManager introduces functionality to store and
access fundamental information on the nodes, such as their
address, their battery level, the sink they are connected to
or, in case the node is a sink itself, the switch that it is
ANADIOTIS et al.: TOWARD UNIFIED CONTROL OF NETWORKS OF SWITCHES AND SENSORS THROUGH NOS 901

attached to. Note that this information is either directly com-

ing or implicitly derived from the REPORT packets. Sensor
nodes are abstracted by the SensorNode data structure, which
extends the device, thus directly enabling its integration with
the topology, while providing the abstractions described ear-
lier in this section. All information regarding sensor nodes are
currently stored in hash tables. However, in case the networks
grow larger, more sophisticated and highly efficient in-memory
databases can be leveraged (e.g., Hekaton, VoltDB, etc.), in
order to address any scalability issue, without any changes to
the existing API.

VI. N ETWORK -W IDE PACKET R EPRESENTATION

AND F ORWARDING
In our solution, the packets sent by sensor nodes to ONOS,
in order to obtain a corresponding rule, are delivered to
the Integrated Forwarding application. The latter extracts the
source and the destination and calculates the shortest path con-
sidering the whole topology.5 Wireless link cost is calculated
based on the link RSSI and is dynamically passed to the path
extraction algorithm. Accordingly, in the example of Fig. 1, a
packet from node n1 to node n5 will follow the path includ-
ing switches, since it contains fewer wireless links, which are
most costly. Whereas, a standard controller would suggest the
path of sensor nodes only, since it would not have a full view
of the topology.
Unfortunately, packets coming from sensor nodes have dif-
ferent format than the standard IP packets, sent by hosts
and are currently supported by ONOS. Accordingly, we have
introduced a new API, the SensorPacket API, on top of the
existing PacketManager, as shown in Fig. 2, in order to enable
ONOS to deal with the new packet formats. SensorPacket API
includes the PacketType, which represents sensor packet types
similarly to the type of service of IP packets. The main novelty
of this API is that it allows different network applications to
register their own packet types, without interfering with each
other, thereby fostering reuse of existing packet headers with
application decoupling.
Once ONOS can deal with packets generated by sensors,
then, in order to achieve integrated forwarding we need to
define how ONOS will instruct the network nodes in such a
way that the traffic flow can traverse sequences of switches
and sensors, transparently. A major problem in this context is
the passage between one segment of switches and one segment
of sensors (or viceversa), since they use different instruction
sets. In our solution, when a packet arrives to the system,
the SensorPacket Provider checks its type and, if it is not
used for signaling, it encapsulates it into an ONOS-specific
packet format that can be accessed by the SensorPacket API.
Accordingly, incoming packets are transformed to Ethernet
frames, using ONOS core APIs. Then, they are delivered to
the PacketProcessors, which are typically implemented in the
application layer.
5 Note that, even though not explicitly studied, multicast can also be sup-
ported, since it can be approached as a set of multiple source, destination
pairs.
Fig. 6. Integrated forwarding flowchart.
Fig. 6 describes the steps followed by the Integrated
Forwarding application when it receives a packet (typically
from the Packet Provider). After the whole path from source
to destination has been calculated, it is broken into individual
segments, which separate the sequence of switches and sensor
nodes, as described in the flowchart of Fig. 6. Each segment
is represented as a set of links connecting only devices of the
same type. Accordingly, every time a link connecting devices
of different type is met, the current segment is closed and a
new one is opened.
For example, considering that node n1 of Fig. 1 sends a
packet with destination node n5 and that ONOS calculates as
shortest path the one passing through the OpenFlow switches,
Integrated Forwarding will split the path into two different
segments. More specifically, when ONOS receives the packet,
it triggers the process described in the flowchart of Fig. 6.
After the path is calculated, the first link is considered and
switch s1 is added to the segment of switches, by entering
the flowchart parts 1, 2, 5, and 6. Then, the switches s2 and
s3 are also added to the same segment by entering parts 1,
2, and 5. When the link s1 → n7 is visited, the segment of
switches is closed, by executing the operations shown in parts
1, 2, and 4. Observe that ONOS understands n7 as host h2 in
this case, since it is connected to switch s3 . Finally, for each
one of the two remaining sensor links from n7 to n5 , parts
1, 2, and 3 are considered. After that step, there are no more
links in the path and hence, the active sensor node segment is
closed as well and the process ends.
For each generated segment, the appropriate flow rule sub-
system is used in order to specify the forwarding rules.
More specifically, if the request is made by a sensor, the
902
Fig. 7. Network topology with different reducers for each case.
SensorFlowRule Subsystem is used, whereas if it is made by
a switch, the OpenFlowRule API is leveraged.
VII. C ASE S TUDY—I N -N ETWORK P ROCESSING
W ITH M AP R EDUCE
As we have already discussed, the proposed solution has
several advantages ranging from very high flexibility to the
possibility for developers to implement IoT services and appli-
cations ignoring the specific low level details of the physical
objects and their interconnections. In this section, however,
we will focus on the performance improvement that can
be achieved by exploiting the overall view of the network
topology, including both infrastructured and infrastructureless
segments.
To this purpose we will compare the performance obtained
by the proposed approach to the performance obtained when
the infrastructured segments of the network implement a
shortest path routing whereas the infrastructureless segments
implement RPL,6 which is the most widely deployed routing
protocol for 6LoWPAN networks [17].
More specifically, the effectiveness of the proposed system
is assessed by considering the case, where MapReduce opera-
tions are executed inside the WSN network and the results are
sent to a service deployed in a host outside the WSN, as sug-
gested in [18]. For example, considering the system of Fig. 1,
suppose that the sensor nodes are executing MapReduce oper-
ations and then, the results are sent to service SVC-1, deployed
in the cloud, by sending the data over the OpenFlow network.
Let us explicitly observe that in-network processing for big
data applications has already been successfully applied in data
centers [19], [20]. In the use case addressed here, we shift this
paradigm to in-network processing for sensor networks.
6 Although a large number of RPL improvements have been proposed in the
recent past, we have chosen the standard RPL implementation because we are
focusing on the advantages obtained by exploiting the complete and integral
view of the overall network rather than the possibility to select shorter paths
inside the infrastructureless segment.
IEEE INTERNET OF THINGS JOURNAL, VOL. 5, NO. 2, APRIL 2018
We consider a WSN of 23 nodes connected with a transport
network of six switches, as depicted in Fig. 7. Each sen-
sor node implements the SDN-WISE protocol, whereas each
switch implements OpenFlow. Switches and sensor nodes are
deployed in Mininet, which has been extended to support the
SDN-WISE SSNs.
Sensor nodes are equipped with sensors of four different
types: 1) temperature; 2) humidity; 3) pressure; and 4) noise.
Values generated by each of these types are associated with
the corresponding key in the context of the map function.
According to MapReduce, there has to be exactly one node,
namely the reducer, which collects all values associated
with a single key. The selection of the reducer for each key
is made as explained in [18] and is outside the scope of this
paper.
In order to select the appropriate reducer for each key, it
is essential to know the sensor types that every node car-
ries, which is not included in the signaling messages. This
is achieved in the application layer by leveraging the exten-
sibility of the SensorNode API, based on the inherent ONOS
Annotations API. This enables the update of the nodes descrip-
tion by inserting the corresponding sensor types as key-value
pairs of the form sensor, sensor type.
In our experiments, every node carries a random number of
sensor types. Furthermore, a map function is loaded in each
node by ONOS, which periodically generates key-value pairs
for each sensor type and pushes them to the network layer.
At the first execution, the node does not know how to treat
the packet generated by the map function and therefore, it
makes a request to the NOS. The latter selects the reducer
and, through Integrated Forwarding application, installs the
appropriate rules to every device, as described in the previous
section.
We have implemented a prototype of the proposed solution
for TI CC2530 devices as well as the software modules that
extend Mininet to consider sensor nodes also. Our prototype
has been released in open source and can be downloaded at
the following URLs.
1) https://github.com/sdnwiselab/onos (source code of
ONOS).
2) https://github.com/sdnwiselab/sdn-wise-cc2530 (source
code for the TI CC2530 devices).
3) https://github.com/sdnwiselab/onos-sdn-wise-app-
samples/tree/master/mapreduce (source code of the
MapReduce application and the Mininet scripts).
Since we have a small number of sensors, the results shown
in this paper have been obtained by using the Mininet sim-
ulator. In our experiments energy related metrics have been
calculated by elaborating the logs of the simulation consid-
ering the information about battery and energy consumption
provided in the datasheets of the TI CC2530.
The use of Integrated Forwarding first affects the selection
of the reducers, which is made considering the cost of commu-
nication among the nodes, derived from the forwarding path.
In particular, as shown in Fig. 7, in the standard WSN case, the
node in the center of the network is selected to be a reducer.
On the other hand, when Integrated Forwarding is enabled, the
optimal reducer is one of the sinks (node 1 in this case). In
ANADIOTIS et al.: TOWARD UNIFIED CONTROL OF NETWORKS OF SWITCHES AND SENSORS THROUGH NOS
(a)
(c)
Fig. 8. Nodes resource usage. (a) Remaining battery CDF. (b) delay. (c) nodes traffic CDF. (d) Rules CDF.
order to have a common point of reference, the evaluation will
be made considering that the reducer will be node 1 in both
the standard WSN and the integrated forwarding approach,
since the goal is to study the communication aspects of the
proposed architecture and not the MapReduce implementation
as a whole.
The use of sensor nodes resources after executing the experi-
ment for 10 min is shown in Fig. 8. More specifically, Fig. 8(a)
shows the CDF of the remaining energy level of the sensor
nodes after the experiment is over.
As shown in the figure, in the integrated forwarding case,
there is always a larger fraction of nodes with more residual
battery than in the typical one. Moreover, since the difference
increases as the energy levels become lower, the network life-
time is expected to be significantly higher when using the
proposed approach.
Furthermore, as shown in Fig. 8(b), in case the network of
switches is used, i.e., when nodes 18, 19, 21, 22, and 23 are
sending data to node 1 (reducer), the delivery times are sig-
nificantly smaller than in the typical WSN-specific approach.
The reason is that the core network has much larger capacity,
and therefore packets are forwarded much faster than in the
context of the WSN.
The effectiveness of the use of the core network also
becomes clear by considering Fig. 8(c), which shows the CDF
of the overall number of packets sent and received by the
903
(b)
(d)
sensor nodes in the plain WSN and the integrated forwarding
approach. Overall, the integrated forwarding approach results
in less communication between the nodes, which is essential
for WSNs and is the key factor for the reduction on the energy
consumption shown in Fig. 8(a).
Finally, the proposed integrated forwarding approach results
in less flow rules in the nodes, as depicted in Fig. 8(d). This
means less signaling messages and lower memory occupancy
for the nodes, which is very important given the strict limi-
tations on both communication and hardware resources. The
reduction of flow rules in the integrated forwarding approach
is the result of the use of the core network. Indeed, when
using the WSN alone to forward packets, each intermediate
node requires 1 rule per destination. Therefore, for each path
of N hops, which traverses the core network, a reduction of N
rules at most (e.g., in sink-to-sink communication) is achieved
for the WSN.
VIII. C ONCLUSION
This paper presents a network operating system which
is used to integrate networks of switches with networks
of sensors, in view of the IoT. Starting from ONOS, an
existing solution for OpenFlow networks, we have intro-
duced new components, which efficiently abstract both the
network elements and their operations. Moreover, we have
explained how the proposed extensions can work together with
904
existing modules, eventually enabling ONOS to perform holis-
tic network management and integrated packet forwarding in
software defined transport and WSNs.
The proposed system has been prototyped, whereas its eval-
uation is performed in the context of a novel in-network
packet processing approach, which enables the execution of
MapReduce operations in WSNs.
R EFERENCES
[1] M. Zorzi, A. Gluhak, S. Lange, and A. Bassi, “From today’s INTRAnet
of things to a future INTERnet of Things: A wireless-and mobility-
related view,” IEEE Wireless Commun., vol. 17, no. 6, pp. 44–51,
Dec. 2010.
[2] A. Al-Fuqaha, A. Khreishah, M. Guizani, A. Rayes, and
M. Mohammadi, “Toward better horizontal integration among IoT
services,” IEEE Commun. Mag., vol. 53, no. 9, pp. 72–79, Sep. 2015.
[3] M. A. Razzaque, M. Milojevic-Jevric, A. Palade, and S. Clarke,
“Middleware for Internet of Things: A survey,” IEEE Internet Things
J., vol. 3, no. 1, pp. 70–95, Feb. 2016.
[4] V. Cerf and M. Senges, “Taking the Internet to the next physical level,”
Computer, vol. 49, no. 2, pp. 80–86, Feb. 2016.
[5] P. D. Dio et al., “Exploiting state information to support QoS in
software-defined WSNs,” in Proc. Mediterr. Ad Hoc Netw. Workshop
(Med-Hoc-Net), Jun. 2016, pp. 1–7.
[6] N. McKeown et al., “OpenFlow: Enabling innovation in campus net-
works,” SIGCOMM Comput. Commun. Rev., vol. 38, no. 2, pp. 69–74,
Mar. 2008.
[7] L. Galluccio, S. Milardo, G. Morabito, and S. Palazzo, “SDN-WISE:
Design, prototyping and experimentation of a stateful SDN solution
for wireless sensor networks,” in Proc. IEEE Conf. Comput. Commun.
(INFOCOM), Apr. 2015, pp. 513–521.
[8] T. Luo, H.-P. Tan, and T. Q. S. Quek, “Sensor OpenFlow: Enabling
software-defined wireless sensor networks,” IEEE Commun. Lett.,
vol. 16, no. 11, pp. 1896–1899, Nov. 2012.
[9] A. C. Anadiotis, L. Galluccio, S. Milardo, G. Morabito, and S. Palazzo,
“Offloading software defined WSNs through distributed geographic for-
warding,” Dipartimento di Ingegneria Elettrica Elettronica e Informatica,
Univ. Catania, Catania, Italy, Rep. 1, Jan. 2017. [Online]. Available:
http://bit.ly/2EpwlKp
[10] A.-C. G. Anadiotis, L. Galluccio, S. Milardo, G. Morabito,
and S. Palazzo, “Towards a software-defined network operat-
ing system for the IoT,” in Proc. IEEE 2nd World Forum
Internet Things (WF-IoT), Milan, Italy, Dec. 2015, pp. 579–584,
doi: 10.1109/WF-IoT.2015.7389118.
[11] A. Mahmud and R. Rahmani, “Exploitation of OpenFlow in wire-
less sensor networks,” in Proc. Int. Conf. Comput. Sci. Netw. Technol.
(ICCSNT), vol. 1. Harbin, China, Dec. 2011, pp. 594–600.
[12] S. Costanzo, L. Galluccio, G. Morabito, and S. Palazzo, “Software
defined wireless networks: Unbridling SDNs,” in Proc. Eur. Workshop
Softw. Defined Netw. (EWSDN), Darmstadt, Germany, Oct. 2012,
pp. 1–6.
[13] B. Trevizan de Oliveira, C. B. Margi, and L. B. Gabriel, “TinySDN:
Enabling multiple controllers for software-defined wireless sensor net-
works,” in Proc. IEEE Latin America Conf. Commun. (LATINCOM),
Nov. 2014, pp. 1–6.
[14] C. Buratti et al., “Testing protocols for the Internet of Things on the
EuWIn platform,” IEEE Internet Things J., vol. 3, no. 1, pp. 124–133,
Feb. 2016.
[15] J. Medved, R. Varga, A. Tkacik, and K. Gray, “OpenDaylight: Towards
a model-driven SDN controller architecture,” in Proc. IEEE 15th Int.
Symp. World Wireless Mobile Multimedia Netw. (WoWMoM), Sydney,
NSW, Australia, Jun. 2014, pp. 1–6.
[16] “ONOS—A SDN network operating system for service providers,”
Menlo Park, CA, USA, ONF and Open Netw. Lab, White Paper, 2014.
[Online]. Available: http://bit.ly/2HE8WpZ
[17] T. Winter et al., “Rpl: Ipv6 routing protocol for low-power and lossy
networks,” IETF, Fremont, CA, USA, RFC 6550, 2013.
[18] A.-C. G. Anadiotis, G. Morabito, and S. Palazzo, “An SDN-assisted
framework for optimal deployment of mapreduce functions in WSNs,”
IEEE Trans. Mobile Comput., vol. 15, no. 9, pp. 2165–2178, Sep. 2016,
doi: 10.1109/TMC.2015.2496582.
[19] L. Mai et al., “Supporting application-specific in-network processing
in data centres,” in Proc. ACM SIGCOMM Conf. (SIGCOMM),
2013, pp. 519–520. [Online]. Available: http://doi.acm.org/
10.1145/2486001.2491733
[20] P. Costa, A. Donnelly, A. Rowstron, and G. O’Shea, “Camdoop:
Exploiting in-network aggregation for big data applications,” in
Proc. 9th USENIX Conf. Netw. Syst. Design Implement. (NSDI), San
Jose, CA, USA, 2012, p. 3. [Online]. Available: http://dl.acm.org/
citation.cfm?id=2228298.2228302
IEEE INTERNET OF THINGS JOURNAL, VOL. 5, NO. 2, APRIL 2018
Angelos-Christos G. Anadiotis (GS’13–M’14) received the Dipl.-Ing. degree
from the Department of Electrical and Computer Engineering, University of
Thessaly, Volos, Greece, in 2008, the M.Sc. degree in advanced information
systems from the Department of Informatics, University of Piraeus, Piraeus,
Greece, in 2011, and the Dr.-Ing. degree from the School of Electrical and
Computer Engineering, NTUA, Athens, Greece, in 2014.
From 2014 to 2015, he was with the University of Catania Research Unit,
Catania, Italy, as a Researcher. Since 2016, he has been a Post-Doctoral
Researcher with the École Polytechnique Fédérale de Lausanne, Lausanne,
Switzerland. His current research interests include large scale distributed
systems, software-defined networks, and data management systems. He has
authored or co-authored several publications and input documents to ISO
standards in the above areas.
Dr. Anadiotis has served as a Project Editor of the second version of
ISO/IEC 23006-3 and a Co-Guest Editor of the Transactions on Emerging
Telecommunication Technologies (Wiley).
Sebastiano Milardo received the bachelor’s and master’s degrees in computer
engineering from the University of Catania, Catania, Italy, in 2011 and 2013,
respectively. He is currently pursuing the Ph.D. degree in information and
communication technologies at the University of Palermo, Palermo, Italy.
From 2014 to 2015, he was with the Italian National Consortium of
Telecommunications, Catania, as a Researcher within the NEWCOM# and
SIGMA Projects. From 2016 to 2017, he was with the Senseable City
Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA,
as a visiting student. His current research interests include network operating
systems, software defined networks, sensor networks, and protocols for the
Internet of Things and Big Data.
Giacomo Morabito received the Laurea degree in electrical engineering and
Ph.D. degree in electrical, computer, and telecommunications engineering
from the Istituto di Informatica e Telecomunicazioni, University of Catania,
Catania, Italy, in 1996 and 2000, respectively.
From 1999 to 2001, he was with the Broadband and Wireless Networking
Laboratory, Georgia Institute of Technology, Atlanta, GA, USA, as a Research
Engineer. Since 2001, he has been with the Dipartimento di Ingegneria
Elettrica Elettronica e Informatica, University of Catania, where he is cur-
rently an Associate Professor. His current research interest includes analysis
and solutions for wireless networks.
Dr. Morabito serves on the Editorial Boards of IEEE Wireless
Communication Magazine, Computer Networks (Elsevier), and Wireless
Networks (ACM/Baltzer). He has been Co-Guest Editor for the IEEE
T RANSACTIONS ON M ULTIMEDIA, Computer Networks (Elsevier), and
Mobile Networks and Applications (ACM/Baltzer).
Sergio Palazzo (M’92–SM’99) received the degree in electrical engineering
from the University of Catania, Catania, Italy, in 1977.
Since 1987, he has been with the University of Catania, where he is cur-
rently a Professor of telecommunications networks. In 1994, he was with the
International Computer Science Institute, Berkeley, CA, USA, as a Senior
Visitor. In 2003, he was with the University of Canterbury, Christchurch,
New Zealand, as a recipient of the Visiting Erskine Fellowship. His current
research interests include modeling, optimization, and control of wireless net-
works, with applications to cognitive and cooperative networking, SDN, and
sensor networks.
Prof. Palazzo has been serving on the Technical Program Committee of
INFOCOM, IEEE Conference on Computer Communications, since 1992.
He has been the General Chair of some ACM conferences (MobiHoc 2006
and MobiOpp 2010), and currently is a member of the MobiHoc Steering
Committee. He has also been the TPC Co-Chair of some other conferences,
including IFIP Networking 2011, IWCMC 2013, and European Wireless 2014.
He also served on the Editorial Board of several journals, including the
IEEE/ACM T RANSACTIONS ON N ETWORKING, the IEEE T RANSACTIONS
ON M OBILE C OMPUTING , IEEE Wireless Communications Magazine,
Computer Networks, Ad Hoc Networks, and Wireless Communications and
Mobile Computing.