Modeling & Simulation of Hardware-in-the-Loop using OPNET Process Models

Ralph Martinez, PhD, Hau Nguyen, MS

Wenji Wu, MS, Dan Bradford, MS, Neil Peterson
Computer Engineering Research Laboratory
Electrical & Computer Engineering Department
The University of Arizona
And the U.S. Army Information Systems Engineering Command
Tucson, AZ 85721
martinez@ece.arizona.edu
Abstract
This paper is a continuation of the Hardware in the Loop
project that was presented to the OPNETWORK 2001
Conference. This paper further describes the design and
implementation of a "Hardware in the Loop (HITL)"
capability within the discrete-event modeling package,
OPNET Modeler. The project objective is to develop a
capability to evaluate Information Infrastructure Architectures
(I3A) for Army communications networks and networkcentric systems.

Task 3.0
Use RTE Traffic
Data to Drive
OPNET Models

Task 1.0
Develop Process Model
Process Model
In
C Code
(Deliverable)
Task 2.0
Develop OPNET
Performance
Measures for
Process Model

Task 4.0
Interface
Process Model to
a VPN

The four main tasks for this project are illustrated in Figure 1.

A HITL capability has been developed that allows the

Information Systems Engineering Command (ISEC) engineers
to insert network components into an OPNET model that
represents an edge device to a sustaining Army base. The
"Hardware in the Loop" is a real network component that is
interfaced with the OPNET models using process models
coded in the C language. The process models translate
OPNET packet data structures into IP packets and vice-versa.
The process models are generic and represent a hierarchy of
network components and network-topology configurations, so
that entire virtual private networks (VPNs) can be built into
the models. The HITL process model is called the Virtual
Gateway.

Task 1.0 involved the development of a process model in

OPNET that enables connection to specific vendor
communication hardware components. This task covered the
design of the generic Process Model, known as the Virtual
Gateway, that enables communications to a vendor
communications device or to a real network, such as a VPN.
In this task, a Process Model was developed for three
scenarios and types of hardware devices. Routers/switches are
Layer2/3 hardware devices that examine input physical, Maclayer, and network layer packets and route them to output
ports. The interfaces to these devices are either frames or IP
packets. Once this interface to a router/switch type device is
developed, it can be used in other scenarios. A second scenario
is a campus network configuration. Interfaces to these
networks are IP packets via the Process Model. A third
scenario is a VPN that has an interface via a router/switch.
These three scenarios for Hardware in the Loop are depicted
in Figure 2 and have been completed.

A statistics-gathering process model, called Statistic

Workstation, has been implemented that allows a hardware
device under test to be probed for its internal statistics. The
OPNET package captures and record these performance
measure statistics during the hardware in the loop simulations,
and then display the statistic results after the simulation.
Remote Terminal Emulation (RTE) traffic will be used to
interface with the process models that were developed. With
these capabilities, ISEC will be able to evaluate individual
network components as well as entire networks, such as
VPNs. This work was sponsored by the U.S. Army ISEC.

Router/Switch

Campus LAN
(CUITN)

R/S

Packets

Process Model

Virtual Private
Network (VPN)

R/S

Packets

Process Model

R/S

Packets

Process Model

Figure 1. Sequence and Relationship of Tasks in this Project

Events

Accomplished Objectives
1

Events

Events

Figure 2. Task 1.0 Addresses 3 Scenarios for Interfaces to the

Process Model

capture of keystrokes and application message traffic using a

RTE software package. The RTE software package then
executes on a farm of PC and Unix servers and the application
traffic is directed to the device or network under test. In this
task, we developed a method make use of RTE traffic as input
into the Process Model developed in Task 1.0. Figure 3 shows
the RTE interface to a communications device and Process
Model.

The Process Model that interfaces with the Hardware in the

Loop communications network devices is contained in the
state transition diagram and C-code. Depending on the
communication device, the interface will be different. The
Process Model has two main functional parts as described in
the OPNETWorks 2001 paper on HITL: a) IP_Rcv_Hdw
Receiving Process, and b) IP_Send_Hdw Sending Process.

Communications
Hardware
Component

Once the Process Model is developed it can be used to connect

to a new device, and obtain its configuration files containing
the device attributes and mapped to the OPNET environment.
This enables a fast method of characterizing new vendor
devices using the Process Model. Using the same techniques
above, existing OPNET models (CampusLAN scenario) can
be used to interface to the Process Models that represent a new
vendor communication device. The resulting system can be
used to perform evaluation of new devices in the Common
User Installation Transport Network (CUITN) architectures.
The results are gathered in the same way as before, using the
OPNET parameter measurement system for accumulating
simulation data.

CELLplex
7000

IP Traffic Load or
Traffic Generators

3COM

Current RTE Method

OPNET
Process Model
RTE Server
Farm
Remote
Terminal
Emulation

OPNET Model

Figure 3. RTE Traffic Load Can Drive the OPNET Process

Model.

In summary, this objective resulted in a prototype generic

Process Model using OPNET as the software, a translator
module in the Process Model that converts OPNET events to
packets and packets to OPNET events, and C-code that
interfaced the Process Model to the communication hardware
device. The three types of device scenarios that were
considered were Router/Switches, CampusLANs, and VPNs.
The interface to the Process Model for the Technology
Integration Center (TIC) RTE was developed later in Task 3.0.

The RTE traffic is presented to the device under test. The

device then forwards the traffic load to the Process Model.
The Process Model then converts the traffic load to OPNET
events or into inter-arrival times for traffic generators in the
Process Model. A reverse traffic scenario is possible since the
Process Model can generate IP packet to the device.
In order to interface to the RTE traffic, we place a computer,
Carnousite, at the TIC with our HITL process model already
loaded. We placed Carnoustie on the same LAN as the RTE
and had the RTE traffic sent to Carnoustie via an IP address
and associated port number. Our HITL process model was
then able to receive the RTE traffic data and process it. In
summary, this task produced a method to use RTE traffic to
drive the OPNET Process Model that drives the
communication device.

Task 2.0 invloved the development of the OPNET code for

capturing the performance measures of the communications
hardware device. This process model used OPNET methods
for capturing the performance measures of the
communications hardware device.
In this task, we used the existing OPNET methodology for
statistics gathering and tool sets to capture the performance
measures of the communications hardware device.
In
summary, this task produced the methods to measure statistics
of the hardware device under test. Further information about
this is described in the Design and Implementation section of
this paper.

Task 4.0 involved the development of the methods in the

process model that will allow a VPN to be tested using
OPNET modeling techniques. The VPN will drive the OPNET
model or the OPNET model will drive the VPN by using the
RTE traffic data in the TIC.
The Process Model developed in Task 1.0 will allow an
external network cloud or a VPN to be tested using the
Process Model. Since the Process Model contains an interface
to a router or switch type communication device, it can be
interfaced to an external network via a router or a switch. The
network behind the router can be a VPN that allows IP
tunneling through the router and into the VPN nodes. Figure 4

Task 3.0 involved the development of the methods to utilize

the RTE to provide traffic data to the OPNET model that
drives the communication hardware device.
RTE is used in the TIC to provide a semi-realistic traffic
workload to communication network devices and network
configuration under test. The RTE process involves the
2

shows the OPNET Process Model and interface to a VPN via a

router.

Figure 5. Actual Network Hardware Inserted into the OPNET

Simulation Loop Using a Process Model

VPN

A generic Process Model, HITL Virtual Gateway, was

developed to interface to the hardware communications device
(i.e. router). Once the HITL Virtual Gateway was developed,
it was fairly simple to interface to an entire LAN network and
even a VPN. The same approach can be used to interface the
RTE traffic load to the OPNET models via the Process Model.
In this way, previous traffic loads from the RTE can be used to
load the OPNET model. The HITL Virtual Gateway Process
Model essentially serves as an interface portal to different
communications devices.

VPN
Process Model

Router

In order to develop a statistics-gathering process model, we

first learned about how HP Openview monitored and graphed
the hardwares (i.e. router) statistics. We primarily used HP
Openview to obtain OID (Object Identification) numbers of
the statistics that we wanted to graph using OPNET. An
example of some of the OID numbers that were obtained are:

Figure 4. The Process Model Interfaces to a VPN via a Router

With the configuration in Figure 4, we have used the OPNET
models to send traffic to a VPN via the Process Model
interface. Likewise, the VPN can send traffic via the Process
Model to the OPNET model and thus drive the OPNET model.
The VPN can be a portion of a CUITN (Common User
Installation Transport Network) base network, such as Ft.
Carson, or it can be an actual external network or cloud, such
as an Internet, NIPRNET, or DISN. This capability will allow
ISEC and TIC modelers to evaluate the performance of
interfaces to actual external networks and OPNET models. In
summary, this task developed the methods in the Process
Model to allow it to interface to a VPN or external network.
Further information about the VPN is described in the testing
section of the paper.

These three statistics were chosen to demonstrate the

capability of graphing and monitoring the hardwares statistics
using OPNET. Additional statistics can be easily added to the
code by merely obtaining the corresponding OID number for
the statistic that is desired.
A Cisco 2611 router was used as the network device. SNMP
(Simple Network Management Protocol) was used to access
the network devices statistics tables. SNMP is an applicationlayer protocol designed to facilitate the exchange of
management information between network devices. SNMP
was used because it provides the most generic way to get
statistics files from the network devices. As a result, this
approach can be applied to other network devices.

Design and Implementation

A communication network hardware device (i.e. router) was
placed in the loop of an OPNET discrete event model. Figure
5 illustrates an example of the overall concept of the
Hardware in the Loop application, showing a network
switch, router, or firewall in the simulation loop of an OPNET
Model. The communication device was interfaced to the
OPNET model by using a Process Model. A process model is
an OPNET mechanism that allows users to represent new
models that are not in the OPNET model library.

In order to separate the functional processing of the statistic

gathering mechanism with the HITL Virtual Gateway process
model, the statistical gathering mechanism was designed as a
separate process model. The statistics process model is called
the Statistics Workstation. This makes the display of the
statistics using OPNET procedures easier and modular.

Receive Packet
Switch
C Code Process
Model
Interfaces
to Comm.
Components

CELLplex
7000

3COM

Input IP packets discarded: .1.3.6.1.2.1.4.8

UDP datagrams delivered: .1.3.6.1.2.1.7.1
UDP datagrams not delivered: .1.3.6.1.2.1.7.3

Receive Event

We have integrated the SNMP client side code within

OPNETs process model to create a Statistics Workstation
Process Model.
When the simulation runs, OPNETs
simulated virtual network will exchange IP packets and
interact with the real hardware component (i.e. router).
Periodically, the SNMP agent within the Statistics
Workstation Process Model will send out poll messages to the

Router
CB 2500

Firewall

individual network hardware, and collect their performance

statistics.
Once the simulation is over, the network
components performance statistics can be processed and
displayed in OPNET. The Statistics Workstation can be run
on in the same simulation run as the HITL Virtual Gateway or
can be run on another computer.

Figure 7 illustrates the OPNET Network Model

communicating with an ECHO Server through ECE Network
or MIS Network.
OPNET Computer in CERL Lab: ECE #401
(opnettest2)

Figure 6 illustrates the interaction between the HITL Virtual

Gateway process model, Statistics Workstation process model,
WorkStations (WS) and the network hardware device (i.e.
router). When the simulation runs, the HITL Virtual Gateway
will convert OPNET virtual IP packets into real IP packets,
and transmit them to the real network through the router.
Once the echo server receives the real IP packets from the
OPNET simulation, it will bounce the IP packets back to
OPNET simulation via the router. The HITL Virtual Gateway
will then translate the real IP packet back into virtual IP
packets within OPNET.

Computer in ECE #320M or MIS Lab

(opnettest1 or opnettest3)

Packets From OPNET to ECHO server

Packets From ECHO server

Figure 7. OPNET Network Model communicating with an

ECHO Server through ECE Network or MIS Network.
The echo-server was set up in the computer in ECE Rm 320M
and at the E-Commerce Lab in the MIS building. We were
able to send IP packets to and from the router to the PC in Rm
320M and the PC in the MIS building. The OPNET Process
Models were able to translate the IP packets into OPNET
events and vice versa through the router hardware device.

The Statistics Workstation within the same OPNET simulation

will send out SNMP polls periodically to the router, which is
SNMP enabled. Once the router receives the SNMP poll
messages from the Statistics Workstation process model, it
will respond with statistics packets from router files. These
packets are collected by Statistics Workstation process model
and handed over to the statistical procedures that process
OPNET statistics. The statistics from the router can then be
displayed using any of the OPNET display results commands.

The C code developed to capture and display statistics for

OPNET Simulation Network

WS

W
HITL S
Virtual
Gateway

W WS
S
S
W

SNMP Agent

Simulation Packets
WS

W
S
Statistics
Workstation

SNMP Poll

Simulati
on
SNM
SNMP Response
packets
P
POL
L
ECHO
SERVER

HITL Virtual Gateway resulted in the creation of the statistics

panel as seen on the right-hand side of Figure 8 illustrated
below.

ROUTER

Figure 6. Interaction between OPNET & network components

Figure 8. Statistics Panel to Capture and Display Statistics for
HITL Virtual Gateway

Testing and Results

For the CampusLAN scenario, we connected an OPNET model
to another OPNET model located in another network domain
on campus. We used the ECE (Electrical & Computer
Engineering) network domain first and then the MIS (College
of Business domain) network domain second. These are two
distinct domains on campus and the CERL facility is within
the ECE building cloud.

The four types of statistics that were developed and analyzed

are shown below.

For IP_Rcv_Hdw
o Packets Lost (Unite Packets)
o Packets Received (Unit Packet)
o Transmission Delay (Unit Miliseconds)

For IP_Send_Hdw
o Packets Sent Out (Unit Packets)

Figure 10a. Packets Lost (ECE Rm 320) for

IP_Rcv_Hdw of HITL Virtual Gateway

When we tested OPNET with the echo server running at the

MIS building & ECE Rm 320M, there was in increase in delay
and packets dropped from the MIS building as compared to
the ECE Rm 320M. This is shown in Figure 9a/9b and Figure
10a/10b. This is what was expected since the data that was
going to the MIS building had a larger traffic domain.
Figure 11a/11b and 12a/12b illustrate the packets received and
sent to/from the OPNET process model to the echo server in
ECE Rm 320M or the MIS Lab.

Figure 10b. Packets Lost (MIS Lab) for

IP_Rcv_Hdw of HITL Virtual Gateway

Figure 9a. Transmission Delay (ECE Rm 320) for

IP_Rcv_Hdw of HITL Virtual Gateway

Figure 11a. Packets Received (ECE Rm 320) for

IP_Rcv_Hdw of HITL Virtual Gateway

Figure 9b. Transmission Delay (MIS Lab) for

IP_Rcv_Hdw of HITL Virtual Gateway

Figure 11b. Packets Received (MIS Lab) for

IP_Rcv_Hdw of HITL Virtual Gateway

Cisco 2611
Router

ECE 3COM SuperstackII

OPNET Test2 Computer

(w/ Statistics Workstation)

Starrpass Computer (Echo Server)

Figure 14. Statistic Workstation Test

Figure 12a. Packets Sent (ECE Rm 320) for IP_Send_Hdw of
HITL Virtual Gateway

Figure 15. Statistics Obtained from Router: IP Packets

Received (Accumulated)

Figure 12b. Packets Sent (MIS Lab) for IP_Send_Hdw of

HITL Virtual Gateway

Figure 13 illustrates the OPNET Network Diagram of Router

Statistics Workstation & HITL Virtual Gateway
.
Figure 14 shows the Statistic Workstation test setup.
Figures 15-17 displays the Statistics Obtained from Router

Figure 16. Statistics Obtained from Router:UDP Datagrams

Not Received

Figure 13. OPNET Network Diagram of Router Statistics

Workstation & HITL Virtual Gateway
Figure 17. Statistics Obtained from Router: UDP Datagrams
Received (Accumulated)

For the first VPN test, we ran the OPNET process models on
Augusta (machine at University of Arizona) and ran an echo
server on Carnoustie (machine at Ft. Huachuca) through the
VPN. Carnoustie was able to receive and echo the OPNET IP
packets. IP/UDP packets were observed being sent back from
Carnoustie to Augusta.

The RTE was connected to the HITL Virtual Gateway process

model as shown in Figure 20. The hardware device under test
(i.e. router) was configured with subnet 98 and 99. The PC
(Carnoustie) was placed on subnet 98 and the RTE was placed
on subnet 99. The PC, RTE, and hardware device (i.e. router)
were connected via a switch.
PC: Carnoustie
(98 subnet)

For the second test, we ran an OPNET process model on

Carnoustie and the same OPNET process model on Augusta
through VPN. The Process models may also run in other
domains, via a VPN connection. For example, in the Internet
2 and DREN internetworks or UA and TIC Domains.

RTE
(99 subnet)
Summet 40 Switch
Data Packets From
RTE to Carnoustie

Figure 18 illustrates the OPNET Process Model

Communicating with ECHO Server through VPN Tunnel, Test
1.
Figure 19 shows the OPNET Process Model
Communicating w/ another Process Model through VPN
Tunnel, Test 2.

1/0: 99 subnet

0/1: 98 subnet

Figure 20. OPNET HITL Virtual Gateway Process Model

OPNET Computer in CERL Lab: ECE #401

(Augusta)

Data was constantly transmitted from the RTE to the Virtual

Gateway process model on the PC. The Virtual Gateway
process model was able to receive the real RTE data via an
associated IP address and port number. The Statistics
Workstation process model was able to probe the hardware
device under test (i.e. router) during the transfer of the RTE
data. The graph results seen from the analysis of the router
statistics are similar to Figures 15-17.

Computer in TIC
(Carnoustie)

Packets From OPNET to ECHO server

VPN Tunnel
Via Internet

Packets From ECHO server

Figure 18. OPNET Process Model Communicating w/ ECHO

Server through VPN Tunnel, Test 1

OPNET Computer in CERL Lab: ECE #401

(Augusta)

Conclusion and Future Work

This paper describes the integration of COTS network
hardware into the OPNET Modeler for Hardware in the
Loop Modeling & Simulation. Accomplished tasks were
indicated, which assisted in the implementation of the system.
The design and implementation of the system has been
described. The testing and results were also illustrated to
demonstrate the capability of the system. The release of the
process models to the OPNET University depends on the
decision of the U.S. Army Information Systems Engineering
Command due in August 2002.

Computer in TIC
(Carnoustie )

Further work may include the ability to time synchronize

between the real hardware device time with the virtual
OPNET time. This may be accomplished once OPNET
provides the appropriate APIs to do this. The use of HLA
(High Level Architecture) and the RTI (Run Time Interface)
module from OPNET may help to solve some synchronization
issues between two OPNET process models being simulated
on different machines. However, this will not solve the time
synchronization problem between a real hardware device and a
virtual OPNET model. The methods developed under this
project that allow two OPNET models to communicate over a
real network will result is less overhead than using the
HLA/RTI method.

VPN Tunnel
Packets From Augusta to Carnoustie

Cisco 2611
Router

Packets From Carnoustie to Augusta

Figure 19. OPNET Process Model Communicating w/ another

Process Model through VPN Tunnel, Test 2

Other applications that may benefit from the HITL project

include evaluation of Public Key Infrastructure (PKI)
hardware and virtual model scenarios, security using Top
Level Architecture (TLA), and the Joint Tactical Radio
System (JTRS).

References
[1] OPNET Modeler Documentation, OPNET Technologies
Inc., Bethesda, MD, 2001.
[2]
CUITN Core Switching Architecture Requirements
Analysis, U.S. Army Information Systems Engineering
Command, Technology Integration Center, Dec. 2000

Other future work is the development of a practical application

for using the process models that were developed. Basically
any hardware device with an IP address can interface with our
process model, so the applications of Hardware in the Loop
techniques may be limitless.

[3] Methodology for Incorporating Commercial off the Shelf

(COTS) Network Hardware into OPNET Modeler for
Hardware in the Loop Modeling & Simulation,
OPNETWorks 2001, Washington DC, Aug. 2001.