QualNet-OPNET-Comparison LB 06 06

Comparisons of OPNET and QualNet by Third Parties
Confidential and Proprietary - Scalable Network Technologies, Inc.
Summary

Objective: Fair comparisons of two commercial simulators via studies conducted by third parties for their own projects Three studies:
Selection of the best simulator on the market for MANET studies: BISON report (by a European research project) Evaluation of OPNET and QualNet on their runtime (sequential) performance: OSPFv2 report (by Boeing Phantom Works) Practicality of parallelizing existing simulators designed for sequential execution: WSC paper (by Georgia Institute of Technologies)
Benchmark project Quotes from users of both QualNet and OPNET
Architecture of the Simulation Environment*

From a report (June 2003) for the BISON project (Slides prepared by SNT) A. Montresor, G. D. Caro and P. E. Heegaard Universita di Bologna (Italy), Telenor Communication AS (Norway), Technische Universitat Dresden (Germany), IDSIA (Switzerland), Santa Fe Institute (USA)
* Funded by the European Commission under the Information Society Technologies Programme of the 5th Framework (1998-2002)
Confidential and Proprietary - Scalable Network Technologies, Inc. 3
Study 1: Overview

BISON: Biology-Inspired Techniques for Self Organization in Dynamic Networks Discrete event simulation as the main tool to study and predict the behavior of communication networks Simulation requirements completely different for different types of networks
Mobile ad hoc networks (MANETs): routing, traffic and mobility patterns Overlay networks: millions of nodes
Identifying the best simulation tool to meet various requirements for the BISON project (SNT note: this slide set summarizes only the MANET section)
Selection Criteria
Selection criteria for mobile ad hoc networks (18)
Selection Criteria (Contd)
Selection criteria for mobile ad hoc networks (915)
State-of-the-art of Network Simulators
Possible candidates to build up our simulation environment: OPNET GloMoSim QualNet NS-2 OMNeT++ Investigate the characteristics of these simulators and their general compliance to the selection criteria
Study 1 Conclusions
QualNet as Simulation Framework
None of reviewed simulators possesses set of characteristics required by BISONs research plans and objectives However, QualNet appears as the best compromise: An extensive set of pre-built models, protocols and algorithms A good level of acceptance from the scientific community An excellent scalability A rather good, highly modular, software design A satisfactory level of usability, modifiability and expandability Advanced graphical and mathematical tools for experiment building, monitoring and post-processing Good documentation Possibility of parallel and/or distributed implementations
Conclusions (Contd)
Other simulators
OMNeT++ does not include an extensive set of models, protocols and algorithms Only QualNet and OMNeT++ can scale up to thousands of nodes For the ease to use/modify/extend, NS-2 scores poorly, QualNet and OPNET are comparable, and OMNeT++ seems the best among all
We leave open the possibility to use either NS-2 or OMNeT++ in the future
If QualNet limits for our research
Study 2: QualNet and OPNET Evaluation

From a report (December 2002) approved for release (Slides prepared by SNT) T. Henderson and J. Kim Communications Network Technologies Boeing Phantom Works
OSPFv2 model comparison
Software

OPNET Modeler 8.0 QualNet ver. 3.1e 1 GHz Pentium 4 machine (single processor) with 1 GB RAM
Source OPNET library, based on RFC 2328 QualNet, based on RFC 2328
No No No No No No No No No support support support support support support support support support for for for for
Hardware
Protocol OSPFv2
Limitations or issues
virtual paths AS external routes authentication and checksum processing Type of Service (TOS)-based routing
OSPFv2
for virtual paths for AS external LSA for authentication and checksum processing of incremental LSA update of equal cost multipath
11
Target Scenario
A simplified topology analogous to the current Naval afloat scenario

192.4.1.2 192.1.1.1
A0
192.4.1.1
Antenna_SAT_1 (17) Ship_Router (11)
Satellite (23) Ship_0 (2)
Host (14)
A1
192.4.2.2 192.4.2.1
192.1.2.1
A0
Antenna_SAT_1 (18) Ship_Router 192.3.2.1 (12)
Host (15)
A2
A0
NOC_Antenna_1 (8) Antenna_LOS_1 (21)
NOC_Antenna_2 (9) NOC_Antenna_3 (10)
192.1.3.2
A0
192.1.2.2
Ship_1 (3)
NOC_Antenna_0 192.1.1.2 (7)
A0
192.2.3.2
A0
A0 NOC_Router_0 (6)
192.1.3.1 192.4.3.2 192.4.3.1
A0
A0
Antenna_SAT_1 (19)
192.2.3.1
Host (16)
A3
Ship_Router (13)
192.3.2.2
Antenna_SAT_2 (20)
A0
NOC (1)
Antenna_LOS_1 (22)
Ship_2 (4) Fleet (5)
12
Target Scenario (Contd)
Two clusters of simulated topology:

Ship 0
CBR HTTP
192.5.1.1 192.3.1.1 192.1.1.1 192.1.2.1
Ship 1
CBR HTTP
192.5.2.1 192.3.2.1 192.3.6.1
Ship 5
CBR HTTP
192.5.6.1
Ship 6
CBR HTTP
192.5.7.1
FTP
FTP
FTP
FTP
SHF=256 kb/s, metric 800 (SAT) DWTS =256 kb/s, metric 700 (LOS) INMARSAT=64 kb/s, metric 1300 (SAT) EHF LDR=32 kb/s, metric 2700 (LOS)
192.1.6.1
192.1.7.1
192.3.7.1
Ship 2
CBR HTTP
192.5.3.1 192.3.1.2 192.1.3.1
Ship 3
CBR FTP HTTP
192.5.4.1 192.3.6.2 192.1.4.1 192.3.4.1 192.1.8.1
Ship 7
CBR HTTP
192.5.8.1 192.1.9.1
Ship 8
CBR HTTP
192.5.9.1
FTP
FTP
FTP
FTP
192.3.4.2
Ship 4
CBR HTTP
192.5.5.1 192.3.2.2
Ship 9
CBR HTTP
192.5.10.1 192.3.7.2 192.1.5.1 192.1.10.1 192.1.6.2 192.1.1.2 192.1.5.2 192.1.7.2 192.1.10.2 192.1.9.2 192.1.8.2 192.1.4.2 192.6.1.1 192.6.1.2 192.6.1.3
FTP
FTP
192.1.3.2
192.1.2.2
NOC (Network Operating Center)
13
Validation of Simulation Results
Traffic load comparison for 20 clusters (500-600 nodes)
Considerable amount of similarity between results of the two simulators except for the OSPF database description traffic
(SNT note: while both are valid, a database description packet likely contains multiple LSA headers in real implementation) (SNT note: database description packets: 3.3% of total number of packets)
Packet Type HELLOs DBASE_DESCs LSUs LSAs LSRs CBR FTP HTTP TOTAL Total Number of Packets OPNET 73,950 14,824 46,708 12,416 22 15,660 222,285 56,450 442,315 QualNet 73,950 1,011 46,844 16,420 18 15,660 217,929 62,524 434,356 Total Number of Bytes OPNET 3,875,880 767,248 6,853,834 1,743,244 792 23,490,000 184,046,684 12,317,747 233,095,429 QualNet 3,875,800 1,123,496 6,996,788 1,134,180 888 23,490,000 185,473,106 10,075,932 232,170,190
14
Memory Consumption
Memory Usage vs Number of Clusters (log scale)
QualNet provides one order of magnitude memory usage reduction over OPNET
1000
QualNet
Memory Usage (Mbytes)
OPNET
100
10
1 2 4 6 8 10 12 14 16 18 20
Number of Clusters
15
Runtime
Simulation Time vs Number of Clusters (log scale)
QualNet provides two orders of magnitude simulation time reduction over OPNET Even with 20 clusters, the QualNet simulations executed in better than real-time
55.5 Simulation time/Real time ratio
QualNet OPNET
5.5
0.55 0.055
5.5e-3 5.5e-4 50 100 150
Faster than real time
200 250 300 Number of Nodes
350
400
450
500
16
Study 3: Parallel Simulations Using OPNET*

Prof. Richard Fujimoto Georgia Institute of Technology fujimoto@cc.gatech.edu (excerpted and reformatted)
* Funded by a grant from the National Science Foundation
Executive Summary
Key issues in federating network simulators
Event exchange and synchronization

Implemented using proxies
Static global variables

Implemented using ghost nodes
Dynamic global variables and zero lookahead events

Problematic: requires substantial revision to model
OPNET
Simple models can be readily parallelized (UDP/IP) Many models require substantial revision because of global state and zero lookahead events [Federating: running the simulator on parallel architectures.]
18
The Global State Problem

read
read
Federate 1
Federate 2
Network simulators are often designed assuming complete, global information of the network; e.g., computing routes A federate simulating a subnetwork has incomplete information
Solution Approach: Ghost Nodes

read ghost node actual node uninstantiated read
Assume state information does not change during execution (static state) Ghost node: a skeleton model representing a remote LP which caches remote state information Instantiate ghost node object for each remote LP referenced by an LP
Lookahead
SNT Proprietary
Lookahead is defined as the minimum simulation time into the future that a federate can schedule an event
event
problem: limited concurrency each federate must process events in time stamp order
Federate D Federate C Federate B Federate A TA TA+LA Simulation Time without lookahead possible message OK to process with lookahead possible message OK to process not OK to process yet
Lookahead is necessary to allow concurrent processing of events with Confidential Proprietary - Scalable Network Inc. different time stampsand (unless optimistic event Technologies, processing is used)
21
Zero Lookahead Events

Interrupt @ time 100
Simulation Time = 100
OPNET: Interrupts generate zero lookahead events
FTP schedules an event for destination indicating end of transmission
Zero lookahead events essential serialize execution Zero lookahead does not happen in nature; artifact of the way the network model was developed Solutions

Modify model to eliminate zero lookahead interactions Require substantial revision to existing software
22
Study 3:Conclusions
OPNET models will require significant revision to federate with other network simulations (even, other OPNET federates) Because of the above difficulties, work on OPNET for NSF project has been stopped Extension to include Opnet appears to be problematic, will add significant risk to the project
23
Study 4.
Tool Comparison by a Commercial European Organization
24
Study 4: Comparison
Product 1 NS-2 Product 2 OMNET++
Commercial license: Sim. Platform: US $ 15000.Support: US $ 1800.- p.a No costs for runtime license Sim. Platform and modules good documented.
Product 3 OPNET
Commercial License (Floating License) Sim. Platform: $ 40000.Runtime license: $ 4000
QualNet
Commercial License (Floating License) Sim. Platform: $ 24000 Runtime license: $ 12000 Sim Platform and modules documented. Source code reading necessary. No access to simulation kernel source code. Full access to modules source code. ANSI C / C++ Procedural model design!
License Model / Costs
Open Source Free Available
Available Documentation
Sim. Platform well documented. Rudimentary modules documentation available. Full access to simulation kernel and modules source code.
Very good documentation.
Access to Source Code
Full access to simulation kernel and modules source code. OO-Design / C++ OO-Designed models
No access to simulation kernel source code. Full access to modules source code.
Language Model
ANSI C / C++ Procedural model design! Confidential and Proprietary - Scalable Network Technologies, Inc. Combination of TCL and ANSI C
25
Study 4: Comparison continued

Product 1 NS-2 Product 2 OMNET++
Main modules available. (IPv4, IPv6, MPLS) Routing Protocols missing! (OSPF)
Product 3 OPNET
Main modules available. Necessary Investment: IPv6: US $ 25000.MPLS: US $ 25000.-
QualNet
Main modules available. (IPv6 still trail with limited functionality) Necessary Investment: QoS Module: US $ 10000 Moderate
Model Library
Main modules available. Interoperability between different modules not given!
Assumed Period of Vocational Adjustment Supported Platforms
Very high
Moderate to high
Moderate to high
Windows 95/98/ME Windows NT/2000 Linux
Windows NT/2000 Linux
Windows NT/2000 Sun OS (SPARC)
Windows NT/2000 Linux Sun OS (SPARC) Yes (Each job requires one runtime license!)
Usage in Linux GRID Possible
Yes
Yes
No
26
Study 5.
Benchmark study by a major defense contractor
27
Study 5. Benchmarking Project

Number of Nodes A single division is to be modeled Division - has 4 brigades Brigade each brigade has 4 battle groups Battle group each has 2 companies and 2 squadrons Company each has 16 Warriors organized into 4 platoon commands Squadron each has 12 tanks organized into 3 troop commands
Study 5. Benchmarking tools
Voice traffic will use VHF or HF. Data traffic will use HCDR if available. The division HQ, Brigade HQ, and Battle Group HQ will be equipped with HCDR. If an entity is not equipped with HCDR, data will be sent over VHF / HF. Voice traffic always takes priority over data traffic.
29

Radio Parameters Radio Type, Transmit power, Frequency Range, Data Rate, VHF16W, 30-88 MHz, 2400 bps, HF100W, 1.5-30 MHz, 2400 bps, HCDR 20W, 225-450 MHz, 244 Kbps All radios have an antenna height of 2 meters. Each radio runs 802.11 and utilizes OSPFv2 routing.
30

Data Traffic All vehicles send a position report every minute. These reports get routed to all other vehicles. Size of position report: assume 100 bytes 200 users send a 50 kbyte message every 5 minutes Voice Traffic Voice traffic on each of the defined voice nets: Division: 50% of the time Brigade: 45% of the time Battle group: 45% of the time

Protocols required are as follows: Protocol Simple Jammer models (repeater, look-thru, follow, spot, barrage) RS-423FED-STD-1052 (ARQ)STANAG-4538 HF Data Link, Clustering Protocol, QBL-MSK Modulation TCP, UDP, IPv4, IEEE 802.11 CSMA/CA, Ethernet OSPFv2 DHCP FEC SNMPv1 Interface

Terrain Gaming area: Terrain model:
120 km by 70 km Irregular Terrain Model (Longley-Rice)
33

Movement Division HQ none Brigade HQ - none Battle group HQ - none Company / Squadron group yes (60% of entities in constant movement) Assume the moving entities move in a square 1 km x 1 km Scenario Duration The scenario will run for 30 minutes.
Scenario preparation

SNT 1.5 weeks and working flawlessly OPNET 8 weeks and modeled only fractions of the network with no terrain effects.

Problems with OSPFlong run times and crashes No Linux implementation No GUI implementation
35
Study 5. Benchmark Setup

QualNet PC OS
Nodes Protocols Terrain DTED level Propagation Path Loss Vehicle speed Propagation Limit Elevation update resolution
OPNET Dual 1.2GHz Pentium III1GB Windows 2000

1930 802.11b, routing OSPFv2 (RFC 2328), IPv4, TCP, UDP
Dual 1.2GHz Pentium III 1GB Windows 2000

1866 802.11b, routing OSPFv2 (RFC 2328)
NE 34.99 -119.01; Fort Hood, display: SW 34.01 -119.7772; contour lines cross 1 Longley-Rice (ITM) 30 km/h -111 dBm 5m 0 Longley-Rice (ITM) 20 km/h none
Position updateConfidential resolution and Proprietary 100 m 100 m Inc. - Scalable Network Technologies,
36
Study 5. Benchmark Setup (Cont.)

QualNet Voice equivalent Data traffic Data equivalent Data users Position reports Transmit power VHF Transmit power HCDR Antenna height Encoding Propagation fading model Propagation model 16 kbit / s 1.25 kbytes every 7.5 s 50 kbytes every 5 min 200 100 bytes / min, staggered 16 W 20 W 2 meters, omnidirectional gmsk - BER table none statistical 50 kbytes every 5 min 200 100 bytes / min, staggered 16 W 20 W 2 meters, omnidirectional gmsk - BER table none OPNET 16 kbit / s
37
Study 5. Benchmark
Timing Results
Events Execution time (1 CPU) Execution time (2 CPUs) Initialization time QualNet 175 million 3130 sec 2966 sec 22 sec 3-20 minutes OPNET 156.5 million 1h 46m 7s = 6367 sec
Core memory usage
800 MB
Slowly increases to 1.2 GB
still more than twice as fast as OPNET.
Despite using a full OSPF implementation, 10 times the number of terrain posts along a line of sight, and 50% more terrain calculations, QualNet was
38
Study 5. Benchmark
Conclusion

QualNet twice as fast as OPNET OPNET numerous crashes, questionable computed delays, twice as slow QualNet delivered on time, no crashes, and computed results were as expectedtwice as fast. QualNet executed faster than real time QualNet is the clear winner
39
QualNet Customer Experiences

We chose QualNet because it is the most promising tool to realistically simulate battlefield communications in real-time.
-John Powers, Raytheon
The parallel simulation kernel has the potential to allow our proposed simulation environment to execute in near real time. The serial simulation execution appears to be very efficient.
-Highland Systems
40
Customer Experiences
We ran a model in QualNet of a hybrid mobile and satellite network with over 500 nodes, running the Open Shortest Path First (OSPF) routing protocol and application traffic. The model ran faster than real time and was about 100 times faster than another widely used network simulator modeling the same scenario. Furthermore, the output from QualNet was validated at a packet-by-packet level. Tom Henderson, Boeing Phantom Works
About Scalability I use QualNet because scalability is my #1 concern. My typical simulation scenarios have thousands of nodes. There is no other simulator that can deliver that kind of scalability. Chien-Chung Chen, University of Delaware
42
About High Fidelity Models We recently finished porting our MANET routing protocol into QualNet. It turns out that QualNet also makes a good stress-testing tool for MANET code. Larger scale tests in QualNet uncovered a bug in one of the routing table calculation routines that did not manifest itself in any of our previous testing. Andreas Yankopolus, Scientific Research Corporation QualNet is interesting to our team because we do a lot of our own coding. The QualNet code base is clean and intuitive and has easy-to-follow APIs. Tom Henderson, Boeing Phantom Works
About the QualNet Modeling Environment [QualNets] fixed layer architecture approach makes adding, deleting, inserting layers possible, but difficult to maintain for non-expert users this will be important in proposed simulation environment as layer structure needs to be flexible and include layers other than traditional communications layers SNT is working on improving this in the next few months. -Highland Systems
"My focus is building protocols for reliable and scalable networks in mobile wireless environments. My productivity is high with QualNet because of the product's detailed physical layer models and diverse set of protocol models. I can build my own models fast using QualNet." -Chien-Chung Chen, University of Delaware Currently a fixed set of layers is in place in QualNet (application, transport, network, link, physical & antenna). A set of APIs is defined to allow communications across the layers. This can be limiting if there is a need/desire to include an intermediate layer such as an adaptation layer or as an example, if the network layer is providing control directly to an antenna by going around the link layer. -Highland Systems
(QualNets layer-based modeling paradigm is intuitive because it matches the ISO stack. Models based on layers enable and enforce high fidelity protocol models. There is a learning curve in switching to any new technology, and QualNet is not an exception.)
45
QualNet does not allow use of global variables in parallel simulations. This is for purposes of running parallel simulations so that strict partitions can be drawn between objects. This can be limiting when it comes to keeping statistics across the network. -Highland Systems
(QualNet outlaws global variables to enable fast execution speeds. In QualNets defense, there are ways to collect statistics throughout the simulation that still allow parallel execution of a model. In other words, solutions exist for this limitation.)
About the Physical Layer Modeling SNT is focused on the wireless, mobile, DoD environment and new product developments directly support this. -Highland Systems QualNets channel modeling includes path loss, antenna, transmit power, interference, SINR vs. BER, transmission delay, propagation delay. QualNet also includes a TIREM interface, 2-ray reflection model, and turbo-coding. Appears sufficient for proposed simulation environment. -Highland Systems
Antenna Support [includes] custom 3-D antenna pattern support with full 3-D specification. SNR Curve/FEC Support--QualNet provides a general SNR vs. BER input capability and has facilities for providing turbo-code inclusion. Potentially useful in detailed communications simulations. -Highland Systems
48
About the Company QualNet is an up and coming tool that is improving rapidly. The SNT staff is talented and motivated. SNT is also very responsive to user needs and has been very willing to listen to customers. -Highland Systems
49

QualNet-OPNET-Comparison LB 06 06

Hochgeladen von

Dokumentinformationen

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

QualNet-OPNET-Comparison LB 06 06

Hochgeladen von

Copyright:

Verfügbare Formate

Comparisons of OPNET and QualNet by Third Parties

Confidential and Proprietary - Scalable Network Technologies, Inc.

Benchmark project Quotes from users of both QualNet and OPNET

Confidential and Proprietary - Scalable Network Technologies, Inc.

Architecture of the Simulation Environment*

Confidential and Proprietary - Scalable Network Technologies, Inc.

Selection criteria for mobile ad hoc networks (18)

Confidential and Proprietary - Scalable Network Technologies, Inc.

Selection Criteria (Contd)

Selection criteria for mobile ad hoc networks (915)

Confidential and Proprietary - Scalable Network Technologies, Inc.

State-of-the-art of Network Simulators

Confidential and Proprietary - Scalable Network Technologies, Inc.

QualNet as Simulation Framework

If QualNet limits for our research

Confidential and Proprietary - Scalable Network Technologies, Inc.

Study 2: QualNet and OPNET Evaluation

OSPFv2 model comparison

Confidential and Proprietary - Scalable Network Technologies, Inc.

A simplified topology analogous to the current Naval afloat scenario

Antenna_SAT_1 (17) Ship_Router (11)

Satellite (23) Ship_0 (2)

Antenna_SAT_1 (18) Ship_Router 192.3.2.1 (12)

NOC_Antenna_1 (8) Antenna_LOS_1 (21)

NOC_Antenna_2 (9) NOC_Antenna_3 (10)

NOC_Antenna_0 192.1.1.2 (7)

192.1.3.1 192.4.3.2 192.4.3.1

Ship_2 (4) Fleet (5)

Confidential and Proprietary - Scalable Network Technologies, Inc.

Target Scenario (Contd)

Two clusters of simulated topology:

Confidential and Proprietary - Scalable Network Technologies, Inc.

NOC (Network Operating Center)

Validation of Simulation Results

Traffic load comparison for 20 clusters (500-600 nodes)

Confidential and Proprietary - Scalable Network Technologies, Inc.

Memory Usage vs Number of Clusters (log scale)

Memory Usage (Mbytes)

Confidential and Proprietary - Scalable Network Technologies, Inc.

Simulation Time vs Number of Clusters (log scale)

5.5e-3 5.5e-4 50 100 150

Faster than real time

Confidential and Proprietary - Scalable Network Technologies, Inc.

200 250 300 Number of Nodes

Study 3: Parallel Simulations Using OPNET*

Key issues in federating network simulators

Event exchange and synchronization

Static global variables

Dynamic global variables and zero lookahead events

Confidential and Proprietary - Scalable Network Technologies, Inc.

The Global State Problem

Solution Approach: Ghost Nodes

Zero Lookahead Events

Simulation Time = 100

OPNET: Interrupts generate zero lookahead events

FTP schedules an event for destination indicating end of transmission

Confidential and Proprietary - Scalable Network Technologies, Inc.

Confidential and Proprietary - Scalable Network Technologies, Inc.

Confidential and Proprietary - Scalable Network Technologies, Inc.

License Model / Costs

Open Source Free Available

Very good documentation.

Access to Source Code

Study 4: Comparison continued