Vesper RF Acceptance Test Plan: CDMA2000 1xEV-DO Network

Core RF Engineering
Vesper RF Acceptance Test Plan

CDMA2000 1xEV-DO Network
Customer:
Project Name:
Site:
VESPER
Vesper CDMA2000 1xEV-DO Network
Sao Paulo, Brazil
Document Reference:
Version:
Document Status:
Issue Date:
Author:
Contributors:
Acceptance Test Plan

2.3
First Release
February 19, 2003
Rajeev Verma, Email: rajeev@nortelnetworks.com
Department:
Contact:
Authorizing Managers:
Core RF Engineering, 5395

+1-972-685-7653, ESN 445-7653
Frank Jager, Farhad Bassirat and Victor Salinas Lopez
Rubianto Satrio, Miro Budic, Scott Tran, Muhieddin Najib, Jim Oliver, Doug
Klymyshyn, Eric Jerumanis, Brian Troup, Michael Woodmansee and David
Paranchych
REVISION HISTORY
Version
Issue Date
No. of
Pages
Initials
Details of Revision Changes
1.0
10/29/2002
67
RV
First Draft Release for Vesper
1.1
11/11/2002
68
RV
Test procedures updated (sec. 2.3, 5 & 7)
1.2
11/16/2002
75
RV
Updated Appendix A and Section 5, 6, 10
1.3
11/19/2002
75
RV
Updated section 3
1.4
11/20/2002
83
RV
Updated test plans, section 6, 9 & 10
1.5
11/21/2002
83
RV
Second Draft Release for Vesper
1.6
11/22/2002
86
RS
Improved the section on PER, added the section

on Reverse FER and an Appendix on Reverse
FER scripts and logs.
1.7
12/09/2002
99
RS
1.8
12/12/2002
98
RV
Added sections on RL power control messages,

DRC stats, idle handoff measurement, and
Appendix on Chariot scripts. Updated section on
connection drop rate, access failure rate, ATterminated connection success rate and TCP
logging for IPSec test.
Updated TCP RTT details
Third Draft Release for Vesper
1.9
01/22/2003
98
RV
2.0
01/23/2003
97
RV
2.1
02/04/2003
101
RV
Update based on Vesper meeting on Jan 27, 03
2.2
02/12/2003
101
RV
Update on Vesper meeting on Feb.05, 2003
2.3
02/19/2003
102
RV
Updated the test procedures
Update based on Vesper meeting on Jan 20, 03

Format update
Fourth Release for Vesper on 01/23/2003
Fifth Release for Vesper on 02/19/2003
Nortel Networks Proprietary

13_Performance_Acceptance_Test_Plan_Verma.doc
Vesper EV-DO RF Acceptance Test

Page 2
ACCEPTANCE AND SIGN-OFF

This section is a generic sign-off form for this document and test plan, which is required to
be signed by Vesper Nortel and Networks RF acceptance primes.
Signatures from parties involved in the RF acceptance process indicate that the information
contained within this particular document is correct at the date of signing and has been
agreed by the acceptance primes from Vesper and Nortel Networks.
Nortel Networks RF Acceptance Prime

Name:
Title:
Signature:
Date:
Vesper RF Acceptance Prime

Name:
Title:
Signature:
Date:


Page 3
TABLE OF CONTENTS
1. INTRODUCTION.................................................................................................................................3
1.1 PURPOSE ..................................................................................................................................3
1.2 SCOPE ......................................................................................................................................3
1.3 RELATED DOCUMENTS ...............................................................................................................3
2. ACCEPTANCE PROCESS.................................................................................................................3
2.1 VESPER ACCEPTANCE TESTS .....................................................................................................3
2.1.1 RF Performance Tests ...............................................................................................3
2.1.2 Application Performance Tests ..................................................................................3
2.1.3 Supplementary Tests .................................................................................................3
2.2 ACCEPTANCE TEST ENTRANCE CRITERIA ....................................................................................3
2.3 EXCLUSION AREA CRITERIA........................................................................................................3
3. PERFORMANCE METRICS...............................................................................................................3
3.1 RF PERFORMANCE PARAMETERS ...............................................................................................3
3.1.1 RF Coverage ..............................................................................................................3
3.1.2 Connection Drop Rate................................................................................................3
3.1.3 Handoff Failure Rate ..................................................................................................3
3.1.3.1 Forward Link Sector Switching Delay ...................................................................... 3
3.1.3.2 Reverse Link Soft Handoff Failure........................................................................... 3
3.1.4 Access Failure Rate ...................................................................................................3

3.2 APPLICATION PERFORMANCE PARAMETERS ................................................................................3
3.2.1 Single User Application Throughput...........................................................................3
3.2.2 Packet Loss Rate TCP Segment Error Rate...........................................................3
3.2.3 Latency (TCP Round Trip Time) ................................................................................3
4. ACCEPTANCE TEST SETUP ............................................................................................................3
4.1 NETWORK SETUP.......................................................................................................................3
4.2 TRAFFIC MODELS FOR SYSTEM ACCEPTANCE .............................................................................3
4.2.1 Forward Link Throughput Call Model .........................................................................3
4.2.2 Reverse Link Throughput Call Model.........................................................................3
4.3 MEASUREMENT POINTS SELECTION ............................................................................................3
5. VESPER ACCEPTANCE TEST PROCEDURES ...............................................................................3
5.1 RF PERFORMANCE TESTS (SMP1/SP11, SMP2, SMP3, SP9, SP10) ........................................3
5.1.1 Forward Link Coverage Performance Test ................................................................3
5.1.1.1 System Coverage Results ....................................................................................... 3
5.1.1.2 Forward Link single user FTP throughput................................................................ 3
5.1.1.3 Forward link Packet Error Rate................................................................................ 3
5.1.2 Reverse link Performance Test ..................................................................................3

5.1.2.1 Reverse Link single user FTP throughput................................................................ 3
5.1.2.2 Reverse link Frame Error Rate ................................................................................ 3
5.1.3 Connection Drop Test (SP9, SP10, SP11) ................................................................3

5.1.3.1 Connection Drop Results......................................................................................... 3
5.1.3.2 Latency Results ....................................................................................................... 3
5.1.3.3 Forward link sector switching delay ......................................................................... 3
5.1.3.4 Reverse link handoff failure ..................................................................................... 3
5.1.4 Access Failure Test (SP8, SP11)...............................................................................3

5.1.4.1 Access Failure Results ............................................................................................ 3


Page 4
5.2 APPLICATION PERFORMANCE TESTS ...........................................................................................3

5.2.1 File Transfer Protocol (SP1).......................................................................................3
5.2.1.1 Forward Link FTP Measurement ............................................................................. 3
5.2.1.2 Reverse Link FTP Measurement ............................................................................. 3
5.2.2 Hyper Text Transfer Protocol (SP2)...........................................................................3

5.2.3 Email (SP3) ................................................................................................................3
5.2.3.1 Forward Link Email Measurement ........................................................................... 3
5.2.3.2 Reverse Link Email Measurement........................................................................... 3
5.2.4 Near Real Time Video (SP4)......................................................................................3

5.2.5 File Transfer Protocol using IPSec (SP5) ..................................................................3
6. SUPPLEMENTARY TESTS................................................................................................................3
6.1 SUPPLEMENTARY APPLICATION TEST (SP6)................................................................................3
6.2 AT TERMINATED CONNECTION SUCCESS RATE (SP7) .................................................................3
6.3 MAXIMUM SIMULTANEOUS USERS PER SECTOR CAPACITY TEST (SP12) ......................................3
6.4 IDLE HANDOFF TEST (SP13) ......................................................................................................3
6.5 SUPPLEMENTARY RF PERFORMANCE TEST (SP14, SP15, SP16, SP17, SP18) ....................3
6.5.1 Reverse Link Power Control Messages (SP14).........................................................3
6.5.2 Session Success and Drop Rate (SP15, SP16) ........................................................3
6.5.3 DRC Statistics (SP17) ................................................................................................3
7. DATA COLLECTION TOOLS .............................................................................................................3
7.1 INSTALLING QUALCOMM USB DRIVER .........................................................................................3
7.2 SETTING UP WINDOW2000 PCS 1XEV-DO DIALUP .....................................................................3
7.3 TEST ENVIRONMENT ..................................................................................................................3
7.4 CAIT STARTUP PROCEDURE ......................................................................................................3
7.5 CREATING CAIT MONITORING PROFILE ......................................................................................3
7.6 CAIT LOGGING CONFIGURATION ................................................................................................3
7.7 SETTING CAIT LOG MASK ..........................................................................................................3
7.8 SETTING UP GPS.......................................................................................................................3
7.9 CONVERTING THE MDM LOG FORMAT FOR AT ANALYZER TOOL .....................................................3
8. CHARIOT TOOL .................................................................................................................................3
8.1 CHARIOT TOOL DESCRIPTION .....................................................................................................3
8.2 CHARIOT TEST PROCEDURE .......................................................................................................3
9. TCP PERFORMANCE LOGGING ......................................................................................................3
9.1 PROCEDURE TO INSTALL WINDUMP IN THE COMPUTER ................................................................3
9.2 PROCEDURE TO INSTALL ETHEREAL IN THE COMPUTER................................................................3
9.3 PROCEDURE TO INSTALL TCPTRACE IN THE COMPUTER ..............................................................3
9.4 PROCEDURE TO COLLECT SNOOP AND WINDUMP LOGS ...............................................................3
9.4.1 Snoop Logs ................................................................................................................3
9.4.2 Windump Logs............................................................................................................3
9.5 CONVERTING SNOOP LOGS TO TCPDUMP FORMAT .....................................................................3
9.6 PROCEDURE TO RUN THE TCPTRACE TOOL ...............................................................................3
9.6.1 TCP segment Error rate calculation ...........................................................................3
9.6.2 Windump Logs TCP Roundtrip Time..........................................................................3
10. DELIVERABLES ...............................................................................................................................3
10.1 PERFORMANCE TEST DELIVERABLES ........................................................................................3
10.2 SUPPLEMENTARY TEST DELIVERABLES .....................................................................................3


Page 5
11. NORTEL-VESPER CONTRACT RF TEST CASES .........................................................................3

12. ABBREVIATIONS .............................................................................................................................3
APPENDIX A: DATA LOGS REQUIREMENT ........................................................................................3
APPENDIX B: REVERSE LINK FER SCRIPTS AND SAMPLE LOG ....................................................3
APPENDIX C: CHARIOT TEST SCRIPTS .............................................................................................3
C.1. FORWARD FTP TEST ...............................................................................................................3
C.2. REVERSE FTP TEST ................................................................................................................3
C.3. HTTP TEST .............................................................................................................................3
C.4. NRTV TEST .............................................................................................................................3
C.5. FORWARD EMAIL TEST .............................................................................................................3
C.6. REVERSE EMAIL TEST ..............................................................................................................3
C.7. FTP TEST WITH IPSEC .............................................................................................................3
APPENDIX D: ACCESS POINT FUNCTIONALITY TEST FORM.........................................................3
APPENDIX E: ACCEPTANCE TEST LOG SHEET...............................................................................3


Page 6
1. INTRODUCTION
1.1 Purpose
This document explains the Vesper EV-DO RF Acceptance Test procedures as agreed
between Nortel Networks and Vesper.
Customer will be timely updated on any improvements to the Acceptance Test Procedures
as described in this document.
1.2 Scope
This document explains the RF performance acceptance parameters, test procedures and
related tools to evaluate network performance for Vesper EV-DO Network, comprising of 30
sites in downtown Sao Paulo, Brazil.
1.3 Related Documents

This document should be considered with reference to the following documents:
Document Reference
Document Title
Core RF Engineering
EV-DO Link Budget
Qualcomm Tools
CAIT Users Manual
Net iQ
Chariot Tool Users Manual
3GPP2
IS-856 (Standard for CDMA2000 1xEV-DO)
Core RF Engineering
AT Analyzer Tool
Qualcomm Tools
PCubed Users manual


Page 7
2. ACCEPTANCE PROCESS
Network Acceptance is the final performance verification test to validate Network
Performance before handing over the network to customer.
Before staring the Acceptance test, each Access Point will be tested for correct installation
and commissioning. This will help in identifying any problem that might otherwise affect the
network performance. Following are major functions that will be tested.
a. AT originated data connection on each sector
b. Sector switching and handoff to the neighbor site
c. Average RX, TX and Ec/Io values close to the site
Please refer to Appendix-D. AP Functionality Test form will be filled for each AP.
If any problems are identified, a root cause analysis will be performed and the test will be
repeated after correcting the problem.
2.1 Vesper Acceptance Tests

Vesper CDMA2000 1xEV-DO Network Acceptance tests will be broadly divided into
following three sections.
2.1.1 RF Performance Tests

Network Performance Acceptance Tests include RF coverage verification and related
performance over the EV-DO network. Please refer to table-1 for test details. RF
Performance Acceptance tests require Nortel to meet a minimum performance level against
each parameter except for network coverage, as agreed between Vesper and Nortel
Networks.
Since Vesper EV-DO is an IS-95 overlay Network without site densification, the 1x EV-DO
Network performance could be influenced by various unchangeable attributes, namely
existing site locations, inter site distance, cell heights etc. Such limitations would be taken
into consideration during performance evaluation.
RF performance tests will be executed in two phases, first using the calibrated Qualcomm
Chester EV-DO Access Terminal for acceptance performance verification, and again using
LG LST-D400 for information purpose.
2.1.2 Application Performance Tests

Application Performance Acceptance Tests will validate application performance over the
EV-DO network with associated traffic model and RF conditions.


Page 8
Application performance tests will be conducted by stationary measurements using

Qualcomm Chester EV-DO Access Terminals. Thereafter Vesper will repeat this test using
LG LST-D400 for information purpose.
Following five applications will be tested:
a. FTP
b. HTTP
c. Email
d. NRTV
e. FTP with IP Security
Application Performance Acceptance tests require Nortel to meet a minimum performance
level against each performance parameter as agreed between Vesper and Nortel Networks.
Application tests will be conducted by stationary measurement with associated traffic model
and RF environment under each sector.
2.1.3 Supplementary Tests

Supplementary Tests are required for information purpose only. They have no binding on
Vesper acceptance pass or fail criteria.
Supplementary tests will be conducted one time using LG LST-D400 Access Terminals.
Table-1 includes a list of Supplementary tests for Vesper EV-DO network.
2.2 Acceptance Test Entrance Criteria

Acceptance testing will start with the successful installation and commissioning Vesper EVDO network.
Following items represent the minimum entrance criteria for Vesper Acceptance exercise:
1. Acceptance exit criteria are defined, understood and agreed to by both Nortel Networks
and Vesper.
2. Methods of performance measurement, exclusion area criteria, traffic models for all
applications and performance calculation for acceptance metrics has been defined and
agreed.
3. Network is optimized for best achievable performance.
4. EV-DO Network performance limitations due to existing IS-95 site locations and network
design are identified and agreed between Nortel Networks and Vesper.
5. Nortel Networks and Vesper have agreed on the acceptance process deliverables.


Page 9
6. Areas where the actual coverage margins fall short of the original design requirements
due to deployment restrictions (appropriate site availability, antenna-mounting
restrictions, antenna sharing limitations, etc.) will be pre-determined. Vesper and Nortel
Networks will review maps, and an agreement will be reached on the existing coverage
and coverage holes.
7. Maps of the exclusion i.e. non-warranty areas reviewed by Vesper and agreed with
Nortel Networks. Both Nortel Networks and Vesper regional prime will sign for these
areas.
8. All the RF Measurement and post processing tools are accepted and approved by Nortel
Networks and Vesper.
9. Maps of the measurement points (for stationary tests) and acceptance drive routes (for
mobility tests) for data collection has been reviewed and agreed by Vesper and Nortel
Networks.
10. Databases of system design/propagation tool have been updated and verified to reflect
optimized system configuration. Maps of the coverage margins achieved in the design
will be generated using RF design tool as per the optimized system configuration.
11. Network must be locked from any hardware, software, parameter or other changes
during the network acceptance test period. To get a realistic view of system
performance, there will be no phones (Access Terminals) other than the acceptance test
ATs operating in the network during acceptance period.
12. All Network Performance Acceptance tests and performance numbers will be based on a
single user (AT) on the network wide average performance. Multiple ATs will be used
only for three simultaneous applications and sector capacity tests.
13. All logistical requirements for acceptance test are ready. Test equipment calibration,
coordination procedures both internal to Nortel Networks and between Nortel Networks
and Vesper are in place.
14. If due to any unforeseen Network event beyond Nortel's control, or equipment malfunctioning (e.g. unplanned outages, GPS loss etc.) the collected data seems
erroneous, the Acceptance test will be repeated for the affected area.
15. Notice to begin network acceptance test is given by Vesper regional prime.
If above conditions are not met, or due to any other unexpected circumstances, it becomes
necessary to deviate from the Acceptance process as described in this document; Nortel
Networks and Vesper will jointly agree to an alternate feasible process that is mutually
acceptable to both parties.
2.3 Exclusion Area Criteria

Exclusion areas are defined as areas where despite best optimization efforts it is not
possible to generate a data call, mainly due to following reasons:
Coverage hole or link failure on either side due to existing site locations.
Excessive interference due to existing site locations.

Page 10
These areas will be identified using iPlanner, PlaNet (or similar RF prediction tool) and
further verified and finalized using drive-test measurements during coverage test.
Application performance numbers will be measured in areas where Ec/Nt is better than 2.5
dB (i.e. Ec/Io > -2dB) to validate the performance numbers as committed by Nortel
Networks.
Measurements for performance acceptance assessment will not include exclusion areas. If
the drive-test measurements include exclusion areas, then they will be excluded from
acceptance data. To ease the execution of the acceptance without many interruptions, it is
suggested that the drive-test need not be stopped unless the size of the exclusion/nonwarranted area is larger than 10 percents of the cell radius. In this case, the data collected
from non-warranted areas will be excluded during post-processing.
Similarly, stationary measurements will not be conducted in exclusion area.


Page 11
3. PERFORMANCE METRICS
After successful completion of Network deployment and optimization, a date that is
acceptable to both parties would be agreed to begin Network Acceptance. Acceptance
testing will cover five applications as described in following table for various performance
parameters.
S.N.
Test /
Performance Parameter
Application
Mobility /
Stationary
Vesper Contract
ATP Test #
test #
RF Link
RF Performance Acceptance Tests

1
RF Tests
SP11/SPM1
Test 1
System RF Coverage
FWD
Connection Drop Rate
FWD
SP9
Average Handoff Failure Rate
FWD
REV
SP10
Access Failure Rate (PING)
AT Originated
SP8
Test 3
Average Single User Throughput
FWD
REV
Packet Loss (TCP Segment Error Rate)
FWD
REV
SP1
Test 4
Latency (Average TCP RTT)
FWD
REV
SP2
Test 5
SP3
Test 6
SP4
Test 7
SP5
Test 8
Test 2
Application Performance Acceptance Tests

5
FTP
HTTP
FWD
FWD
10
FWD
11 Email
FWD
12
FWD
13
FWD
14 NRTV
FWD
REV
15
FWD
REV
16
Latency (Average One-Way Delay)
FWD
REV
17 FTP / IPSec
FWD
REV
18
FWD
REV
19
FWD
REV
Supplementary Tests (for information only)

20 RF Tests
Network wide single user Throughput
FWD
REV
SPM2
21
Network wide Packet/Frame Error Rate
FWD
REV
SPM3/SP18
FWD
Packet Loss (TCP Segment Error) Rate
FWD
FWD
Connection Success Rate
22
23
24
FTP, HTTP
and NRTV
(Simultaneously)
Test 9*
SP6
Test 10
SP7
Test 11
25 Additional
Sup. Tests
26
Fwd. Link Sector Capacity (# act. users)
FWD
SP12
Test 12
27
Idle Handoff
REV
SP13
Test 13
28
Rev. Link Power Control Msg.
REV
SP14
29
Session drop and success rate
FWD
REV
SP15, SP16
30
DRC Statistics
FWD
SP17
AT Terminated
Test 9*
Table 1


Page 12
Please refer to section 3.1 for the definition of these performance parameters. Traffic models
as agreed in section 4.2 will be used for these tests.
Any exclusion areas identified during the RF design and Network Optimization process will
not be accounted towards performance degradation or included for any acceptance test
measurements. Please refer to section 2.3 for exclusion area definition.
Since the warranted coverage is targeted to perform at 90% or better area reliability, 10%
(100-90=10) of the worst performance data samples will be discarded and the remaining
data will be considered for performance calculation purpose. This will apply to all the
performance parameters in Table-1.
A statistical confidence interval for varying sample size to validate the final result for 99%
reliability will be applied.
Acceptance test performance results will be based on network wide average performance.
3.1 RF Performance Parameters

This section gives a brief description of the major RF Performance parameters as mentioned
in Table-1.
3.1.1 RF Coverage
RF Coverage verification will be performed to validate Networks coverage reliability. The
objective of this test is to validate that network provides sufficient coverage such that the
Access Terminal can generate data call (irrespective of data rate) across warranted
coverage area with acceptable performance in the network.
System coverage performance verification will be accessed via FTP download call using the
throughput and call setup performance within the network coverage area.
Exclusion areas where despite best optimization efforts, a data call cannot be generated will
be identified and a report will be prepared for Vespers review. There is no pass or fail
criteria for coverage test.
3.1.2 Connection Drop Rate

Connection drop is defined as a transition from a connected state to an idle state without an
explicit request from either the Access Node (AN) or Access Terminal (AT).
RF connection drop is not critical in 1xEV-DO networks, as the end users will not know if a
RF connection is dropped. RF connection is always re-established without end user
intervention unless there is a major failure in the network. The end user may experience
some delay at application level just like in case of network congestion. Connection drop is
more critical in voice networks because the user has to redial to start the call again. In
wireless data networks, the higher layers keep their logical connections up even if RF
connection (Physical Layer) is lost momentarily. Therefore, the end users do not notice RF
connection drops directly.

Page 13
A successfully originated call is defined as a successful origination, leading to traffic

channels on both links.
For Vesper EV-DO Acceptance Test the connection drop will be measured in terms of RF
unavailability duration while the AT goes to idle state from a continuous Connected state
in the warranted coverage area. Nortel AT Log Analyzer tool will be used for Drop
Connection measurements.
Ping data from the same test will also be used to determine average Packet round trip delay
time.
Please refer to section 5.2.1 for Connection Drop Rate measurement test procedure.
3.1.3 Handoff Failure Rate

Handoff Failure Rate testing will be divided into following two parts:
3.1.3.1 Forward Link Sector Switching Delay

The EVDO system does not support forward link soft handoff (make-before-break) in the
same manner as cdma2000 / IS95, but does maintain an Active set in the same manner as
cdma2000 / IS95. The Access Terminal (AT) has a single serving sector at any one time
while in an active connection. The AT uses the DRC cover to specify the transmitting sector.
This DRC cover corresponds to a sector in the ATs Active Set.
The AT periodically monitors the pilot channel to determine the desired serving sector.
When a pilot channel corresponding to a non-serving sector becomes more favorable, the
AT signals the ANC to switch the serving sector. It is therefore recommended to refer it as
Forward Link Sector Switching.
For an EVDO system the concept of Serving Sector (SS) switch delay is more relevant than
handoff failure rate on Forward Link. The Serving Sector switch delay is the time between
receiving the indication the AT has changed its desired Serving Sector and the setup of
forward link data queue on the new Serving Sector. If there is data remaining in the serving
sector to be transmitted, the Serving Sector switch ends with the arrival of the data for the
AT in the target sector data queue. For sector switching within the same DOM, the target
data queue and source queue are the same reducing the switching time. For serving
sectors on different DOMs the source data must be re-queued on the target sector, thus
increasing the switching time.
Please refer to sections 5.1.3 and 5.1.3.3 for Handoff Failure Rate measurement test
procedures.
3.1.3.2 Reverse Link Soft Handoff Failure

Handoff Failure refers to the situation where the Network does not respond to the received
Route Update Message from Access Terminal, and as a result the handoff does not take
place correctly, leading to the connection drop. In the reverse link the EV-DO handoff works
similar to in IS-95.

Page 14
Same setup as connection drop test can be used for Handoff Failure test. Please refer to
sections 5.1.3 and 5.1.3.4 for Handoff Failure Rate measurement test procedures.
3.1.4 Access Failure Rate

Access Failure is defined as a connection attempt at which the AT sends a Connection
request message over Access Channel, but the AT does not get the traffic channel
assigned.
Access Failure rate is the ratio of the unsuccessful connection attempts that could not lead
to traffic channel assignment, to the total connection attempts in good coverage area.
PING application will be used to determine Access Failure Rate. Please refer to section
5.1.4 for Access Failure Rate measurement procedure.
3.2 Application Performance Parameters

Following Application performance parameters will be measured:
3.2.1 Single User Application Throughput

The 1xEV-DO access terminal continuously monitors the RF channel conditions in the
forward link and reports this information to the Access Point (BTS). The RF channel is
estimated from the forward pilot bursts, and by comparing the measured Ec/Nt to a set of
predefined thresholds data rate requests are generated. Based on this comparison and
factors proprietary to the Access Terminal, the access terminal determines which of the 12
data rate allowed by the Standards to request. The selected data rate requests are
transmitted to the access point using DRC channel. Although channel conditions are
reported to the access point every time slot, the information is updated every DRCLength
slots depending on the value taken by the DRCLength. While DRC selection algorithm is
proprietary to Qualcomms MSM 5500 chip, the AT vendor and may have other limitations to
generate DRC requests.
Similarly, the base station does not have a complete control on the reverse link data rate.
Access Terminal (AT) determines the reverse link data rate using the Reverse Activity Bit
(RAB), maximum data rate as a function of number of active connections and the transition
probabilities. The base station transmits the RAB bit and maximum data rate as a function of
active connections to the Access Terminals. This information is then used by AT to select
reverse link data rate.
3.2.2 Packet Loss Rate TCP Segment Error Rate

Packet loss rate determines the total packets that get lost despite RLP retransmission. This
loss is compensated at the TCP layer by retransmitting the lost packets. Therefore Packet
loss rate is essentially the TCP retransmission rate or Segment Error rate (SER).
Average Segment Error Rate will measure Packet Lost Rate from TCP Trace logs.


Page 15
TCP trace logs will be generated through Windump and Snoop logs
Only transmitter side logs will be used for TCP SER calculation. Please refer to section 9 for
Packet Loss Rate measurement test procedure using windump and snoop logs.
Please refer to section 8 for Chariot tool details.
3.2.3 Latency (TCP Round Trip Time)

For the applications that use TCP, i.e. FTP, HTTP, and E-mail; latency will be measured
using the average TCP Round Trip Time (RTT). This metric is obtained from the TCP Trace
logs.
For the application that uses UDP, i.e. NRTV, latency will be measured using the average
one-way delay through Chariot tool.
Please refer to section 8 for Chariot tool and section 9 for latency measurement procedure
using TCP Trace.


Page 16
4. ACCEPTANCE TEST SETUP

Acceptance Test preparation involves setting up the processes for data collection, RF
Database management, desk tops, man power/shifts management, log files/sheet
management, and related issues for better efficiency, and to avoid any unforeseen problem
or work redundancy. Following processes are recommended:
Drive test areas, Data collection tools including desktops and vans should be divided
into definite sets to have better process/inventory control, identifying mal functioning
equipments and avoid delays related to the equipment repair and replacement.
The data should be processed and analyzed as early as possibly to keep good track of
the network performance and nature of data.
All the processed and raw data must be backed up on CDs using CD writer. Each CD
must be clearly labeled about site/sector number, date, test type, engineer etc.
Network Acceptance data collection will be accomplished by stationary measurements at

mutually agreed test point under each sector, and mobile measurements by traveling around
in the warranted coverage area. Please refer to Table-1 for specific stationary or mobility
requirement for each test. Each measuring point would represent a typical customer
location. The purpose of mobile data collection is to collect enough statistics about RF
Coverage, Connection success, Access Failures and Handoff Failure rate caused by
network fault, shadowing, terrain and man made obstructions.
4.1 Network Setup

This section describes the network setup that is needed for Vesper EV-DO Network and
application Performance Acceptance Test.
Please refer to section 4.2 for corresponding traffic model for each of these application.
Qualcomm CAIT-2.9 tool will be used for Forward Link Data collection. Net-iQ Chariot tool
will be used to emulate various applications with associated traffic models. Windump and
Snoop logging (TCP trace logs) will be used for Packet Loss (SER) measurements. Please
refer to Appendix-A for required logs for each test.
Figure below gives the ATP Network Setup diagram. An application server is connected to
the PDSN. In order to perform the ATP Test in a controlled environment, the Application
server should be connected in the same subnet as the PDSN. This will prevent unpredicted
traffic over a LAN to introduce undesired RNC
performance degradation. Application sever shall
be used for transmitting the data on Forward link and to simulate application traffic using
Chariot tool (Performance Endpoint). Simultaneously, Access Terminal, along with the EVAP (Qualcomm CAIT 2.9) and Chariot Console will cover various
DO data collection tool
measurement points around the network uploading and downloading the application data
PDN will be
from the application server. Traffic based on the call model for each application
simulated using Chariot Tool. Due to the limitation on the data collection tools, Reverse link
and forward link testing will be performed separately and PDSN
will be counted as separate tests.
Various RF and application related information such as Ec/Io, data throughput, Packet Loss
Rate, etc. would be captured during RF data collection process.
AP
EV-DO TEST Kit
with AT &
Chariot
APPLICATION
SERVER &
Chariot

Page 17
Chariot Console PC
Radio Network Controller (RNC) Logs will be collected for reverse link information as
required.
Chariot console PC will be used to run the Chariot scripts. Snoop logging will be conducted
via Telnet or directly at the application server.
Following are the basic tool requirements to conduct ATP Test:
a) Network setup with application server as mentioned above
b) Data collection Tools: CAIT-2.9 with entire peripherals and GPS unit installed and
calibrated
c) Calibrated Access Terminals: Qualcomm Chesters units with enabled receive diversity
d) Calibrated LG LST-D400 terminals
e) Application emulation tool as Chariot with 200 license capacity
f)
Post Processing Tools: Nortel Networks Scripts, AT Analyzer Tool
g) Updated Cell site Database

h) Laptop computers for data collection/Shakedown: >1GHz, 256MB RAM 20GB of hard
drive
i)
Desktop computers for data processing/Analysis: >1.6GHz, 256MB RAM, 40 GB of hard

drive with 21 monitor
j)
Planet or iPlanner RF propagation tool
k) Data storage/backup units such as CD writer

l)
Drive test vans/drivers
m) Data transfer facilities: LAN
4.2 Traffic Models For System Acceptance

Chariot tool will be used for various applications emulation, which can model the transfer of
data between test mobile and an in-network application server. Chariot can perform simple
sequences of data exchanges between the endpoints (AT and application server). Average
packet call values will be used as the input to the Chariot tool.
The basic unit of testing in Chariot tool is the timing record. Chariot records statistics for a
number of timing records, each record being a single data point. Therefore, one timing
record should correspond to a packet call in 3GPP2 framework. Reading time between
packet calls can be fixed, and only large enough to ensure connection has timed out (~10
seconds).
Following are Vesper suggested call models for Single User Throughput Performance
Acceptance Tests for five applications:

Page 18
4.2.1 Forward Link Throughput Call Model

FTP:
HTTP:
Email:
NRTV:
IPSec:
Each timing record is one file of size 10 MB

Each timing record contains 6 separate transfers an initial one of
10.7 kB and 5 subsequent ones of 7.8 kB each. All transfers use
HTTP-1.0 with no delays between each transfer.
Each timing record is a single email download of size 100 kB
Source rate of 500 kbps, as in 3GPP2. Each file is 17 MB.
Same as FTP test with VPN set up.
4.2.2 Reverse Link Throughput Call Model

FTP:
Email:
IPSec:
Each timing record is one file of size 1 MB

Each timing record is a single email upload of size 50 kB.
Same as FTP upload test with VPN set up.
Application Throughput results are directly dependent on its Traffic Models. Application
throughput will be lower than the raw physical layer throughput for a number of reasons.
Following are some considerations in relating the physical layer and application layer
throughputs.
Average physical layer throughput is approximately equal to the average DRC

Overhead due to TCP/IP/RLP of approximately 20%
AT originated connection setup time of approximately 500-1000 msec.
TCP slow start, forward link scheduler inactivity, and spurious retransmission on
forward link due to reverse link rate control may decrease throughput.
It is assumed that Chariot measures throughput based on data transferred between
connection request time and the time of final packet delivery.
4.3 Measurement points Selection

Acceptance Tests will be divided into following sections:
1. Mobility Measurements for RF Performance Tests
2. Stationary Measurements for Application Performance Tests
3. Mix of mobility and stationary measurements for supplementary tests
Mobility measurements should cover main roads, residential and office areas where
customer traffic is expected to be high. The objective is to gather a bigger sample of test
data for better reliability in less time.
Similarly, for stationary measurements, test measurement points should be identified using
coverage for each sector. To complete the test within the specified time, there will be one
measurement per sector. Each measurement will comprise of conducting more than three
application sessions as defined in the traffic models. Measurement point location can vary
within the warranted coverage area for each sector provided it meets the RF requirement of


Page 19
Ec/Io > -2 dB. This will provide a realistic view to validate Nortel Networks performance
numbers and measure network performance.
Emphasis will be placed on residential and office areas, around main streets where
customer traffic is expected to be high. If there is no access to a particular measurement
point, nearest available location will be used for measurement.
Nortel unwarranted exclusion areas should be taken into consideration during drive route
and measurement point selection. Data collected over any exclusion zone would be
excluded from the analysis and performance assessment.
Appendix-E form will be used during data collection process.


Page 20
5. VESPER ACCEPTANCE TEST PROCEDURES

This section describes various test procedures for 1x EV-DO RF performance measurement.
5.1 RF Performance Tests (SMP1/SP11, SMP2, SMP3, SP9, SP10)

As the name suggests, this test will verify network coverage performance. There will be no
coverage performance measurement on per site basis. Per site coverage information will be
deduced from system wide performance.
This test will be further divided into two parts, each for forward and reverse link separately.
Testing will be accomplished by generating a FTP call while moving around the expected
coverage area in the network.
5.1.1 Forward Link Coverage Performance Test

This test will be used to verify the forward link system performance. This test will measure
following:
Forward Link system coverage
Forward Link single user FTP throughput across the network
Forward link Packet Error Rate across the network
A laptop computer running the CAIT-2.9 tool through Access Terminal will be used for FTP
download purpose. Application server will be used for FTP. A sample file of 500MB will
reside at this server for FTP purpose.
a. Turn the AT off to allow any session associated with it to be released, and allow a
new set of statistics to be generated when the AT requests a new session. This
will also configure the AT to open new connections in variable reverse rate mode.
b. Go to the starting point.
c. Configure the CAIT log mask for forward link measurements
d. Start the CAIT logs
e. Start FTP by invoking a connection setup request. Start downloading a 500 MB
file.
f.
Drive around the expected coverage area following the drive route.
g. Watch the connection state, signal strength and throughput.

h. Re-start the FTP downloads if the original session is completed before
completing the drive run.


Page 21
i.
Stop logging at the point where either link fails or a data call cannot be setup
anymore i.e. outside the cell coverage area.
j.
Note down any coverage holes, coverage boundary and time to complete each
download.
k. Stop and save the CAIT log.

l.
Repeat the process thus covering the complete coverage area.
m. Fill out the drive test form as in Appendix-A

Following procedures will be used to obtain the information from AT logs.
5.1.1.1 System Coverage Results

System coverage will be evaluated through the areas where the Access Terminal
was able to sustain a data call. Alternatively, areas where the AT could not sustain a
call irrespective of the data rate will be qualified as no coverage or exclusion areas.
Individual cell coverage information will be deduced from the system coverage map.
5.1.1.2 Forward Link single user FTP throughput

Forward link single user throughput will be measured by dividing the total time that it
takes to download the file to the size of this file.
FTP Throughput Rate (kbps)
FileSize(kbps)
Total time for download (sec.)
5.1.1.3 Forward link Packet Error Rate

Forward link Packet Error rate will be measured from the CAIT logs. After CAIT log is
processed with the AT Log Analyzer tool, click on the Average PER button. The
result will be displayed (example below).
Packer error rate is the ratio of error packets to the total packets sent.
5.1.2 Reverse link Performance Test


Page 22
This test will be used to verify the reverse link system performance. The test will measure
following:
Reverse Link single user FTP Throughput across the network
Reverse link Frame Error Rate across the network
A laptop computer running the CAIT-2.9 tool through Access Terminal will be used for FTP
upload purpose. Application server will be used for FTP. A sample 50MB file for FTP will
reside on the CAIT laptop.
b. Go to the starting point.
c. Configure the CAIT log mask for reverse link measurements
d. Telnet to RNC and start the Reverse Link FER measurement script. See section
5.1.2.2
e. Start the CAIT logs
f.
Start FTP by invoking a connection setup request. Start uploading a 50 MB file.
g. Drive around the expected coverage across the network following the drive route.
h. Note down the connection state, signal strength and throughput.
i.
Start the FTP uploads again if the original session is completed before
completing the drive run.
j.
Stop logging at the point where either link fails or a data call cannot be setup
anymore i.e. outside the cell coverage area.
k. Note down any coverage holes, coverage boundary and time to complete each
upload.
l.
Stop and save the CAIT log.
m. Stop the reverse Link FER script. See section 5.1.2.2.

n. Repeat the process thus covering the complete coverage area.
5.1.2.1 Reverse Link single user FTP throughput

Reverse link single user throughput will be measured by dividing the total time that it
took to upload the file to the size of this file.


Page 23
FTP Throughput Rate (kbps)
FileSize(kbps)
Total time for upload (sec.)
5.1.2.2 Reverse link Frame Error Rate

Reverse link Frame Error rate will be measured from RNC counters using the FER
script. The process to obtain reverse link FER is discussed below. First we give the
summary of the process, and then the detailed procedure. The scripts and example
logs and final data are given in Appendix-B.
Summary of the Process to Obtain Reverse Link FER
Use the script logrlfer to get FER statistics from the RNC.
Use the script getrlfer to thin out just the timestamps and FER
measurements.
Log at the xterm to run logrlfer and rename the log to rlfer-measurements.
The final data will be in the file rlfer_data.
Detailed Procedure to Obtain Reverse Link FER

In any UNIX window, on a station that has access to your RNC, enter the following
on the command line:
mkdir fertest_run01
cd fertest_run01
xterm&
Click the mouse inside the new xterm window.
Use <ctrl-middle_button> and select the logging menu item.
Use <ctrl-middle_button> and ensure there is a checkmark beside the logging
menu item.
Dial up a session and open a connection by starting an FTP transfer.
At the RNC, get the UATI for the session by entering:
enable
show 1xevdo session all 1001 200
Note the UATI of your session and enter it in the logrlfer command below. You can
omit leading zeros in the hex value, but you need to include the 0x such as in
0x3f2.


Page 24
Replacing the UATI below with your actual UATI, enter the following on the
command line:
kermit
take logrlfer 0x3F2
When the run is complete, enter <ctrl-c> in the xterm window to stop the script.
Use <ctrl-middle_button> and select the logging menu item to stop logging.
Click the mouse in the old xterm window from which you invoked the new xterm
session.
Enter the following on the command line:
ls -ltr
This should show only one file - the log from the xterm session.
Using the actual filename from the above list, enter the following to rename the file:
mv Xtermfile.123 rlfer-measurements
Run the thinning script to leave just the timestamps and FER measurements:
getrlfer
To convert the file to a datapoint format having timestamps and FER
measurements on one line, separated by tabs, use search-and-replace in MS Word
or other application, and replace line-feeds preceding The Frame Error Rate text
with tab characters. In the MS Word dialog, a line feed is ^p and a tab is ^t.
5.1.3 Connection Drop Test (SP9, SP10, SP11)

Connection Drop test will measure following:
Connection drop rate
Handoff performance
Latency (IP packet round trip delay)
WS-FTP PING application with one continuous connection will be used for this test. Quick
reconnection will be established upon interruption.
A laptop computer running the CAIT-2.9 tool through Access Terminal will be used to run
PING application.

Page 25
b. Set RNC Connection Dormancy timer to 0 seconds for a continuous connection.

c. At the Laptop, Set Ping interval of 1 second with number of pings to 99999.
d. Set ping packet size to 50 bytes.
e. Go to the test run start point
f.
Configure the CAIT log mask for forward and reverse link measurements
g. Take the RNC counter snap shot for reverse link handoff performance. See
section 5.3.3.
h. Start the CAIT logs
i.
Start WS-FTP PING, invoking a connection setup request.
j.
Drive around the network following the data collection route.
k. Each run should be for about 15-20 minutes, to limit the log file size. Thereafter
stop the run and save AT logs.
l.
Stop the Ping application and close the dialup connection. This will force the
connection to close (which has a timeout of 0) and cause traffic messages to be
logged, capturing the statistics for the run.
m. Stop CAIT log

n. Take the RNC counter snap shot for reverse link handoff performance after the
test. See section 5.1.3.3.
o. Repeat the process from point e.
5.1.3.1 Connection Drop Results

Connection Drop Rate will be measured as the ratio of the unintended idle time to the total
active state call time.
Once all the performance data is collected Nortel Networks AT Analyzer tool will be used to
process the AT log and calculate the Connection Drop Rate.
Connection Drop Rate (%)
IdleTime
Total Call Time
The following picture shows an example of the Connection Drop Rate window in AT
Analyzer tool.


Page 26
5.1.3.2 Latency Results

Average ping turn-around time from the ping application will be used for latency
measurements.
5.1.3.3 Forward link sector switching delay

Data collected for Connection drop rate will be used for calculating the handoff failure rate.
Forward link handoff failure will be measured through sector switching delay.
Process the forward link data through Nortel AT Analyzer tool. The sector switching
information for each incident will be displayed.
5.1.3.4 Reverse link handoff failure

In 1xEV-DO, reverse link soft/softer handoff is initiated by AT sending a RouteUpdate
message to request the addition or removal of pilots to its active set. Each RouteUpdate
message is counted as a handoff attempt regardless of the number of pilots in the message.
A handoff is considered to be successful if all the requested pilots are added (or removed).
The handoff is considered failed even if only one of the pilots is not successfully added (or
removed) for any reason.
This metrics measures soft and softer handoff performance.


Page 27
The Reverse Link Soft Handoff Success Rate measurement process is as follows:
Telnet to RNC.
At the RNC, record the UATI of the session using the following command:
Check the UATI number corresponding to the test ATs
Get the initial RNC counter connection statistics for the test AT.
In the Privileged EXEC Mode, type:
AIRVANA-07#show session 0x3FF 1xevdo counters all
Where 0x3FF is the UATI (example)
Setup the laptop and the AT. Start the laptop and the AT. Using the 1xEV-DO Dialup
icon, connect/login to the network.
Start the CAIT log and test as given in section 5.2.1
At the end of drive test, stop the FTP and the CAIT logs.
Get the final RNC counters.
AIRVANA-07#show session 0x3FF 1xevdo counters all
Data Analysis
Reverse Link Soft Handoff Success Rate is defined as:
Reverse Link Soft Handoff Success Rate = Rev. Link SHO Successes / Rev. Link SHO
Attempts
The Reverse Link Soft Handoff Success Rate is calculated from the RNC counters as
follows (from the measurement above):
From the initial RNC counters, find out the following numbers:
o Reverse Link Soft Handoffs:
Attempts
Successes
Find out the same numbers from the final RNC counters.
Subtract the initial numbers from the final numbers to obtain the statistics just for the test
session.
======================================================
Statistics
Reverse Link Soft Handoffs :Attempts
Successes
: 13
: 13
======================================================
Example of RNC counter statistics
Example using the numbers above:

Page 28
Reverse Link Soft Handoff Success Rate = 13/13 = 100%
5.1.4 Access Failure Test (SP8, SP11)

Access Failure test is very similar to the connection drop/system coverage test and will
measure AT Originated Access Failures. WS-FTP PING application with multiple short
connections will be used for Access Failure measurements.
Laptop computer running the CAIT-2.9 tool through Access Terminal will be used to run
PING application.
b. Set RNC Connection Dormancy timer to +2 seconds for multiple short
connections.
c. At the laptop, Set Ping interval of 15 second with number of pings to 99999.
d. Set ping packet size to 50 bytes.
e. Go to the test run start point
f.
Configure the CAIT log Mask for forward link measurements
g. Start the CAIT logs

h. Start WS-FTP PING invoking a connection setup request. This will record the first
initial set of access statistics, which need to be subtracted from the final set,
generated at the end of the run, to obtain the stats for this run.
i.
Drive around the network following the data collection route.
j.
Connection Releases and Access Attempts will be logged by AT and AN, which
will capture messaging for individual attempts as well as store cumulative
statistics.
Note:
1. AT may loose the statistics upon power down

2. AN will loose the statistics if the session is released
k. Each run should be for about 15-20 minutes. Thereafter stop the run and save
the logs.
l.
Stop the PING application. This will cause traffic messages to be logged,
capturing the stats.
m. Stop and save the CAIT log.

n. Repeat the process from point e.


Page 29
5.1.4.1 Access Failure Results

Ping statistics results will be used for Access Failure Rate measurements. Record the Ping
statistics at the end of each run like the example below.
Access Failure Rate (%)
Number _ of _ lost _ pings

Total Ping Attempts
5.2 Application Performance Tests

Application performance tests will verify application performance with associated traffic
models. Application performance testing will be based on stationary measurements under
each sector where Ec/Nt is better than 2.5 dB (i.e. Ec/Io > -2dB), to validate the performance
numbers as committed by Nortel Networks. Application simulation will be performed through
Chariot tool. Following parameters will be calculated:
Single User Throughput
Packet Loss (TCP Segment error rate)
A single AT connected to EV-DO Data collection tool (Qualcomm, CAIT) and Chariot Client
will be used for these tests. In addition Windump or Snoop logs will also be logged for TCP
Segment error rate (Packet Loss) measurement at the data transmitter side.
Chariot tool will be used for Average Single User Throughput and TCP trace logs will be
used for Average packet loss and Latency measurement. For UDP application (NRTV),
latency will be measured through Chariot tool.
Following section describes the test procedures for performance measurement for each
application. Traffic models associated to each Application are defined in section 4.2.
5.2.1 File Transfer Protocol (SP1)

The objective of this test is to verify average Single user Throughput, Packet Loss Rate
(TCP Segment Error Rate) and Latency (TCP RTT) measurements using FTP application.
This test will be performed separately for forward and reverse link.


Page 30
5.2.1.1 Forward Link FTP Measurement

Laptop computer running the Chariot and CAIT-2.9 through Access Terminal would serve as
Client. The Application server running NetIQs Chariot tool will be used to emulate the FTP
Download application.
a. Go to the measurement point under serving sector
b. Power cycle the Access Terminal
c. Open connections in variable rate mode to the AT
d. Start the CAIT log (Alt L)
e. Co-ordinate and start the TCP Snoop lop on the Application server
(transmitter side) by telnet through Chariot Console. Direct snoop logging
at the application server can also be performed.
f.
Start the Chariot Forward Link FTP script at the Chariot Console and run
for five (5) complete transactions.
g. This will start the traffic activity at the CAIT

h. At the Chariot Console: Wait for the Chariot script to complete. When
completed, save the Chariot results and export in text format.
i.
Stop and save the snoop logs (Ctrl C)
j.
Stop and save the CAIT log (Alt L)
k. Note down the connection duration, TX and RX power and location

information
l.
Go to the next measuring point (sector) and repeat from point c until all
the measurement points in a zone are completed
m. CAIT log along with the Chariot output will be used for FTP performance
analysis
The Chariot test results will be used to determine average Single user Throughput. This will
be directly calculated from Chariot. Please refer to Chariot user manual for calculation
details. Snoop logs will be used to determine the TCP Segment Error Rate for Packet Loss
and Latency.
5.2.1.2 Reverse Link FTP Measurement
For the reverse link test, the Laptop computer running the Chariot and CAIT-2.9 through
Access Terminal would be used to emulate the FTP Upload application and serve as Client.
FTP traffic call model as defined in section 4.2 will be used.
a. Go to the measurement point under the serving sector
b. Power cycle the AT

Page 31

e. Start the TCP Windump log on the laptop (transmitter side)
f.
Co-ordinate and start the Chariot Reverse Link FTP script at the Chariot
Console port and run for five (5) complete transactions.

i.
Stop and save Windump logs (Ctrl C)
j.
Stop the CAIT log (Alt L)

information
l.
Go to the next measuring point (sector) and repeat from point c until all
m. CAIT logs along with the Chariot output will be used for FTP performance
analysis
The Chariot test results will be used to determine the average Single user Throughput. This
will be directly calculated from Chariot. Please refer to Chariot user manual for calculation
details. Windump logs will be used to determine the TCP Segment Error Rate for Packet
Loss and Latency.
5.2.2 Hyper Text Transfer Protocol (SP2)

The objective of this test is to verify average Single User Throughput, Packet Loss Rate
(TCP Segment Error Rate) and Latency (TCP RTT) measurements using HTTP application.
Client. The Application server running NetIQs Chariot tool will be used to emulate the HTTP
application (fwd link).
n. Co-ordinate and start the TCP Snoop lop on the Application server
(transmitter side) by telnet through Chariot Console. Direct snoop logging
at the application server can also be performed.


Page 32
e. Start the Chariot Forward Link HTTP script at the Chariot Console and run
for fifteen (15) complete transactions.
f.
This will start the traffic activity at the CAIT
g. At the Chariot Console: Wait for the Chariot script to complete. When
h. Stop and save the snoop logs (Ctrl C)
i.
j.
Note down the connection duration, TX and RX power and location

information
k. Go to the next measuring point (sector) and repeat from point c until all
l.
CAIT log along with the Chariot output will be used for HTTP performance
analysis
and Latency.
5.2.3 Email (SP3)

The objective of this test is to verify the average Single User Throughput, Packet Loss Rate
(TCP Segment Error Rate) and Latency (TCP RTT) measurements using Email application.
This test will be performed separately for forward and reverse link.
5.2.3.1 Forward Link Email Measurement

Client. The Application server running NetIQs Chariot tool will be used to emulate the Email
Download application.
e. Co-ordinate and start the TCP Snoop lop on the Application server
(transmitter side) by telnet through Chariot Console. Direct snoop
logging at the application server can also be performed.


Page 33
f.
Start the Chariot Forward Link Email script at the Chariot Console and
run for ten (10) complete transactions.

i.
Stop and save the snoop logs (Ctrl C)
j.

information
l.
Go to the next measuring point (sector) and repeat from point c until
all the measurement points in a zone are completed
m. CAIT log along with the Chariot output and Snoop logs will be used for
FTP performance analysis
and Latency.
5.2.3.2 Reverse Link Email Measurement
For the reverse link test, the Laptop computer running the Chariot and CAIT-2.9 through
Access Terminal would be used to emulate the Email Upload application. While the
Application server will serve as Client. Email traffic call model as defined in section 4.2 will
be used.
a. Go to the measurement point under the serving sector
b. Power cycle the AT
e. Start the TCP Windump log on the laptop (transmitter side)
f.
Co-ordinate and start the Chariot Reverse Link Email script at the
Chariot Console port and run for ten (10) complete transactions.

i.
Stop and save Windump logs (Ctrl C)
j.
Stop the CAIT log (Alt L)


Page 34

information
l.
Go to the next measuring point (sector) and repeat from point c until
m. CAIT logs along with the Chariot output will be used for Email
performance analysis
details. Windump logs will be used to determine the TCP Segment Error Rate for Packet
Loss and Latency.
5.2.4 Near Real Time Video (SP4)

The objective of this test is to verify the Packet Loss Rate and Latency measurements using
NRTV application. To simulate NRTV, a real multimedia stream with UDP will be used.
Client. The Application server running NetIQs Chariot tool will be used to emulate NRTV
application (fwd link).
e. No snoop log will be needed since it is a UDP based application
f.
Start the Chariot Forward Link NRTV script at the Chariot Console
and run for one complete transaction.

completed, save the Chariot results and export in verbose HTML
format.
i.
j.
Note down the connection duration, TX and RX power and location

information
k. Go to the next measuring point (sector) and repeat from point c until
l.
CAIT log along with the Chariot output will be used for NRTV
performance analysis


Page 35
The Chariot test results will be used to determine Single user Throughput, latency and lost
data information. Please refer to Chariot user manual for calculation details.
5.2.5 File Transfer Protocol using IPSec (SP5)

This test would involve the FTP performance measurements when data is transferred over
an IPSec secure tunnel. Same FTP test setup as defined in section 5.2.1 will be established
using the Contivity Virtual Privet Network. IPSecturity will be achieved using the Virtual
Privet Network.
File Transfer Protocol application will be emulated with Chariot tool. Same call model and
test procedure as for FTP (5.3.1) will be used.


Page 36
6. SUPPLEMENTARY TESTS
Two of Supplementary Tests (SPM2 and SPM3) are covered while performing Performance
Acceptance tests as explained before.
Following additional supplementary tests that are to be tested for information purpose only
with no pass or fail criteria for Vesper ATP:
Supplementary Application Test - 3 Simultaneous applications

AT Terminated (AN originated) Connection Success rate
Maximum simultaneous users per sector capacity test
Idle handoff
System Packet/Frame Error Rate
Reverse link power control messages
DRC statistics
Session success and drop rate
To meet the Vesper Supplementary Tests requirement, following additional tests will be
required:
6.1 Supplementary Application Test (SP6)

Supplementary Application Performance test setup is basically the same as in the
Acceptance test case, except that in this test, the application performances will be assessed
in mobile environment rather than stationary measurements. Data collected while all three
applications run simultaneously in the warranted coverage area.
Traffic model (Fwd link) as defined in section 4.2 will be used for each application. Following
three applications will be run.
a. File Transfer Protocol (FTP)
b. Hyper Text Transfer Protocol (HTTP)
c. Near real Time Video (NRTV)
The objective of this test is to assess the application performance while three applications
are run simultaneously in a mobile environment.
Following parameters will be measured for each application:
Single user Throughput

Packet Loss Rate (TCP Segment Error Rate)
Latency (TCP Round Trip Time)
Three separate laptops running Qualcomm CAIT and Chariot tool will be used for this test.
Same procedure as explained in section 5.3 will be used.


Page 37
6.2 AT Terminated Connection Success Rate (SP7)

This test will verify 1xEV-DO connection setup based on data being forwarded to the AT
from the AN, instead of connections being opened because of data transmissions originating
from the AT side. Ping will be used to stimulate connections to open, and allow enough time
for the connections to close again after the ping replies are sent back.
The same setup as in section 5.1.1 will be used for this test. CAIT log will be collected while
driving around the network.
The Connection Success Rate will be calculated by comparing the number of Page
Requests sent to the AT versus the number of successful connections opened, using the
difference between the session statistics recorded at the beginning of the test and again at
the end.
Procedure:
1. Establish a dialup session and record the IP address.
2. Use ping to send packets to the AT at 30 second intervals. For example, if the IP address
obtained in step 1is 47.135.201.185, use the following command in UNIX:
ping I 30 47.135.201.185
Where I specifies the interval.
3. At the RNC, record the UATI of the session using the following command:
Check the UATI number corresponding to the test AT
4. Using the UATI recorded in step 3, enter the following command to record the connection
statistics from the RNC counters at the start of the test:
show 1xevdo session 0x3FF counters all
Where 0x3FF is the UATI (example)
5. Record the following values:
Page Requests from RNC
Successful Opens
6. At the end of the test, repeat steps 4 and 5. Subtract the end values of Pages and Opens
from the start values. Calculate the Connection Success Rate as follows:
ConnectionSuccessRate
TotalSuccessfulOpens
TotalPage Re questsSentToAT


Page 38
6.3 Maximum Simultaneous Users per Sector Capacity Test (SP12)

The objective of this test is to verify the maximum number of active users that can be
simultaneously connected to a sector with single DO Module (DOM). This will be a localized
one-time test in a lab environment or a single sector in the network. FTP application will be
used for this test. 24 ATs with Laptops in the warranted coverage point of a sector will be
used.
About five Laptop computers running CAIT-2.9 tool will be used to monitor the RF condition
response and its effect on the AT DRC with successive sector loading.
a. Open connections in variable rate mode to the ATs
b. Start the CAIT logs on the laptops with CAIT
c. Start the FTP session on laptops with CAIT
d. Successively activate other laptops connected through the ATs. Start the
FTP session with Network Dialup through ATs.
e. Note down the maximum number of active ATs
Save the CAIT log files
6.4 Idle Handoff Test (SP13)

This test will show that AT can do idle handoff. This is a one-time test in a localized
warranted area to show idle handoff functionality. AT will not run any application, and hence
made to go idle state. Then collect CAIT log while driving along the sample drive test route.
During the drive testing, monitor the following in CAIT screen:
Active set PNs in the 1xEV Pilot Sets window
Verify that PNs are changing as the AT moves from sector to sector.
Save the CAIT log in case it is needed for playback later.
Note: The serving sector PN in 1xEV AT Status window may not change as the AT moves
in idle mode, because the forward link handoff only happens when the AT is in Connected
state.
6.5 Supplementary RF Performance Test (SP14, SP15, SP16, SP17, SP18)

This test will be used to measure following performance parameters:
Reverse Link Power Control Messages
Session Drop and Success rate
DRC Statistics


Page 39
System Packet/Frame Error rate
Same test setup as in section 5.1.1 will be used but in a mobile environment around the
warranted coverage area in the network. CAIT and RNC performance counters will be used
with FTP download application from the server.
6.5.1 Reverse Link Power Control Messages (SP14)

To look at the reverse link power control messages, process the CAIT log with AT Log
Analyzer tool. Then from the AT Log Analyzer main menu, press the Reverse Power
Control button. The following window will be displayed:
6.5.2 Session Success and Drop Rate (SP15, SP16)

The following is the procedure to measure the 1xEV-DO Session Setup Success Rate. An
AT must establish a 1xEV-DO session with a DO-RNC before the AT is allowed to access
the 1xEV-DO network. During the 1xEV-DO session setup process, an UATI is assigned.
Configuration is also negotiated. A PPP session setup will also be attempted. However, the
PPP session setup performance is outside the scope of this measurement.
The session setup success rate measurement process is as follows:
Telnet to RNC.
Set up the RNC Connection Inactivity Timer to 2 s.
In the Global Configuration Mode, type:
AIRVANA-07(config)#1xevdo connection inactivity-time 20
Get the initial RNC counters.
AIRVANA-07#show 1xevdo counters all


Page 40
Setup the laptop and the AT. Start the laptop and the AT. Using the 1xEV-DO Dialup
icon, connect/login to the network.
Run CAIT, setup the appropriate logmasks.
Wait until the AT is idle, then start the CAIT (AT) log.
Start the Ping application in WS-FTP Pro using the following parameters:
o Host: IP address of the FTP server
o Packet count = 99999
o Packet size = 50
o Delay = 15 s
Do the following in CAIT:
o Check the 1xEV AT Status window and verify that the AT state is changing
between connected and idle and the session state is open.
o Monitor the 1xEV Pilot Sets window and verify the AT is served by the right
sector.
At the end test period, stop the Ping and the CAIT logs.
Get the final RNC counters.
AIRVANA-07#show 1xevdo counters all
Change the RNC Connection Inactivity Timer to the original value.
Data Analysis
1xEV-DO Session Setup Success Rate is defined as:
Session Setup Success Rate = Session Setup Successes / Valid Session Setup Attempts
The Session Setup Success Rate is calculated from the RNC counters as follows (from the
measurement above):
From the initial RNC counters, find out the following numbers:
o (Total) Session Setup:
Attempts
Successful
o Sessions Terminated:
Receiving UATI Request
Find out the same numbers from the final RNC counters.
Subtract the initial numbers from the final numbers to obtain the statistics just for the test
session.
======================================================
Statistics
Num Current Sessions Established

: 3

Page 41
Num Active Registered Sessions
: 2
Num Dormant Registered Sessions
: 1
Num Sessions Awaiting Close from AT
: 0
Session Setup:Attempts
: 26
Successful
: 23
...
Sessions Terminated :Keep Alive Timeout
: 13
AT Close
: 0
Receiving UATI Request
: 0
======================================================
Example of RNC counter statistics.
Example using the numbers above:
Session Setup Success Rate = 23/(26-0) = 88.5%
Session Drop rate = 1 Session Success Rate
6.5.3 DRC Statistics (SP17)

To look at the DRC statistics, process the CAIT log with AT Log Analyzer tool. Then from the
AT Log Analyzer main menu, press the DRC Index Statistics button. The following window
will be displayed:


Page 42


Page 43
7. DATA COLLECTION TOOLS

This section provides a general procedure to install the Qualcomm Chesters AT and setup
the laptop prior to performing 1xEV DO test cases stated in this document. The section also
describes the procedure to use the Qualcomm CAIT Tool.
7.1 Installing Qualcomm USB Driver

1. Get the up-today version of the Qualcomm Chester USB driver. Need to get the right
version for Window 2000 or Window XP.
2. There are four files in the Qualcomm Chester USB Driver folder. Put the folder
anywhere in the C Drive
3. Power up the PC/LapTop and Power up the AT (Chester) and connect the USB
cable to the AT and to the PC/LapTop.
4. After the USB cable is connected to both PC/LapTop and AT, MS (MicroSoft)
Window will detect the new hard ware and MS Window will pop up Welcome to the
Found New Hardware Wizard window. Then click Next.
5. Then select the directory that has the four files mentioned above. Then click Next.
6. Then the MS window will display Driver Files Search Results window as indicated
below.


Page 44
7. Click Next; the Digital Signature Not Found is displayed. Then click Yes.
8. Then the software driver will be installed. There are two files requiring to be installed.
9. After the software driver is installed. The four mentioned files are to be resided in the
path: C:\WINNT\system32\drivers.


Page 45
7.2 Setting up Window2000 PCs 1xEV-DO Dialup
Turn the power on for the test laptop/PC. The test laptop should be running the
Windows2000 operating system.
Ensure that the appropriate USB driver for the AT that is being used is installed on the
test laptop/PC.
Insert the CAIT key/hardware WIBU-key onto the parallel port of the test laptop/PC.
Insert the USB cable connecting the AT to the laptop into the USB port on the test
laptop/PC. Connect the other end of the USB cable to the ATs USB port.
Set up a dial-up networking template/icon using normal Windows2000 operating

commands on the test laptop/PC. The procedure for installing the Qualcomm USB
drivers can be found in Qualcomm document 80-B1311-1, QTP-5500 Access Terminal
Users Guide. Below is the description of how to configure the network connection.
o
o
Go to Start > Settings -> Network and Dial-up Connections -> Make New
Connection
The Network Connection Wizard will be displayed, as shown Figure A1 below.
Figure A1: Network Connection Wizard
Select Next.
Select Dial-up to private network, and chose Next as shown in Figure A2 below.


Page 46
Figure A2: Network Connection type window
Select the appropriate modem for the Qualcomm USB 1xEVDO AT, and select
Next as shown in Figure A3 below.
Figure A3: Select a Device window
Under Phone number: enter #777, and ensure that Use dialing rules is NOT
checked. Then select Next as shown in Figure A4 below.


Page 47
Figure A4: Phone Number to Dial window
Select For all users, under Create this connection, and select Next as shown in
Figure A5 below.
Figure A5: Connection Availability window
Type a name for the network connection, and add a shortcut to the desktop.
Then select Next to close the connection wizard as shown in Figure A6 below.


Page 48
Figure A6: Completing the Network Connection Wizard window
Double-click on the dial-up networking icon that was created in the previous
steps, and you will see Dial up window popped up as shown in Figure A7 below.
Then Click on Properties.
Figure A7: Dial Up window
Confirm the number to be dialed (#777) is correctly entered in the

General tab under phone number.
Select the correct modem for the Qualcomm USB port and uncheck any
other modems.
Select the modem and click configure as shown in Figure A8 below.


Page 49
Figure A8: General configuration window
Ensure that all boxes are unchecked, and that the maximum speed is set to
115200. Select OK to return to the previous screen as shown in Figure A9
below.
Figure A9: Modem Configuration window


Page 50
Under the Options tab, check the following options as indicated in Figure A10
below):
Figure A10: Option Tab window
Under the Security tab of your dial-up networking connection, ensure that the
security options are set as indicated in Figure A11 below:
Figure A11: Security Tab window
Under the Networking tab, highlight Internet Protocol (TCP/IP) and

chose Properties as shown in Figure A12 below


Page 51
Figure A12: Networking Tab window
Ensure that no IP address or DNS server is specified, and the configuration is

similar to the one shown below. Select OK to return to the previous menu as
shown in Figure A13 below.
Figure A13: General Tab via Internet Protocol (TCP/IP) Property window


Page 52
Select Settings, and ensure that the settings are similar to Figure A14 below.
Then click OK.
Figure A14: PPP Setting window
o
o
o
Under the Sharing tab of your dial-up networking connection, ensure that
sharing is turned off as indicated in Figure A15 below.
Select OK to close the window and return to the dial-up adapter window.
Now 1xEV DO dial up is ready to communicate with DO network via AT.
Figure A15: Sharing Tab window of Dial Up Connection Window
7.3 Test Environment

This section provides information on specific parameters used to perform the test cases
Standard Windows-2000 settings will be used, except:
MTU Size: 1500 (decimal)
VJC = Off

Page 53
TCP Receive Window size = 64240 Bytes (decimal)

TCP Time Stamp = ON
7.4 CAIT Startup Procedure

1. Ensure that the 1xEv DO system is up and running.
2. Launch CAIT by double clicking on the CAIT icon on the window desktop.
3. When CAIT is up, go to CAITs main menu and select file --> Load Profile icon
(the predefined Monitoring profile window). See section 7.2 (Creating CAIT
Monitoring Profile) on how to create it.
4. Connect the AT to the computing device (Laptop, PC desktop, etc.)
5. Power up the AT by the on/off switch. The first LED close to the on/off switch should
be green and lit.
6. Check the CAIT 1xEV AT Status window, in the section AT Status, which is
described in Figure A1 below, to ensure that the AT is in idle mode, the DO session
is established (i.e. Open) and that a UATI is assigned to the AT. For example, the
following parameters should be: AT state = idle, Session state = Open, UATI024 =
0x24041a (the 0x24041a is the assigned UATI identification number. The
0x24041a used here is just an example assigned number). Additional information is
also available in this window such as channel and band used.
7. Check the Pilot sets window via CAIT 1xEV AT Status window, which is described
in Figure B1 below, to ensure that the AT is locked to the right PN of the tested
sector.
Figure B1: CAIT 1xEV AT Status window
8. Launch the Dialup application to establish a 1xEV-DO connection.


Page 54
9. Enter correct User name and password as shown in Figure B2 below. Then click on
the Dial soft key for network connection.
Figure B2: Dial Up Connection window
10. Once the PDSN has established the PPP session with the AT, check from CAITs
11. AT Status window (Figure B1), which the AT state is connected. If no data is
transferred for some time, the active PPP session will go to Dormant and the AT
state will go back to idle.
12. To ensure that the dial up session is still up check the small network connection
icon at the lower right corner of the Window display.
13. Launch the application (e.g. FTP). Provide the correct server IP address; Enter the
correct User ID and correct Password.
14. Select the desired directory corresponding to the network application server side
that you want to use to FTP a file to the Laptop and select the desired director
corresponding to the Laptop side for receiving a file. This step is for Forward Link
tests with FTP.
15. For Reverse Link FTP tests, select the desired directory corresponding to the Laptop
side that you want to use to FTP a file to the network server and select the desired
director corresponding to the network side for receiving a file.
7.5 Creating CAIT Monitoring Profile

The Monitoring Profile consists of several different information windows that are selected
previously when creating the Monitoring Profile. The information windows to be launched
are dependent on the 1xEV-DO information required, on a per test case basis.
To create a CAIT Monitoring Profile:


Page 55
From CAITs main menu

Select View -> 1xEV-> Select desired window information attributes.
Then select file -> Save <profile name> (for example DO_General).
Figure C1 below is an example of the Monitoring Profile window, which consists of the
typical CAIT information windows that are used for most of the tests.
They are: 1xEV AT Status window, 1xEV Forward Link Statistics window, 1xEV Reverse
Link Statistics window, 1xEV RLP Statistics window.
Figure C1: Example of Monitoring Profile Window
7.6 CAIT Logging Configuration

This section provides some basic information on setting up logging in CAIT.

Page 56
1.
2.
3.
4.
From CAITs main menu, select Option then Configure CAIT.

A CAIT Configuration window will pop up as described in Figure D1.
Select Logging Tab.
Select Log only packet matching the logging mask in the Logging mask usage
section.
5. Select MDM style (mDDHHMMl.lll) in the Log file naming section
6. Select Directories from the configuration window.
7. Specify the log repository location.
Figure D1: CAIT Configuration window
7.7 Setting CAIT Log Mask

From the CAIT main menu select View Logging Mask Expand the 1xEV Logging.
Then select the appropriate CAIT log attributes.
Click OK.
Save the profile.
This section provides a brief description of some log mask attributes that can be used to
gather useful 1xEV-DO information. Also, four different log mask profiles are proposed, that
can be used for specific test scenarios: Forward Link Throughput testing Log Mask, CallP
testing Log Mask, Reverse Link testing Log Mask, and RLP testing Log Mask.


Page 57
Definitions of CAIT log Attributes
Handoff State
Mobility testing only, Data generated only when changes in Active or Candidate set.
Strength of the pilot(s) in the active set also includes(s).
Active Pilot Set
This supplies the details of the serving pilots for the AT, the Active Set Pilots, the DRC cover
& SNR of the serving pilot.
Data Rate Control Channel ARQ Buffer
DRC rate requested by the AT. DRC cover (i.e. which sector is active on the FL) ACK/NAK
and if ACK/NAK enabled on the RL ACK channel.
ARQ Effective Receive Rate
Provides Rate at which the packet (physical layer) was decoded. DRC value requested and
the number of slots required to receive the packet at the requested rate. A physical layer
packet can contain from 1 to 4 MAC packets depending on data rate. The number of MAC
packets per frame depends on how many bits are to be sent during that frame. For example
for 38.4 only 1024 (including header bits) is needed so only one MAC packet per physical
layer frame. For 2457.6, 4096 bits are needed so there are 4 MAC packets per physical
layer frame.
Forward Traffic Rate Slot Count
Can create a histogram of what rate traffic channel packets were received and the number
of slots required to decode that packet.
Air Link Summary
PERInst - Instantaneous Packet Error Rate (calculated once 16 packets are received).
Shows PNs of pilots in the Active set.
FWD Rate Stat
Forward Rate Statistics CC & TC count & TC PER (physical layer packets). Also, shows
AvgFwdThroughput (always shows zero at this time).
Note: TC307200Long_Good, 1863,, is an excerpt from the log. The 1863 value is in units of
slots. Since this a 307200 DRC it requires 4 slot per MAC packet (in this case 1 MAC per
Physical). The number of slots per data rate needs to be factored in when determining the
number of physical layer packets between log timestamps.
Session Attempt
This holds a record containing Session attempt information. Counters are reset when AT is
power cycled.
Access Attempt
This contains access probe sequence information. Attempts, Success, and Failure counters
are reset at start of session.
Connection Attempt


Page 58
This contains the connection attempt information such as traffic channel allocation
success/failure and the reason for such. Counters are reset when AT is power cycled.
Page Message
This provides the statistic for the number of page messages received by the AT. Page
message is received on Control Channel. Counters are reset when AT is power cycled
session.
Connection Release
This represents the reason for a call termination.

Reverse Activity Bits Buffer
This logs the reverse activity (RA) bits received for all pilots in the active set. Data for this
packet is sampled one every slot (1.67 ms). The data is logged once every PN roll (16 slots)
when in Connected State.
Reverse Activity Bits Information
This provides information about the reverse activity bit (RAB). The information given here is
obtained from the Traffic Channel Assignment Message.
Reverse Link Packet Summary
This provides a summary of the reverse link packet at the level of the physical layer. Every
frame that is sent out at the AT is logged. Turning on this log may result in a drop in other
log packets.
Reverse Traffic Rate Count
This contains the number of reverse link traffic packets transmitted at various rates. The
data is logged every 1 sec. Counters are reset at the start of the session.
RLP Statistics
This contains information relevant to the transmit and receive chains of the RLP stack.
Counters are reset at start of session.
RX Partial RLP
This contains RX partial RLP logging information. The data is logged asynchronously every
RLP packet that is received. The log is sent when the log buffer is full.
TX Partial RLP
This contains TX partial RLP logging information. The data is logged asynchronously every
RLP packet that is transmitted. The log is sent when the log buffer is full.
Logging mask profiles
Forward Link Throughput Testing Log Mask
Handoff State
Active Pilot Set
Data Rate Control Channel ARQ Buffer
ARQ Effective Receive Rate
Forward Traffic Rate Slot Count
Air Link Summary


Page 59
FWD Rate Stat

Session Attempt
Access Attempt
Connection Attempt
Page Message
Connection Release
Signaling Access Channel
Signaling Reverse Traffic Channel
Signaling Control Channel Directed
Signaling Forward Traffic Channel
Traffic Channel Assignment Info
Signaling Control Channel Broadcast
Power
RLP Statistics
Shake-down/Call Processing Testing Log Mask
Session Attempt
Access Attempt
Connection Attempt
Page Message
Connection Release
Reverse Link Testing Log Mask
Pilot Sets
Reverse Activity Bits Buffer
Reverse Activity Bits Information
Reverse Link Packet Summary
Reverse Traffic Rate Count
Reverse Power Control
Session Attempt
Access Attempt
Connection Attempt
Page Message
Connection Release
Power
RLP Statistics
RLP Testing Log Mask
RLP Statistics
RX Partial RLP


Page 60
TX Partial RLP
Session Attempt
Access Attempt
Connection Attempt
Page Message
Connection Release
7.8 Setting up GPS

This section provides the procedures on how to setup the GPS to works with CAIT.
From CAIT main menu, select Option --> Configure CAIT. Then CAIT Configuration
window is popped up as shown in Figure E1 below.
Then select GPS Setting Tab and set all the fields as indicated Figure E1.
Figure E1: GPS Settings Tab via CAIT Configuration window
Then connect the GPS receiver to the serial Com Port of the Laptop.
Turn the GPS receiver ON. Then we will see the pop menu with Continue display and
then the Acquiring Stats Display.


Page 61
Then we will see the display with Message: Need to Select Init Method. Then we need to
press enter.
Then we need to select which Method option we want to use: Map Method or
Autolocate Method.
If we have electronic map then select Map Method if not then select AutoLocate
Method.
A Globe icon will show up on the low right corner of the Window 2000 display as shown
in Figure E2 below.
Figure E2: Globe icon
Right click on the Globe icon and then select the Communications Option to see the
COM Port menu as shown in Figure E3 below and select all the COM Ports that we
think the GPS receiver is interfaced with one of these COM Ports
Figure E3: GPS Search Configuration window
Then right click on the Globe icon and then select the Raw GPS Data as shown in
Figure E4 below.


Page 62
Figure E4: Raw GPS Data selection
If the GPS receiver detects the Satellites, then we will see the Raw GPS Data shows up
in the Raw GPS Data window as shown in Figure E5 below.
Figure E5: Raw GPS Data window
7.9 Converting the mdm log format for AT Analyzer Tool

The mdm format log collected from the CAIT tool will need to be converted to the raw text
format for AT Log Analyzer tool for post processing purpose. Friendly Viewer is a tool utility
within the CAIT Tool. This section explains the procedure to convert the mdm logs using the
Friendly Viewer Tool:
Select
Start -> Programs -> QCTest CAIT -> Tools -> Friendly Viewer


Page 63
Following window will appear
Figure: F1
Select
File -> Open MSM File
Select the mdm file that needs to be converted to the Raw text format and click
OK
This will open a bart chart and it will take few seconds to read and open the file
Select Log File View -> Configure -> Export Formatting

Following window will appear:
Figure: F2
Deselect Use Default Export Formatting if it is selected


Page 64
Select Raw Data from Insert Formatting Field

Click OK
At this point we have defined the format for the Raw File
Select Log File View -> Export

Give the name and location for the Output Text File
Click OK
A bar chart will appear and it will save the file in few seconds
Select File -> Close

This will close the file.
Repeat from step 2 to convert the next mdm file.


Page 65
8. CHARIOT TOOL
Chariot tool will be used to emulate and measure the performance of following five
applications:
a. File Transfer Protocol (FTP)
b. Hyper Text Transfer Protocol (HTTP)
c. NRTV (Near Real Time Video)
d. Email
e. File Transfer Protocol (FTP) using IPSec
Chariot is used to validate the performance characteristics of a wireless operation. It works
by exactly emulating transaction traffic from real applications.
8.1 Chariot Tool Description

Chariot consists of three main components: Performance Endpoints, Scripts and Chariot
Console:
Performance Endpoints
Endpoints are small software agents that are installs on computers throughout your network
(Laptops with AT and CAIT or Application server), which run as background tasks (no user
intervention required). Endpoints support tests that emulate real users, sending traffic with
multiple data types at variable data rates, and using 6 different protocols. As they mimic the
traffic generated by real users on your network, endpoints gather statistics that measure the
end user's experience. Endpoints are supported on most major operating systems.
Application Scripts
The application scripts generate the actual traffic. Endpoints running application scripts
make the same calls to the network protocol stacks, and produce the same load on the
stacks, as your applications. Application scripts let you test application traffic on your
network before rolling out a new application. They can be extensively customized to fit
unique environment. Based on trace files from actual network transactions, application
scripts can emulate anything from a simple file transfer to a complicated SAP R/3
transaction or streaming multi-media application. Application scripts are paired with a
network transport (such as TCP or UDP).
Chariot Console
The console runs on a WinNT or Win2000 PC. It controls and configures the test setup,
specifies and configures the endpoints involved in the test, collects the test data, and
generates test result reports. It also allows you to view/modify/create application scripts.
This is also where the Chariot license resides.

Page 66
8.2 Chariot Test procedure

The Chariot Console tells the endpoints how to emulate a particular application by sending
them an application script and other test setup information. The script tells the endpoints the
type and amount of data to send and receive, when to connect and disconnect, and what
delays should be used to emulate end-user or application behavior.
The endpoints execute the transactions in the script and measure throughput, and other
performance metrics.
At the end of the test, the endpoints send test results back to the Chariot Console. Results
can be directly used to evaluate network performance or exported to text, CSV, or HTML for
later use.
For Vesper ATP test plan Chariot will be used to simulate five applications as described on
Table-1. Chariot console will reside on a PC within the network to easily access the
application server, PDSN and AT in field. Following procedure will be used for Chariot
logging:
Select
Start -> Programs -> Chariot
Following window will appear.
Select
Open -> Select the location and open the configuration file
There will be a separate script for each application and related forward or reverse
link test. Script will simulated the test application with associated traffic model as
defined in section 4.1.
Open the script file
Once the script is opened, following window will appear:


Page 67
Select
Run ->
At the Pair comment input the correct sign-in user number for the Access Terminal.
Input the correct IP address for the Access Terminal (using ipconfig command at the
AT, see section 9) for Endpoint 1 network address.
Similarly, input the correct IP address for the Application Server for Endpoint 2
network address.


Page 68
Select Network Prococol. TCP for FTP, HTTP and Email. UDP for NRTV.
If no editing is required (since the scripts are predefined based on the application
and traffic model). Click OK
Select
Run -> Set Run Options,
Following window will appear
Select Run until any pair ends and click OK
At this point we are ready to run the script.
Coordinate with the field team.
To star the Chariot script, click the Run icon or select

Run -> Run
Once the Chariot script is completed. Save the results.
Select


Page 69
File -> Save as, and give the correct file name and location
It is also suggested to save the Chariot result in verbose HTML format. Select
File -> Export -> HTML; and save the HTML file
Various application performance results like single user Throughput and can be
directly read from the result file.


Page 70
9. TCP PERFORMANCE LOGGING

Snoop and Windump logs are generated to analyze the TCP performance between the client
and the server. For vesper ATP purpose, these loge will be used to monitor the performance
between AT and the Application server. These logs will be used to calculate the TCP
Segment Error Rate (Packet Loss Rate).
Snoop is a Unix based utility that is run on the Application server (if a Unix based system)
and Windump is a windows utility that will be run at the AT end. Windump and Snoop
logging can be performed from the Chariot console PC itself. Following is the procedure to
install Ethereal, TCP Trace and Windump tools:
9.1 Procedure to Install Windump in the Computer
Download WinPcap version 3.0 from http://winpcap.polito.it/install/default.htm . This is

an install program. You just need to run it and follow along. WinPcap is the packet
capture driver that Windump uses.
Download Windump version 3.6.2 from http://windump.polito.it/install/default.htm . This
doesnt have to be installed. It only needs to be copied into the c:\winnt directory.
Check the properties of the 1xEV-DO dial-up networking connection. Click the
Networking tab. Verify that the Network Monitor Driver is listed in the component list
for your dial-up networking connection, and that it is checked. If it is not, then it needs to
be installed in Windows (as part of the SMS software).


Page 71
9.2 Procedure to Install Ethereal in the Computer

Ethereal software is needed to view the Windump output file (in binary form) and to convert
snoop logs into TCPdump format that can read by TCPTrace. Ethereal is available from
http://www.ethereal.com . Just download install program, double-click it and follow the
instruction.
9.3 Procedure to Install TCPTrace in the Computer
Go to www.tcptrace.org and download TCPTrace for Windows. It is in a zipped file called

win_tcptrace.zip.
Unzip the file and follow the instruction in README.Windows file. There is no install
program for TCPTrace. There are only .exe and .dll files.
9.4 Procedure to Collect Snoop and Windump Logs

Snoop and Windump logs are collected to record TCP packet transmission. They are
collected during the traffic model tests (with Chariot) except for NRTV application which
does not use TCP. They need to be collected at the sender side. Therefore, Snoop needs to
be collected for the forward link tests, and Windump needs to be collected for the reverse
link tests.
9.4.1 Snoop Logs

Before starting the logs, we need to check the IP Address assigned to the AT.
At the laptop connected to the AT: Go to the DOS command
C:\> ipconfig <ENTER>
This will give the IP address assigned to the AT
Snoop is collected at the Application server. The following is the Snoop logging procedure
for the forward FTP, HTTP, and Email tests:
1.
2.
3.
4.
Telnet into the server (e.g., from the Chariot console).

Login as administrator
Go to the directory where you want to save the log.
Type >/usr/sbin/snoop o filename AT IP address <ENTER>
This will start the Snoop log. o here implies Output to
5. To stop and save the log Ctrl C

For the forward FTP/IPSec test, step 4 is different:
4. Type > ifconfig a
<ENTER>
You will get something like this:
lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2

Page 72
inet 47.135.201.197 netmask ffffffc0 broadcast 47.135.201.255

ether 8:0:20:ac:4d:d5
hme1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
inet 10.29.1.2 netmask ffffff00 broadcast 10.29.1.255
ether 8:0:20:ac:4d:d5
Then type >snoop -d hme1 -o filename
<ENTER>
9.4.2 Windump Logs

Following is the Windump logging procedure at the Access Terminal connected to the laptop
for the reverse FTP and Email tests:
At the laptop:
1. login as Administrator
2. Make directory where we want to put the Windump logs (example Windump_Logs)
3. Type c:Windump_Logs\> windump D <ENTER>
4. Above command will display some information. See the figure below:
4. Look for the NdisVan Bh and fine the number in front of it.
(In the example above, it is no.2).
5. Type c:\> windump i# -w filename <ENTER>
# is the number in front of NdisVan Bh
For above example the command would be:
c:\> windump i2 -w filename <ENTER>
This will start the Windump log. w here implies Write
6. To stop and save the log Ctrl C
The Windump log is saved in the specified directory.
For the reverse FTP/IPSec test, in step 5, substitute # with the number of the Extranet
Access Adapter.


Page 73
9.5 Converting Snoop Logs to TCPDump Format

To calculate the TCP Segment Error Rate, we will need to run the TCP log (Snoop or
Windump) through the TCPTrace software. However, for the Snoop files, they first need to
be converted to TCPdump format using the Ethereal tool so that the TCP Trace tool can
post process it correctly. The process is explained below.
Double click on Ethereal.exe, this will open the Ethereal tool. Following window will appear:
Select
File - > Open -> Locate the correct location and Snoop log file to open
This can take few seconds to open.
Once the Snoop file is open, following window (example) will appear:


Page 74
Select
File -> Save As ->
Click on the tab and select Libpcap (tcpdump, Ethereal etc.) as the format for the
output file (TCP Dump file)
Give the file TCP Dump file name and location
Click OK and save the file
The purpose of converting the Snoop log to TCP dump format was required so that it is
compatible with TCP Trace Tool.
9.6 Procedure to Run the TCPTrace Tool

TCPTrace tool is used to parse the TCPdump and the Windump logs. This is to get the
information about the TCP Segment Error Rate and Average TCP Packet Round Trip Time.
This is a DOS based tool.
Start the DOS session and go to the location where TCPTrace Tool is located.
WinNT folder program is used for Windows 2000 or XP environment
Following is an example of this exercise:
C:\TCPTrace\WinNT\
Execute the following command:

C:>TCPTrace\WinNT\tcptrace rl TCPDump-filename_with_location
path (eg.c:\>TCPlogs\TCP Trace log) > OutputFile_name.txt with
location path (eg.c:\>TCPlogs\FwdFTP.txt)
Click Enter
This will generate the output file.

Limit the output and TCPDump file name to maximum of seven characters to run this
program.
Open this file on notepad. The Output file gives the TCP performance information for each
connection individually in the file. Following is an example for one connection information.


Page 75
9.6.1 TCP segment Error rate calculation

The TCP Segment Error rate is collected as follows:
TCP Segment Error rate is the ration of Retransmitted Data to the total data sent for each
connection.


Page 76
Segment Error Rate (%)
Re traansmittedDataPackets *100
Actual Data Packets
For above example:

Segment Error rate = (4/1471) * 100 = 0.27%
For results purpose it is measured as the average of all the values each connection and
finally averaged over the network.
Following can be two major reasons for TCP re-transmissions:
Actual Packet Loss leading to TCP retransmission
Network Congestion leading to delayed acknowledgements and thus false retransmissions
Usually network congestion leads to a very high retransmission rate. Therefore, considering
there is no major congestion in the network, the Segment error rate can be correctly
substituted for TCP Re-transmission rate or Packet Loss rate performance.
9.6.2 Windump Logs TCP Roundtrip Time

TCP roundtrip time is directly read from the TCP Trace logs as Average RTT for each
connection.
For results purpose it is measured as the average of all the values each connection and
finally averaged over the network.


Page 77
10. DELIVERABLES
After successful completion of the Acceptance Test following deliverables will be produced.
All the results will be based on Network wide average performance with 99% statistical
confidence reliability.
10.1 Performance Test Deliverables

a) RF Design Coverage Plots
b) System wide Coverage Plots from Drive Test
c) System wide Throughput, PER and FER statistics
d) Connection Drop and Access Failure statistics
e) Average Packet Round Trip Delay statistics
f)
Handoff Failure Rate Statistics
g) Application Performance (FTP, HTTP, Email, NRTV, FTP with IPSec)
Average Single User Throughput Statistics
Average Packet Loss Rate Statistics (TCP Segment Error Statistics)
Latency measurements (TCP Round Trip Time Statistics)
h) DRC, PER, SINR, AT State TX, RX, Best Server plots on system wide
i)
Documentation of all the hardware changes made to the Network
j)
Information related to the problem/exclusion areas as agreed
k) Raw data (CAIT logs)
10.2 Supplementary Test Deliverables

Test procedures to measure following performance matrices will be provided.
1. Packet Error Rate Statistics
2. DRC Statistics
3. AT Terminated Connection Success Rate
4. Maximum simultaneous users per sector capacity
5. Session drop and success rate statistics

Page 78
6. Power control and Idle handoff results


Page 79
11. NORTEL-VESPER CONTRACT RF TEST CASES

This section explains various test cases as were agreed between Nortel Networks and
Vesper in the contract.
Site Performance Acceptance
Test Case
Number
Test Case Title
Group (RF, Core

or RF+CORE)
Test Case Purpose
SPM1
Coverage
Verify and compare, with target value defined, the coverage

probability, at a live environment, according to design
requirements.
SPM2
Single User Throughput Verify, for measurement purpose, throughput of a stationary

Forward and Reverse
single user with FTP application on forward and reverse links, at
Links
a live environment.
RF
SPM3
PER
RF
Verify, for measurement purpose, the PER on both FL and RL,

at a live environment.
RF
System Performance Acceptance

Test Case
Test Case Title
Number
SP1
Throughput / Latency /
Packet Loss Stationary
single AT(s): User
Application "FTP"
SP2
SP3
SP4
SP5
SP6
Group (RF, Core

or RF+CORE
Test Case Purpose

Measure and compare, with agreed target values, FL and RL
throughput / latency / packet loss as applicable with 100%
application "FTP". Test procedure and configuration to be
specified by Nortel and agreed between the parties.
RF + CORE
single AT(s): User
Application "HTTP"
Measure and compare, with agreed target values, FL throughput

/ latency / packet loss as applicable with 100% application
"http. Test procedure and configuration to be specified by
Nortel and agreed between the parties; user application hardware
& software to be provided by Vsper.
RF + CORE
single AT(s): User
Application "E-Mail"

application "e-mail". Test procedure and configuration to be
specified by Nortel and agreed between the parties, user
application hardware & software to be provided by Vsper.
RF + CORE
single AT(s): "NRTV"
User Applications
Measure and compare, with agreed target values, FL throughput

/ latency / packet loss as applicable with 100% application
"NRTV". Test procedure and configuration to be specified by
Nortel and agreed between the parties, user application hardware
& Software to be provided by Vsper.
single AT(s): "IPSEC"
User Applications

application "IPSEC". Test procedure and configuration to be
specified by Nortel and agreed between the parties, user
application hardware & Software to be provided by Vsper.
Packet Loss Mobility
single AT(s): Three User
Applications
Measure FL throughput / latency (PING) / packet loss with

three applications. 15 drive routes (<= 20km) within 31 BTS
coverage. Test procedure and configuration to be specified by
Nortel and agreed between the parties, user application hardware

RF + CORE
RF + CORE
RF

Page 80
& Software to be provided by Vsper.

SP7
SP8
SP9
SP10
SP11
SP12
AT Terminated
Verify the AT terminated connection success rate at a live
Connection Success Rate environment. Test procedure and configuration to be specified
by Nortel and agreed between the parties.
AT Origination
RF
Verify and compare, with agreed target values, access failure /

origination success rate, at a live environment. Test procedure
and configuration to be specified by Nortel and agreed between
the parties.
RF
AT Dropped Connection Verify and compare, with agreed target values, the AT dropped
Rate
connection rate, at a live environment. Test procedure and
configuration to be specified by Nortel and agreed between the
parties.
RF
Soft Inter-cell and Inter- Verify and compare, with agreed target values, the soft / softer
sector Handoff
handoff failure rates, at a live environment. Test procedure and
Performance
parties.
RF
Coverage
ANTS Capacity
Verify and compare, with agreed target values, the coverage

probability according to design requirements, at a live
environment. Test procedure and configuration to be specified
RF
For a cluster of live radiating ANTS, verify the maximum

number of active simultaneous connections supported by one
sector. Test procedure and configuration to be specified by
Nortel and agreed between the parties.
RF
SP13
Idle Handoff (Same

Frequency)
Verify the idle handoff feature, at a live environment. Test

procedure and configuration to be specified by Nortel and
agreed between the parties.
RF
SP14
Reverse Link Power

Control Mechanism
Verify the RL power control mechanism, at a live environment.

Test procedure and configuration to be specified by Nortel and
RF
SP15
IS-856 Section
Establishment Success
Rate
Verify IS-856 section establishment success rate at a live

environment. Test procedure and configuration to be specified
RF
SP16
Dropped IS-856 Session Verify IS-856 session drop rate at a live environment. Test
Rate
SP17
DRC Statistics
SP18
PER
RF
Verify DRC statistics (DRC value, DRC lock, inter-DRC repointing) at a live environment. Test procedure and
parties.
RF
Verify, at a live environment, the PER on both FL and RL. Test

RF


Page 81
12. ABBREVIATIONS
1x EV-DO
1x Evolution for Data Only
3G
Third Generation Wireless Systems
3GPP
Third Generation Partnership Project
AN
Access Network
AP
Access Point
ARQ
Automatic Repeat Request
AT
Access Terminal (Mobile Unit)
ATP
Acceptance test Plan
BTS
Base Station Transceiver System
CAIT
CDMA Air Interface Tester
CDMA
Code Division Multiple Access
CI
Confidence Interval
CL
Confidence Level
CN
Core Network
DNS
Domain Name Server
DO
Data Only
DOM
Data Only Module
DRC
Data Rate Control
FER
Frame Error Rate
FER
Frame Erasure Rate
FTP
File Transfer Protocol
FWA
Fixed Wireless Access
GLF
Grayson Log File
GPS
Global Positioning System
HDR
High data Rate


Page 82
HTTP
Hyper Text Transfer Protocol
ICMP
Internet Control Message Protocol
IP
Internet Protocol
IPSec
Internet Protocol Security
Kbps
Kilo-bits Per Second
KB
Kilo Bytes
LAN
Local Area Network
LAU
Line Access Units
MBL
Mobile Logs
MDM
Mobile Diagnostic Monitor (Qualcomm Tool)
msec
1000th of a second
NRTV
Near Real Time Video
PC
Personal Computer, Desk Top
PC
Power Control
PDN
Packet Data Network
PDSN
Packet Data Service Node
PER
Packet Error Rate
PPP
Point-to-Point Protocol
PS
Packet Switched
QoS
Quality of Service
RAB
Reverse Activity Bit
RF
Radio Frequency
RFO
RF Optimizer (Nortel Tool)
RLP
Radio Link Protocol
RN
Radio Network
RNC
Radio Network Controller (BSC)
RNSM
Radio Node Server Module


Page 83
RTD
Round Trip Delay
RTP
Real-time Transport Protocol
RTT
Round Trip Time
SER
Segment Error rate
SNR
Signal to Noise Ratio
TCP
Transmission Control Protocol
TCP RTT
TCP Round Trip Time
TRT
Transaction Response Time
UDP
User Datagram Protocol
VJC
Van Jacobson Compression
VoIP
Voice over Internet Protocol
VPN
Virtual Private Network
WWW
World Wide Web


Page 84
APPENDIX A: DATA LOGS REQUIREMENT

S.N.
Test /
Performance Parameter
Application
Mobility / ATP Test

Stationary
#
Required Logs
Network Performance Acceptance Tests

1
RF Tests
Per Site RF Coverage
Connection Drop Rate
Handoff Failure Rate
System RF Coverage
Access Failure Rate (PING)
FTP
7
8
9
HTTP
Test 1*
Test 2
AT
RNC FER script
AT
RNC perf. Counters
AT
RNC perf. Counters
AT
RNC perf. Counters
Test 3
Test 4
AT
AT
Windump/Snoop
Chariot
AT
Windump/Snoop
Chariot
Latency (TCP RTT)
AT
Windump/Snoop
Chariot
AT
Snoop
Chariot
AT
Snoop
Chariot
AT
Snoop
Chariot
AT
Snoop
Chariot
AT
Snoop
Chariot
AT
Snoop
Chariot
AT
Windump/Snoop
Chariot
AT
Windump/Snoop
Chariot
AT
Windump/Snoop
Chariot
AT
Windump/Snoop
Chariot
AT
Windump/Snoop
Chariot
AT
Windump/Snoop
Chariot
10
11
Latency (TCP RTT)
12 Email
13
14
Latency (TCP RTT)
15 NRTV
16
17
Latency (Average One-Way Delay)
18 FTP / IPSec Single User Throughput

19
20
Latency (TCP RTT)
Test 5
Test 6
Test 7
Test 8
Supplementary Tests (For Information only)

21 RF Tests
Per Site Single user Throughput - FTP
22
Per Site Packet and Frame Error Rate
23 FTP, HTTP Single User Throughput

24
25
and NRTV
(siml.)
Latency (TCP RTT)
Test 1*
M
M
Test 9
AT
AT
RNC FER script
AT
AT
RNC FER script
AT
Windump/Snoop
Chariot
AT
Windump/Snoop
Chariot
AT
Windump/Snoop
Chariot
-
26 RF Tests
Connection Success Rate (AT terminated)
27
Fwd. Link Sector Capacity (# act. users)
28
29
30
Session drop and success rate
31
DRC Statistics
32
System Packet and Frame Error Rate
AT
Test 10
AT
RNC Counters
Test 11
AT
RNC Counters
Idle Handoff
Test 12
Rev. Link Power Control Msg.

Test 13
AT
RNC Counters
AT
RNC Counters
AT
RNC Counters
AT
RNC Counters
RNC Counters

Page 85
APPENDIX B: REVERSE LINK FER SCRIPTS AND SAMPLE LOG

________________________________________________________________
UNIX Script: logrlfer
This script needs the ketmit software to run
________________________________________________________________
#!/opt/bsmext/bin/kermit +
#
# CDMA Live Air Verification
#
# Change Record:
#
# 2000-11-19 Version 01 (JKO)
#
Original Version
#
define PERIOD
define RNC_IP
echo
echo
echo
echo
echo
echo
echo
echo
echo
echo
1
47.135.201.244
; delay in seconds between FER displays

; IP address of the RNC
This script will query reverse link power control stats

on RNC \m(RNC_IP) for a given UATI.
FER will be shown at \m(PERIOD) second intervals.
FER statistics are cleared each time this script is run.
Save the log using the name rlfer-measurements.
and use getrlfer to get just the timestamps
and FER measurements from the log.
Enter a UATI (format like 0x3F)
if not defined \%1 ask \%1 { UATI: }
; ensure you get a UATI from the user.
set exit warning off

set host \m(RNC_IP)
; Tell Kermit to ignore telnet logout warnings.

; Log in to the RNC over Ethernet backhaul.
lineout enable
lineout
lineout
lineout
lineout
show
show
show
show
power-control
power-control
power-control
power-control
config
configFER all
configFER user \%1
status \%1
lineout configure
lineout clear power-control stats \%1
lineout exit
while 1 {
lineout exit
lineout show sntp time
lineout enable
lineout show power-control stats \%1
input \m(PERIOD) WAIT
}
________________________________________________________________
Sample Log: from Xterm log, renamed to rlfer-measurements
________________________________________________________________
AIRVANA-07>show sntp time
Sntp Time Details - UTC
Timing Source = 47.135.201.245
Base Secs
= 3246722485 (c1850db5)


Page 86
Base Nsecs
= 654816005 (2707b305)
Base TBH
= 381 (0000017d)
Base TBL
= 1367329457 (517fcab1)
Current Secs = 3246722490 (c1850dba)
Current Nsecs = 402755400 (18018f48)
Num Leap Secs (since 1980) = 13
Date = 2002:11:19
Time = 19:21:30.402
Local Time Offset = -300 minutes
AIRVANA-07>enable
AIRVANA-07#show power-control stats 0x3F1
The UATI is = 1009
The total # of received frames with Good CRC = 1026
The total # received frames with Bad CRC = 10
The # of empty frames from the last Rcvd packet = 0
The Frame Error Rate = 0.009653
---------------------------------------------------------------Time in RL Frames at which PCT was computed = 0x4d477ab9
Power Control Algorithm State = ACTIVE-STEADY
CRC of received Reverse Link Frame = GOOD
|Difference| between current & last PCT (1024th of a dB) = 5
Minimum Significant Step Size (1024th of a dB) = 0
Computed Power Control Threshold (1024th of a dB) = -22335
Computed Power Control Threshold (SET POINT) (8th of a dB) = -175
Current PCT value qualifies to be sent to BTS
---------------------------------------------------------------AIRVANA-07#exit
________________________________________________________________
Log Parsing Script: getrlfer
________________________________________________________________
egrep -i "(Time =|Frame Error Rate)" rlfer-measurements* > rlfer_data
________________________________________________________________
Sample Parsed Log: in file rlfer_data
________________________________________________________________
Time = 20:47:14.881
The Frame Error Rate
Time = 20:47:16.012
Time = 20:47:16.918
Time = 20:47:18.048
Time = 20:47:18.162
= 0.000000
= 0.000000
= 0.000000
= 0.000000
= 0.000000


Page 87
APPENDIX C: CHARIOT TEST SCRIPTS

C.1. Forward FTP Test
Endpoint 1 = PC/laptop
Endpoint 1
---------SLEEP
initial_delay=12000
LOOP
number_of_repetitions=1
CONNECT_INITIATE
source_port=AUTO
RECEIVE
login_size=15
control_buffer_size=DEFAULT
SLEEP
user_delay=0
SEND
login_size=15
control_datatype=trans.cmp
send_data_rate=UNLIMITED
RECEIVE
login_size=15
SLEEP
user_delay=0
SEND
login_size=15
Endpoint 2 = Application (Sun) server

Endpoint 2
----------
LOOP
CONNECT_ACCEPT
destination_port=AUTO
SEND
login_size=15
RECEIVE
login_size=15
SEND
login_size=15
RECEIVE
login_size=15
RECEIVE
200
DISCONNECT
close_type=Reset
LOOP
number_of_timing_records=5
START_TIMER
LOOP
transactions_per_record=1
CONNECT_INITIATE
source_port=AUTO
SEND
file_control_size=30
RECEIVE
SEND
RECEIVE

SEND
200
200
DISCONNECT
close_type=Reset
LOOP
LOOP
CONNECT_ACCEPT
RECEIVE
SEND
RECEIVE
SEND
SLEEP
delay_before_responding=0

Page 89
RECEIVE
size_of_record_to_send=10000000
receive_buffer_size=4096
RECEIVE
DISCONNECT
close_type=Reset
INCREMENT_TRANSACTION
END_LOOP
END_TIMER
SLEEP
transaction_delay=12000
END_LOOP
CONNECT_INITIATE
source_port=AUTO
SEND
6
6
RECEIVE
25
DISCONNECT
close_type=Reset
END_LOOP

SEND
send_buffer_size=4096
send_datatype=NOCOMPRESS
SEND
DISCONNECT
close_type=Reset
END_LOOP
END_LOOP
CONNECT_ACCEPT
RECEIVE
6
SEND
25
25
DISCONNECT
close_type=Reset
END_LOOP

Page 90
C.2. Reverse FTP Test

Endpoint 1
---------SLEEP
initial_delay=12000
LOOP
CONNECT_INITIATE
source_port=AUTO
RECEIVE
login_size=15
SLEEP
user_delay=0
SEND
login_size=15
RECEIVE
login_size=15
SLEEP
user_delay=0
SEND
login_size=15


Endpoint 2
----------
LOOP
CONNECT_ACCEPT
SEND
login_size=15
RECEIVE
login_size=15
SEND
login_size=15
RECEIVE
login_size=15

Page 91
RECEIVE
200
DISCONNECT
close_type=Reset
LOOP
START_TIMER
LOOP
CONNECT_INITIATE
source_port=AUTO
SEND
RECEIVE
SEND
RECEIVE
SLEEP
delay_before_responding=0
SEND

SEND
200
200
DISCONNECT
close_type=Reset
LOOP
LOOP
CONNECT_ACCEPT
RECEIVE
SEND
RECEIVE
SEND
RECEIVE

Page 92
RECEIVE
DISCONNECT
close_type=Reset
END_LOOP
END_TIMER
SLEEP
END_LOOP
CONNECT_INITIATE
source_port=AUTO
SEND
6
6
RECEIVE
25
DISCONNECT
close_type=Reset
END_LOOP

receive_buffer_size=4096
SEND
DISCONNECT
close_type=Reset
END_LOOP
END_LOOP
CONNECT_ACCEPT
RECEIVE
6
SEND
25
25
DISCONNECT
close_type=Reset
END_LOOP

Page 93
C.3. HTTP Test

Endpoint 1
---------SLEEP
initial_delay=12000
LOOP
START_TIMER
LOOP
CONNECT_INITIATE
source_port=AUTO
SEND
RECEIVE
file_size_main_page=10700
receive_buffer_size=DEFAULT
RECEIVE
file_size_objects=7800
RECEIVE


Endpoint 2
----------
LOOP
LOOP
CONNECT_ACCEPT
RECEIVE
SEND
file_size_main_page=10700
send_buffer_size=DEFAULT
send_datatype_main_page=news.cmp
SEND
send_datatype_objects=lena.cmp
SEND

Page 94
RECEIVE
RECEIVE
RECEIVE
DISCONNECT
close_type=Reset
END_LOOP
END_TIMER
SLEEP
END_LOOP

SEND
SEND
SEND
DISCONNECT
close_type=Reset
END_LOOP
END_LOOP

Page 95
C.4. NRTV Test

Endpoint 1 = Application (Windows PC) server
Endpoint 1
---------RTP_PAYLOAD_TYPE
H261
SLEEP
initial_delay=12000
CONNECT_INITIATE
source_port=AUTO
LOOP
SEND
file_size=17000000
send_data_rate=500 kbps
END_LOOP
DISCONNECT
close_type=Reset

Endpoint 2
----------
CONNECT_ACCEPT
LOOP
START_TIMER
RECEIVE
file_size=17000000
END_TIMER
END_LOOP
DISCONNECT
close_type=Reset

Page 96
C.5. Forward Email Test

Endpoint 1
---------SLEEP
initial_delay=12000
CONNECT_INITIATE
source_port=AUTO
LOOP
START_TIMER
LOOP
SEND
736
RECEIVE
exchange_mail_size=100000
END_LOOP
END_TIMER
SLEEP
END_LOOP
DISCONNECT
close_type=Reset


Endpoint 2
----------
CONNECT_ACCEPT
LOOP
LOOP
RECEIVE
736
SEND
END_LOOP
END_LOOP
DISCONNECT
close_type=Reset

Page 97
C.6. Reverse Email Test

Endpoint 1
---------SLEEP
initial_delay=12000
CONNECT_INITIATE
source_port=AUTO
LOOP
START_TIMER
LOOP
SEND
160
RECEIVE
160
SEND
RECEIVE
576
SEND
96

Endpoint 2
----------
CONNECT_ACCEPT
LOOP
LOOP
RECEIVE
160
SEND
160
RECEIVE
SEND
576
RECEIVE
96
Page 98
RECEIVE
432
END_LOOP
END_TIMER
SLEEP
END_LOOP
DISCONNECT
close_type=Reset
SEND
432
END_LOOP
END_LOOP
DISCONNECT
close_type=Reset
C.7. FTP test with IPSec

Same script as for regular FTP test


Page 99
APPENDIX D: ACCESS POINT FUNCTIONALITY TEST FORM

Date/Time:
Site ID/AN#
Site Name
Zone ID
Address
Engineer
MDM Log file

Other Logs
AT Model & Version
Data Collection Tool/Version
Network Software Version
Problems Resolved?
_____________
Problems
Status:
Database Updated:
Comments/Observation
Handoff between Sectors

Alpha <-> Beta
Alpha <-> Gamma
ALPHA (X)
Datafill
Gamma <-> Alpha
BETA (Y)
Observed
Datafill
GAMMA (Z)
Observed
Datafill
Pilot PN
Pilot PN
Pilot PN
Height
Height
Height
Orientation
Orientation
Orientation
Mech. DT
Mech. DT
Mech. DT
Antenna
Antenna
Antenna
Neighbor List
Neighbor List
Datafill
Observed
Datafill
Observed
Neighbor List
Observed
Datafill
Call Initiation/Term.
Coverage/HO
Coverage/HO
Coverage/HO
Observed
APPENDIX E: ACCEPTANCE TEST LOG SHEET

Date:
(YYYYMMDD)
Test Information
Test Type: RF Performance, Application Performance
Application:
Technician/Engineer:
AT Kit Serial Number:
Data Collection Tool & Kit Serial Number:
BTS ID or Zone Number:
Run
Number
Test
Location
MIN
Log Filename
Nortel Commercial in Confidence

BTS #
Sector Test
Time
Performance Parameters:
Van/Test unit:
Vesper rep (if present):
Morphology:
Network Software:
Avg.
Throughput (kB)
AF / Drop
Comments (Time, location, unusual events etc.)
Acceptance Test, Logging

Page 101
Copyright 2003 Nortel Networks

All rights reserved
Nortel Networks Confidential:
The information contained in this document is the property of
Nortel Networks. Except as specifically authorized in writing by Nortel
Networks, the holder of this document shall keep the information
contained herein confidential and shall protect same in whole or in part
from disclosure and dissemination to third parties and use same for
evaluation, operation, and maintenance purposes only:
Information is subject to change without notice. Nortel Networks
reserves the right to make changes in design or components as
progress in engineering and manufacturing may warrant
Nortel Commercial in Confidence

Acceptance Test, Logging

Page 102

Vesper RF Acceptance Test Plan: CDMA2000 1xEV-DO Network

Hochgeladen von

Dokumentinformationen

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Vesper RF Acceptance Test Plan: CDMA2000 1xEV-DO Network

Hochgeladen von

Copyright:

Verfügbare Formate

Core RF Engineering

Vesper RF Acceptance Test Plan

Acceptance Test Plan

Core RF Engineering, 5395

Details of Revision Changes

First Draft Release for Vesper

Test procedures updated (sec. 2.3, 5 & 7)

Updated Appendix A and Section 5, 6, 10

Updated test plans, section 6, 9 & 10

Second Draft Release for Vesper

Improved the section on PER, added the section

Added sections on RL power control messages,

Update based on Vesper meeting on Jan 27, 03

Update on Vesper meeting on Feb.05, 2003

Updated the test procedures

Update based on Vesper meeting on Jan 20, 03

Fifth Release for Vesper on 02/19/2003

Nortel Networks Proprietary

Vesper EV-DO RF Acceptance Test

ACCEPTANCE AND SIGN-OFF

Nortel Networks RF Acceptance Prime

Vesper RF Acceptance Prime

Nortel Networks Proprietary

Vesper EV-DO RF Acceptance Test

3.1.4 Access Failure Rate ...................................................................................................3

5.1.2 Reverse link Performance Test ..................................................................................3

5.1.3 Connection Drop Test (SP9, SP10, SP11) ................................................................3

5.1.4 Access Failure Test (SP8, SP11)...............................................................................3

Nortel Networks Proprietary

Vesper EV-DO RF Acceptance Test

5.2 APPLICATION PERFORMANCE TESTS ...........................................................................................3

5.2.2 Hyper Text Transfer Protocol (SP2)...........................................................................3

5.2.4 Near Real Time Video (SP4)......................................................................................3

Nortel Networks Proprietary

Vesper EV-DO RF Acceptance Test

11. NORTEL-VESPER CONTRACT RF TEST CASES .........................................................................3

Nortel Networks Proprietary

Vesper EV-DO RF Acceptance Test

1.3 Related Documents

EV-DO Link Budget

CAIT Users Manual

Chariot Tool Users Manual

IS-856 (Standard for CDMA2000 1xEV-DO)

PCubed Users manual

Nortel Networks Proprietary

Vesper EV-DO RF Acceptance Test

2.1 Vesper Acceptance Tests

2.1.1 RF Performance Tests

2.1.2 Application Performance Tests

Nortel Networks Proprietary

Vesper EV-DO RF Acceptance Test

Application performance tests will be conducted by stationary measurements using

2.1.3 Supplementary Tests

2.2 Acceptance Test Entrance Criteria

Nortel Networks Proprietary

Vesper EV-DO RF Acceptance Test

2.3 Exclusion Area Criteria

Vesper EV-DO RF Acceptance Test

Nortel Networks Proprietary

Vesper EV-DO RF Acceptance Test

RF Performance Acceptance Tests

Connection Drop Rate

Average Handoff Failure Rate