DOCUMENTTYPE

TypeUnitOrDepartmentHere
TypeYourNameHere

1 (13)

TypeDateHere

Project Enigma KPIs

Summary
This document is draft and for technical discussion only.
This document outlines the measurement techniques that will be put in place to verify the satisfactory
operation of those areas of the O2 DE 3G radio network that are affected by the swap of radio
equipment from Nortel to Nokia under Project Enigma. It is expected that this draft document will
undergo several iterations as the boundaries of responsibility during the equipment swap process are
finalised.
The basic principles underlining the swap process are:
The 3G rf performance in terms of coverage and interference after the swap shall be equivalent
or better than before the swap took place.
The network quality in terms of call completion success rate and customer experience of data
services shall be equivalent or better than before the swap took place.
The swap process will involve the upgrading of some sites from Omni Transmit, Sector Receive (OTSR
known within Nokia as ROC) to Sector Transmit, Sector Receive (STSR known within Nokia as CEC).
The possibility exists that it will not be possible to include a limited number of sites as part of the swap
event. The agreed measurement process will take these circumstances into account and is based
around the following assumptions:

Confirmation of basic rf function will be carried out in a method analogous to that used in a 2G
swap programme in the UK.
Confirmation of 3G network quality will follow a process complimenting the current O2 cluster
acceptance procedure.

This document is based upon the O2 UK Project Enigma KPIs document, authored by Adrian Sharples,
Radio Engineering O2 UK, with modifications to make it applicable to the O2 DE network.

DOCUMENTTYPE
TypeUnitOrDepartmentHere
TypeYourNameHere

2 (13)

TypeDateHere
TABLE OF CONTENTS

Scope and Objectives........................................................................................................................................3

1. Introduction..................................................................................................................................................3
2. Key Assumptions........................................................................................................................................3
3. Drive Testing...............................................................................................................................................5
4. Coverage Verification.................................................................................................................................5
4.1
Basic procedure...................................................................................................................................5
4.1.1
OTSR to STSR Conversions.....................................................................................................6
4.2
Processing Of Results........................................................................................................................6
4.2.1
Corrected Signal Level Same Server Analysis (3G)............................................................6
4.2.2
Cumulative Distribution Of Binned RSCP................................................................................6
4.2.3
Valid Bins......................................................................................................................................7
4.3
Acceptance Criteria.............................................................................................................................7
5. Interference Assessment...........................................................................................................................7
6. Network Performance KPIs.......................................................................................................................7
Network Performance....................................................................................................................................8
6.1.1
Success Rate Definitions...........................................................................................................8
6.1.2
Number Of Measurements And Tolerance...............................................................................8
6.1.3
Exclusions....................................................................................................................................9
6.1.4
Data Throughput..........................................................................................................................9
6.1.5
3G-2G Handover Performance..................................................................................................9
6.2
Acceptance Criteria...........................................................................................................................10
7. In-building Measurements.......................................................................................................................10
7.1
In-building Coverage Tests...............................................................................................................10
7.2
In-building Service Tests...................................................................................................................10
8. Call Event Definitions...............................................................................................................................12
8.1
12.2k voice call..................................................................................................................................12
8.2
3G/2G hand-over for a 12.2k voice call..........................................................................................12
8.3
64k CS data call................................................................................................................................12
8.4
PS data session.................................................................................................................................12
8.5
3G/2G hand-over for a PS Data Session.......................................................................................13

DOCUMENTTYPE
TypeUnitOrDepartmentHere
TypeYourNameHere

3 (13)

TypeDateHere

SCOPE AND OBJECTIVES

This draft of the document defines the procedure for establishing the satisfactory operation of the O2 DE
radio network after the equipment swap events carried out under Project Enigma. The objective is to
define an unambiguous test of acceptability of the network along the principle that the network
performance must be equivalent or better than the performance of the network before the swap took
place.
1. INTRODUCTION
To verify the network performance following the swap of 3G radio equipment in the O2 DE network under
Project Enigma, metrics must be defined to enable agreement between Nokia and O2 to be reached
regarding the success of the swap process.
A similar procedure was carried out as part of a 2G programme to swap Nokia equipment into the UK
network. This measurement programme concentrated on the use of drive data to confirm coverage
levels and correct installation.
The process in Project Enigma is further complicated by the fact that:
Site upgrades from OTSR to STSR (ROC to CEC in Nokia terminology) will take place as part of
the swap.
Low traffic levels in the O2 3G network make reliance on network performance counters
inappropriate.
Inter-vendor handover will not be supported as part of the swap programme.
The principles employed in this process are that fundamental rf properties (coverage and interference)
will be verified. Network quality will be assessed in a method using call KPIs (in line with the cluster
acceptance testing currently performed within O2 UK).
2. KEY ASSUMPTIONS
The statements in the remainder of this document are dependent:

There is insufficient customer traffic on the O2 DE 3G network for RNC counters to be used as a
key measure of network performance before and after a swap event.
Measurements relating to the performance of the O2 3G network shall be reliant on drive testing.
The same 3G carrier frequency will be used during all swap events.
Pre swap drive testing will take place before the start of the swap of any adjacent areas.
Post swap drive testing will take place after the completion of swap events in adjacent areas.
All KPI testing will be conducted with devices in dual mode.
No new site integrations will take place between the pre-swap and post-swap periods.
Inter-vendor handover will not be supported as part of the Project Enigma equipment swap.
A site within a swap event cluster that cannot form part of the swap event will not transmit during
the event verification drive testing.
As part of the swap event, some sites will be upgraded from OTSR to STSR (ROC to CEC in
Nokia terminology).
The upgrade from OTSR for STSR will remove a 5.6dB attenuation in the downlink link budgets
which must be discounted in the measurement post processing.

DOCUMENTTYPE
TypeUnitOrDepartmentHere
TypeYourNameHere

4 (13)

TypeDateHere

Given that the downlink power reference points differ between Nokia and Nortel equipment, prior
to the pre-swap drive test the Nortel equipment parameter settings shall be configured to provide
the same top-of rack pilot channel (cpich) power that forms part of the standard Nokia databuild.
A tolerance factor of 1dB shall be allowed on signal level assessments to take account of
inaccuracies in the measurement process.
KPIs related to 3G downlink throughput will fail the principle of pre and post swap performance
equivalence due to known differences between Nokia and Nortel implementations. Results on the
Nokia area of O2s UTRAN network show mobile throughput values approximately 70% of those
achieved in Nortel areas on the 384kbit/s bearer. As a result, a modification factor will be used for
comparisons in the swap process. This modification factor will be reviewed whenever design
changes are made that improve the Nokia UTRAN throughput deficit.
Performance KPI assessments will be made using appropriate test equipment.
The measurement process shall be reviewed after 4 swap events to ensure that the need to meet
the overriding principles is being achieved.

DOCUMENTTYPE
TypeUnitOrDepartmentHere
TypeYourNameHere

5 (13)

TypeDateHere

3. DRIVE TESTING
The routes within the swap event cluster are to be agreed between O2 and Nokia. The drive tests will
span a minimum of * working days. The measurement period may be varied by agreement on an event
by event basis. The selection of the drive route shall adhere to the following principles

All sectors of every site in the cluster are covered to derive the clear footprint of each cell,
including areas with missing sites.
3G-2G transition areas are crossed.
Focus on residential areas, major roads, and routes around key business centres, shopping
areas, tourist attractions and railway stations.
Additional side streets are travelled in city centres.
The entire event area is driven in a uniform fashion such that all areas are driven irrespective of
their distance from a cell site. (i.e. The route did not favour locations close to cell sites).
The event drive route shall be designed to ensure that border areas between the event areas and
into surrounding 2G areas are driven. The route should travel sufficiently close to the first tier of
border cells to ensure that HO to cells outside the swap event is tested.

The border approach is illustrated in the figure below.

4. COVERAGE VERIFICATION
4.1 Basic procedure
Pre and post-swap drive tests shall be carried out along the agreed route that will encompass all sectors
forming part of the swap event. Coverage measurement shall be performed using a scanner.
Measurements shall be binned on a geographic basis into a 150m x 150m resolution grid laid over the
drive route. Each measurement report shall be placed in the appropriate bin indicated by its associated
GPS location sample.

DOCUMENTTYPE
TypeUnitOrDepartmentHere
TypeYourNameHere

6 (13)

TypeDateHere

For each bin, two levels shall be reported. The first is the average level (taken to be the average of the
dB values, not the linear average) of all best server RSCP measurement samples taken within the bin
(i.e. the scrambling code with the highest RSCP value in any measurement report. The second
measurement is the averaged RSCP level, averaged in terms of dB, not linear average, for the dominant
server within the bin. In this case the dominant server is the scrambling code with the largest number of
best server samples received within the bin. If two or more scrambling codes have the same number of
best server measurements then the one with the highest average signal value shall be selected. The
scrambling code of the dominant server will also be stored for each bin.
4.1.1OTSR to STSR Conversions
Where an OTSR to STSR upgrade has taken place, an additional processed set of data is required
relating to measurements from the upgraded site. For each post-swap measurement report that includes
RSCP values for scrambling codes of collocated sectors, the linear sum of these measurements shall be
stored to represent the value of the equivalent power measured from the OTSR site. It is this combined
value that will be used in power level comparisons for the site.
4.2 Processing Of Results
Raw RSCP signal level plots shall be presented without binning or any of the signal level corrections
described in the following sections to indicate the extent of poor coverage in the area. These plots will be
coloured according to agreed thresholds.
In addition, uncorrected RSCP level plots will be shown of the change in the average RSCP level of bins
in the area before and after the swap process. Each bin of these plots shall be coloured according to
agreed thresholds.
Plots shall be presented with each bin coloured according to the scrambling code of the dominant server,
with a key provided showing the mapping of colour to scrambling code. These shall be overlaid with a
diagram illustrating the location of the base stations, the azimuth directions of the antennas and the
planned scrambling codes to be transmitted from each of the antennas. These plots are required to
verify that the scrambling code plan has been implemented correctly and sectors have not been crossed.
4.2.1Corrected Signal Level Same Server Analysis (3G)
These plots will show the difference in RSCP for bins that are served by the same dominant server both
pre and post swap. Where an OTSR to STSR conversion has taken place, the data from the sum of the
collocated sectors (as described Section 4.1.1) shall be used. Before comparison with the pre-swap
signal level, the agreed splitter insertion loss figure shall be subtracted from the signal levels from the
upgraded site.
The number of geographical bins covered by this dominant server both pre and post swap will also be
presented.
Where there has been degradation in the corrected RSCP level in a bin following the swap event this
shall be taken to be indicative of a deficiency in the swap process for which corrective action is
necessary.
4.2.2Cumulative Distribution Of Binned RSCP

DOCUMENTTYPE
TypeUnitOrDepartmentHere
TypeYourNameHere

7 (13)

TypeDateHere

The pre and post swap cumulative distributions shall be produced of the average received level of
measurement samples within each valid bin for which a measurement has been taken both pre and post
swap. The definition of a valid bin is given below.
Where an OTSR to STSR conversion has taken place, the data from the sum of the collocated sectors
(as described Section 4.1.1) shall be used. Before comparison with the pre-swap signal level, the agreed
splitter insertion loss figure shall be subtracted from the signal levels from the upgraded site.
4.2.3 Valid Bins
A valid bin is one where all measured samples post swap are from sites that have been included in the
swap event. Bins that include a measurement from a site that has not been included in the swap event
(and is not transmitting post-swap) will be excluded from both pre and post swap cumulative
distributions. This situation will be reviewed after 4 swap events to ensure that sufficient bins are being
included to make this measurement a statistically valid approach.
The corrected RSCP value at the 10% point of the cumulative distribution post swap shall not be less
than the 10% point of the pre-swap cumulative distribution by an amount that is less than the tolerance
factor.
4.3 Acceptance Criteria
Acceptance of 3G coverage maintenance shall be taken to be the satisfaction of the measure in Sections
4.2.1. and 4.2.2. The failure to meet other coverage criteria shall be taken to indicate an individual fault
that shall be rectified by remedial action by Nokia.
5. INTERFERENCE ASSESSMENT
With the upgrade of some sites from OTSR to STSR, the direct comparison of the interference
landscape in terms of Ec/Io is complicated by the introduction of new sectors since this brings with it
additional cell boundaries. To provide a pre and post-swap comparison the following procedure shall be
followed.
From the scanner data presented previously, the sum of the best three RSCP values in any
measurement report shall be summed (linear summation). This shall be divided by the measured RSSI
value at that location giving a total equivalent active set Ec/Io. For each valid 150m x 150m bin the
average equivalent active set Ec/Io shall be stored. This average shall be the dB average of all
equivalent active set Ec/Io values within the bin.
A valid bin is as defined in Section 4.2.3
A plot of the difference between the pre-swap and post-swap equivalent active set Ec/Io for each valid
bin shall be presented. These shall be coloured according to agreed thresholds.
The intention of this process is to identify a method for ensuring that the pre and post-swap interference
environment has not degraded during the swap process but which also takes account of the alteration of
the number of sectors in the area.
A reduction in the equivalent Ec/Io in any bin that is greater than the tolerance factor will be taken to
indicate a fault in the swap process that shall be rectified by remedial action by Nokia.

DOCUMENTTYPE
TypeUnitOrDepartmentHere
TypeYourNameHere

8 (13)

TypeDateHere

6. NETWORK PERFORMANCE KPIS

Network Performance
The network performance KPIs for assessing the network pre and post swap are based on the current
cluster acceptance criteria used within O2.
The KPIs to be measured are shown in the table below.
Bearer
Voice

Measurement
Overall call completion rate (MOC only)
3G/2G Handover success rate (MOC Only)

PS Data

64/128kbps bearer
DL throughput
UL throughput
FTP session success rate
3G/2G PS handover success rate
Round Trip Time*

Video

64/384kbps bearer
DL throughput

Round Trip Time*

Video overall completion rate*

These KPI requirements will be included in the first 4 swap events and the requirement to continue
through the swap process reviewed at that time.

6.1.1Success Rate Definitions

The definition of overall call completion success rate is (OCCSR)
OCCSR = (Call Attempts - Excluded Calls - Call Setup Failures - Call Drops) / Call Attempts
This shall be applied for voice, data and videotelephony calls. The definition of excluded calls is given in
Section 6.1.3. Definitions of call events are given in Section 8.

6.1.2Number Of Measurements And Tolerance

For each call type (voice, packet and video calls) there shall be at least 300 call attempts.
The success rate of these call types shall be calculated in the way shown in Section 8.

DOCUMENTTYPE
TypeUnitOrDepartmentHere
TypeYourNameHere

9 (13)

TypeDateHere

The percentage call success rate after the swap shall be at equal to or higher than the call success rate
before the swap minus 1.5%. This 1.5% tolerance is included to allow for statistical uncertainty
associated with the call sampling process.
6.1.3Exclusions
Voice and data call handling shall be expected to handover between 3G and 2G networks. As a result,
the only exclusion of failed voice calls from the network performance KPIs shall be those that can be
identified as due to:

Test equipment failure resulting in collection file corruption

Location update clashes
UE issues (e.g. handset reset)

In each case the exclusion of a particular failed call shall be agreed between Nokia and O2.
For videotelephony, the absence of a handover to 2G requires additional allowance for situations where
it has not been possible to include a site in the swap event. Exclusions for failed videotelephony calls will
therefore be allowed if the failure occurs within an invalid 150m x 150m geographical bin. The definition
of a valid bin is given in Section 4.2.3.
6.1.4Data Throughput
There are known deficiencies in the current Nokia downlink packet data implementation. This, combined
with a more reactive 3G to 2G handover process in Nokia, lead to a significant reduction in throughput
when compared to equivalent results on the Nortel part of the O2 network.
It is expected that all swap events would fail an acceptance criteria based on 3G downlink data
throughput equivalence. As a result the following acceptance criteria shall be applied.
For the 384kbit/s downlink data rate, the average throughput post swap shall be greater than the
average pre-swap throughput multiplied by a modification factor of 0.7. This modification factor shall be
reviewed periodically in the light of the results of early swap events. The modification factor shall be
viewed as a concession to allow for known deficiencies. The principle of equivalence would require the
modification factor to be set at a value of one.
For the 128kbit/s downlink rate, the average throughput shall exceed 101.9kbit/s.
For the 64kbit/s uplink data rate, the average throughput shall exceed 50.9kbit/s.
6.1.53G-2G Handover Performance
The number of 3G-2G handovers (both voice and packet data) is expected to be highly variable
depending on the characteristics of the swap event area. The total number of handovers will be
substantially lower than the number of calls, resulting in large statistical tolerances (i.e. the difference
between pre and post swap success rates that indicate a high degree of confidence that performance
has altered). On that basis, a sliding scale of statistical tolerance based on the number of 3G-2G
handovers measured shall be applied.
Number of 3G-2G handovers
(pre-swap or post swap)

Tolerance margin (%)

DOCUMENTTYPE
TypeUnitOrDepartmentHere
TypeYourNameHere
n <= 5
5 < n <= 10
10 < n <= 20
20 < n < 30
30 < n < 40
40 < n < 50
50 < n < 100
n > 100

10 (13)

TypeDateHere
40
25
12
9
8
7
5
2

6.2 Acceptance Criteria

A swap event will have met the KPI acceptance criteria if each of the following conditions is met:

The 3G call completion success rate (voice, data and video) is greater than or equal to the preswap value (less a statistical tolerance factor of 1.5% and allowing for exclusions).
Due to known deficiencies in the Nokia packet scheduler implementation, the data throughput
values post swap shall be compared to absolute values or be compared to the pre-swap values
with a modification factor applied (as described in Section 6.1.4). The post-swap throughput
values shall be greater than the specified absolute values or modified pre-swap values.
The 3G-2G handover success rate (voice and data) is greater than or equal to the pre-swap
value (less a statistical tolerance factor described in Section 6.1.4).

7. IN-BUILDING MEASUREMENTS
As part of the swap process, the in-building performance of the system pre and post-swap should be
compared at a number of locations to be agreed in advance of the swap event. One day shall be allowed
for in-building measurement in the pre and post swap measurement weeks in which at least two
specified building locations shall be measured.
7.1 In-building Coverage Tests
For each in-building location an area of measurement shall be agreed between Nokia and O2. Where
the area of measurement exceeds 10m x 10m, for example in a shopping centre, the total area shall be
split into a number of sub-zones each of approximately 10m x 10m. These areas shall be marked on a
plan to be submitted with the measurements.
Within each sub-zone, RSCP measurements shall be taken. The principles for these measurements are
that they should be collected in as uniform a manner as possible across the whole of the sub-zone area.
At least 500 individual measurements shall be collected in each sub-zone.
The data shall be processed to provide the dB average and standard deviation of the dominant server for
the sub-zone. The dominant server is indicated by the scrambling code that provides the greatest
number of best server measurements across the sub-zone.
If the dominant server in the post-swap measurements has undergone an OTSR to STSR upgrade as
part of the swap event then the resultant average RSCP level shall have the agreed splitter insertion loss
value subtracted before comparison with the pre-swap signal level takes place.

DOCUMENTTYPE
TypeUnitOrDepartmentHere
TypeYourNameHere

11 (13)

TypeDateHere

If the average dominant server RSCP value post-swap (corrected to allow for OTSR to STSR conversion
where appropriate) is less than the pre-swap value by a value greater than the post swap RSCP
standard deviation for a sub-zone, Nokia shall carry out remedial action to resolve the situation.
7.2 In-building Service Tests
Within each of the sub-zones of an in-building area, a number of test service calls shall be carried out.
The sample calls shall comprise:
10x voice calls
10x videotelephony calls
10x 64UL / 384DL data calls
The success criteria for each call type shall be the same as that applied to the drive testing described in
the previous sections. The same exclusions allowed in the drive test cases shall also apply.
If the number of call failures post-swap is greater than one more than the number of failure pre-swap,
Nokia shall identify the cause of the additional failures and carry out remedial action to resolve the
situation.

DOCUMENTTYPE
TypeUnitOrDepartmentHere
TypeYourNameHere

12 (13)

TypeDateHere

8. CALL EVENT DEFINITIONS

8.1 12.2k voice call
For a 12.2k voice call to be counted as successful it must satisfy all of the criteria in The Table
below.
Criteri
a
Call sets up successfully (RRC connection request to alerting)

Call does not drop during the test call holding time
period (disconnect other than normal disconnect)
Call terminates successfully
Call hands-over (if necessary, both 3G-3G and 3G2G).
8.2 3G/2G hand-over for a 12.2k voice call
For a 3G/2G hand-over for a 12.2k voice call to be counted as successful it must satisfy all the
criteria in the Table below
Criteri
a
Test UE receives a RRC HANDOVER FROM
UTRAN COMMAND message from the 3G UTRAN
Error: Reference source not found

UE sends a RR HANDOVER COMPLETE message

to the 2G BSS.

8.3 64k CS data call

For a 64k CS data call to be counted as successful it must satisfy all of the criteria in the Table
below:
Criteri
a
Call sets up successfully (RRC connection request to
alerting)
Call does not drop during test call holding time period.
Call terminates successfully.
Call hands-over (if necessary, 3G-3G).

8.4 PS data session

For a PS data session to be counted as successful it must satisfy all of the criteria in the Table
below:

DOCUMENTTYPE
TypeUnitOrDepartmentHere
TypeYourNameHere

13 (13)

TypeDateHere

Criteri
a
PDP context activates successfully
PDP context or radio bearer does not drop during the
test session holding time period.
Data session hands-over (if necessary, 3G-3G aand
3G-2G)
PDP context deactivates successfully.

8.5 3G/2G hand-over for a PS Data Session

For a 3G/2G hand-over for a PS Data Session to be counted as successful it must satisfy all the
criteria in the Table below
Criteri
a
Test UE receives a CELL CHANGE ORDER FROM
UTRAN COMMAND message from the 3G UTRAN
UE receives a ROUTING AREA UPDATE Complete
message to the 2G BSS.