You are on page 1of 79

Nokia Siemens Networks LTE Radio Access, Rel.

RL15, Operating Documentation Prerelease, Issue 01

LTE E2E Field Network Performance Definitions of Key Performance Indicators


DN0972412 Issue 01 DRAFT APPROVED Approval Date 2010-06-22

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

The information in this document is subject to change without notice and describes only the product defined in the introduction of this documentation. This documentation is intended for the use of Nokia Siemens Networks customers only for the purposes of the agreement under which the document is submitted, and no part of it may be used, reproduced, modified or transmitted in any form or means without the prior written permission of Nokia Siemens Networks. The documentation has been prepared to be used by professional and properly trained personnel, and the customer assumes full responsibility when using it. Nokia Siemens Networks welcomes customer comments as part of the process of continuous development and improvement of the documentation. The information or statements given in this documentation concerning the suitability, capacity, or performance of the mentioned hardware or software products are given "as is" and all liability arising in connection with such hardware or software products shall be defined conclusively and finally in a separate agreement between Nokia Siemens Networks and the customer. However, Nokia Siemens Networks has made all reasonable efforts to ensure that the instructions contained in the document are adequate and free of material errors and omissions. Nokia Siemens Networks will, if deemed necessary by Nokia Siemens Networks, explain issues which may not be covered by the document. Nokia Siemens Networks will correct errors in this documentation as soon as possible. IN NO EVENT WILL Nokia Siemens Networks BE LIABLE FOR ERRORS IN THIS DOCUMENTATION OR FOR ANY DAMAGES, INCLUDING BUT NOT LIMITED TO SPECIAL, DIRECT, INDIRECT, INCIDENTAL OR CONSEQUENTIAL OR ANY LOSSES, SUCH AS BUT NOT LIMITED TO LOSS OF PROFIT, REVENUE, BUSINESS INTERRUPTION, BUSINESS OPPORTUNITY OR DATA,THAT MAY ARISE FROM THE USE OF THIS DOCUMENT OR THE INFORMATION IN IT. This documentation and the product it describes are considered protected by copyrights and other intellectual property rights according to the applicable laws. The wave logo is a trademark of Nokia Siemens Networks Oy. Nokia is a registered trademark of Nokia Corporation. Siemens is a registered trademark of Siemens AG. Other product names mentioned in this document may be trademarks of their respective owners, and they are mentioned for identification purposes only. Copyright Nokia Siemens Networks 2010. All rights reserved

Important Notice on Product Safety


Elevated voltages are inevitably present at specific points in this electrical equipment. Some of the parts may also have elevated operating temperatures. Non-observance of these conditions and the safety instructions can result in personal injury or in property damage. Therefore, only trained and qualified personnel may install and maintain the system. The system complies with the standard EN 60950 / IEC 60950. All equipment connected has to comply with the applicable safety standards.

The same text in German: Wichtiger Hinweis zur Produktsicherheit In elektrischen Anlagen stehen zwangslufig bestimmte Teile der Gerte unter Spannung. Einige Teile knnen auch eine hohe Betriebstemperatur aufweisen. Eine Nichtbeachtung dieser Situation und der Warnungshinweise kann zu Krperverletzungen und Sachschden fhren. Deshalb wird vorausgesetzt, dass nur geschultes und qualifiziertes Personal die Anlagen installiert und wartet. Das System entspricht den Anforderungen der EN 60950 / IEC 60950. Angeschlossene Gerte mssen die zutreffenden Sicherheitsbestimmungen erfllen.

Id:0900d805807a32c5

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

Table of Contents
This document has 79 pages. Summary of Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1 2 2.1 2.2 2.2.1 2.2.2 2.2.3 2.3 2.3.1 2.3.1.1 2.3.1.2 2.3.2 2.3.2.1 2.3.2.2 2.3.2.3 2.3.3 2.3.4 2.3.4.1 2.3.4.2 2.3.5 2.4 2.4.1 2.4.2 2.4.3 2.4.3.1 2.4.3.2 2.5 3 3.1 3.1.1 3.1.2 3.1.2.1 3.1.2.2 3.2 3.3 3.3.1 3.3.1.1 3.3.1.2 3.3.1.3 3.3.1.4 3.3.1.5 3.4 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Measurement Methods and Reference Conditions . . . . . . . . . . . . . . . . Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Reference Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Distinguished Cell Positions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Channel Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Template for the Description of Reference Conditions . . . . . . . . . . . . . Measurement Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Throughput Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Cell Throughput . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Peak User Data Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Latency Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Round Trip Time Measurement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Signaling and Media Delay Measurements . . . . . . . . . . . . . . . . . . . . . . Reference Conditions for Latency Measurements. . . . . . . . . . . . . . . . . Service Accessibility Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . Reliability Measurements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Transport Error Rates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Service Drop Rates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mobility (Handover) Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . Measurement Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Field Network Cluster. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Trial Lab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Measurement Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Subscriber and Equipment Traces. . . . . . . . . . . . . . . . . . . . . . . . . . . . . PM Counters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Measurement Reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Definitions of Key Performance Indicators (KPIs) . . . . . . . . . . . . . . . . . Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Application Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . LTE Network Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . LTE E2E Network Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Radio Bearer Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . KPI Definition Template . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Application Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data Download / Upload (FTP) Service KPIs . . . . . . . . . . . . . . . . . . . . (FTP) Service Accessibility Ratio. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (FTP) Completed Session Ratio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (FTP) Session Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (FTP) Service Access Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (FTP) User Data Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . LTE Network Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 11 12 12 13 14 16 16 17 19 20 20 21 22 23 25 25 27 28 30 30 31 31 31 32 33 34 34 35 36 36 37 38 39 39 39 40 41 42 45 46

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a32c5

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

3.4.1 3.4.1.1 3.4.1.2 3.4.1.3 3.4.1.4 3.4.1.5 3.4.1.6 3.4.1.7 3.4.1.8 3.4.1.9 3.4.1.10 3.4.1.11 3.4.1.12 3.4.1.13 3.4.1.14 3.4.1.15 3.4.2 3.4.2.1 3.4.2.2 3.4.2.3 4

LTE E2E Network KPIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 Attach Time. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 Detach Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 Attach Success Rate. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 Service Request Time (EPS), UE Initiated . . . . . . . . . . . . . . . . . . . . . . . 53 Service Request (EPS) Time, Network Initiated . . . . . . . . . . . . . . . . . . . 57 Service Request (EPS) Success Rate . . . . . . . . . . . . . . . . . . . . . . . . . . 58 Service (EPS Bearer) Drop Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 Handover Procedure Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 Handover Success Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 Paging Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 Paging Failure Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 (LTE) Round Trip Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 (LTE) User Data Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 (LTE) Packet Loss Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 (LTE) Service Interrupt Time (HO) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 Radio Bearer KPIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 (RB) Packet Loss Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 (RB) User Data Rate. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 Cell Throughput . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

Id:0900d805807a32c5

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

List of Figures
Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Figure 7 Figure 8 Figure 9 Figure 10 Figure 11 LTE Bearer Service Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Distribution of Users for Cell Throughput Measurements . . . . . . . . . . . Reference Cluster for Field Performance Verification Tests . . . . . . . . . KPIs for Packet Switched Data Services . . . . . . . . . . . . . . . . . . . . . . . . Attach Procedure w/ Initial EPS Bearer Establishment (3GPP 23.401). UE-Initiated Detach Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . EPS Bearer Setup Procedure, Part 1 . . . . . . . . . . . . . . . . . . . . . . . . . . EPS Bearer Setup Procedure, Part 2 . . . . . . . . . . . . . . . . . . . . . . . . . . Handover Procedure: Intra MME/S-GW, Inter eNB via X2 . . . . . . . . . . Handover Procedure: Intra MME/S-GW, Inter eNB via S1 . . . . . . . . . . Paging Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 17 30 43 48 50 55 56 63 64 68

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a32c5

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

List of Tables
Table 1 Table 2 Table 3 Table 4 Table 5 Table 6 Table 7 Table 8 Table 9 Table 10 Table 11 Table 12 Table 13 Table 14 Channel Models and Doppler Frequencies . . . . . . . . . . . . . . . . . . . . . . 13 Description of Reference Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 KPI Categories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Reference Conditions for Cell Throughput Measurements . . . . . . . . . . 18 Reference Conditions for User Data Rate Measurements . . . . . . . . . . . 20 Reference Conditions for Latency Measurements . . . . . . . . . . . . . . . . . 23 Reference Conditions for Success Rate Measurements . . . . . . . . . . . . 24 Reference Conditions for Packet Loss Rate Measurements . . . . . . . . . 26 Reference Conditions for Service Drop Rate Measurements . . . . . . . . 28 Reference Conditions for Handover Measurements . . . . . . . . . . . . . . . 29 Overview of Application Service KPIs . . . . . . . . . . . . . . . . . . . . . . . . . . 35 Overview of LTE E2E Network Service KPIs . . . . . . . . . . . . . . . . . . . . . 36 Overview of LTE Radio Bearer Service KPIs . . . . . . . . . . . . . . . . . . . . 37 KPI Definition Template . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

Id:0900d805807a32c5

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

Summary of Changes

Summary of Changes
Issue History Issue number 01 DRAFT Date of issue 2010-06-14 Reason for update First issue for RL15 pre-release

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a32af

Summary of Changes

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

Id:0900d805807a32af

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

Introduction

1 Introduction
The current document provides the following information: Definitions of End-to-End Key Performance Indicators (E2E KPIs) for Long-Term Evolution (LTE) Descriptions of measurement methods for field and lab trials

KPIs are basic network quality indicators for offer processing and field acceptance tests. The KPIs specified in this document are E2E related KPIs; they help network operators and Nokia Siemens Networks to determine the quality of deployed LTE networks. In order to prove that the performance of an operating network matches the targeted quality, field acceptance tests are carried out. As multiple factors may influence the performance of an operational network, detailed conditions, under which network KPIs are valid, must be specified and agreed upon. These include system performance, network planning and dimensioning, measurement campaign planning and parameterization. These conditions will be reffered to as "reference conditions" in the sequel. LTE system architecture was designed to meet the following major goals. These are also the constant focus of lab and field performance tests (for more details, see References 7. [3GPP25.913]): Improved system capacity & coverage High user data rates (peak upload and download rates) Reduced latency (one-way delay and RTT) Simplified architecture, IP based transport infrastructure Superior user experience Seamless connection to legacy networks (GSM, UMTS, 3GPP2/CDMA)

These improvements can be assessed with KPI measurements as defined in this document. The KPIs are defined for different service levels (see References 20. [3GPP 36.300], and Figure 1 LTE Bearer Service Architecture): a) b) c) d) Application Services LTE E2E Network Service Radio Bearer (RB) Service IP based LTE Bearer Services: X2, S1, S5/S8 and External

The KPIs of group B (LTE E2E Network Service) characterize the quality of the LTE network as IP transport service for applications. They are in the center of field performance investigations, because they characterize the overall LTE network performance. The Application Service KPIs of group A (Application Services) show the user-perceived quality of application services as impacted by the LTE network. The KPIs of this group are service specific. The following services are of common interest of mobile network operators: Web Browsing Data Upload / Download Audio Video Streaming Conversational Audio Video (VoIP and Video Telephony)

This paper concentrates on the Data Upload / Download (File Transfer Protocol (FTP)/Transmission Control Protocol (TCP)) application service, because it is the

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a32b1

Introduction

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

support of this service that stands in focus of the first commercial NSN LTE product release. In addition, KPIs for the radio link (RB Service, group C - Radio Bearer Service) are also specified, because the radio link is the most critical component of the E2E transport path. The other bearers, the E-UTRAN Radio Access Bearer (E-RAB) and Evolved Packet System (EPS) bearers are composed of the RB Service and of the IP based bearer services (of group D - IP based LTE Bearer Services: X2, S1, S5/S8 and External). Monitoring the performance of all bearer services is necessary to fully understand the E2E transport behavior of the LTE network and to optimize the end-user perceived quality of application services running on top of it.

Figure 1

LTE Bearer Service Architecture

While the KPI definitions (see Definitions of Key Performance Indicators (KPIs)) are independent of LTE product releases, Measurement Scenarios focuses on the first product release in order to restrict the number of possible scenario combinations (e.g. by not considering scenarios with Guaranteed Bit Rate (GBR) bearers).

10

Id:0900d805807a32b1

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

Measurement Methods and Reference Conditions

2 Measurement Methods and Reference Conditions


2.1 Introduction
The measurement method to be applied for field performance verification depends on the specifics of the measured Key Performance Indicator (KPI). End-user perceived KPIs are measured differently from network performance KPIs, and the conditions for delay measurements are different from those for accessibility KPIs. For this reason, this Section discusses measurement methods according to KPI categories. Since the end-user experience of application services is mostly dependent on throughput and latency, these aspects together with Handover (HO) scenarios constitute the core part of field performance verification measurements. While the scope of KPI definitions follows the performance goals of the LTE system (for details see References 7. [3GPP25.913]), the selected methods and conditions of field performance tests have to ensure that typical conditions in operational networks are covered. Additionally, the measurement conditions also include relevant impairment factors of the radio link, because the radio link has the most influence on E2E performance. Especially, those characteristics of the Orthogonal Frequency-Division Multiplexing (OFDM) encoding technique, which have side effects for radio network performance are considered in the selection of conditions, e.g.: Sensitivity to Doppler shift Sensitivity to frequency synchronization problems Sensitivity to multipath effects and cell area (urban, sub-urban, rural) Dependency on used bandwidth and frequency range Dependency on used multiple antenna techniques - Multiple Inputs Multiple Outputs (MIMO) (transmit diversity, spatial multiplexing)

These aspects have a major bearing on the design of measurement scenarios. For example, investigating the impact of Doppler shift on network performance, e.g. on user and cell data rates, on delay and packet loss KPIs requires that high speed movement becomes part of the measurement scenario. Another aspect would be the multipath reception, which decreases the impact of the Doppler shift. Thus, the measurement setup should also include space diversity if worst-case conditions are being discussed measurements with these conditions have to show, how the investigated KPI figure depends of the velocity and terminal category of the receiving user. Some of the measurement scenarios might be much too costly to implement in live network environments, or it may be impossible to ensure the desired radio conditions during the whole duration of the test. In such cases, lab trials with simulated radio links are acceptable as replacement of live network tests. The combination of all different options for reference conditions and measurement scenarios would result in a very large number of test cases, which is simply not feasible. It will be the task of test specifications to select options and scenarios out of the outlined possibilities for a given LTE performance verification campaign considering the available resources and time constraints for test execution.

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a32b3

11

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

2.2

Reference Conditions
Performance figures can only be interpreted correctly if the exact conditions under which they have been acquired are known. This chapter describes the "reference conditions" under which LTE field performance tests should be executed. They are formulated on an abstract level.

2.2.1

Distinguished Cell Positions


Operators are keen to know the distribution of throughput and latency KPI figures (e.g. peak user data rate, cell UL/DL throughput, network delay, etc.) across the cells of a "reference cluster". However, it is impossible or would be associated with prohibitive high costs to measure these KPIs in a live network environment in so many cell positions that a Cumulative Distribution Function (CDF) can be calculated for 5%, 50%, or 95% of the cell area. For this reason, it is proposed to replace the distribution function by KPI figures measured on three characteristic positions of the cell (in the sequel, the term "cell" will be used to denote both omni-directional and directional cells): a) Very Good radio conditions (line of sight close to cell center within 50m, low path loss) b) Medium radio conditions (BTS distance of ~1/2 cell radius, channel with medium noise conditions, medium path losses) c) Cell Edge conditions (cell overlapping area, receive power from neighbor cells at about the same level). The radio conditions are formulated in terms of Signal to Interference and Noise Ratio (SINR) and receive power levels. The measured KPI figures in Good, Medium and Cell Edge positions replace their distribution across the LTE cell and will be considered as their Peak, Average and Minimum values, respectively. The three variants are given by mean and 95% values calculated from all measurement samples, where the number of samples is to be derived from the required confidence level. If no suitable cell positions are available in the reference cluster, the tests can be executed in the laboratory, where the radio conditions are to be simulated. If automated test execution is possible, even the calculation of CDF function values at 5%, 50% and 95% distribution is an option. This shows that often the full set of results can only be produced by combined field and lab trials. In certain cases, even simulations might become necessary to accomplish (incomplete) measurement results. For this reason, it is best practise to measure all E2E field network KPIs in vendor lab before going into field tests with the operator. The description of test conditions (see Table 2 Description of Reference Conditions, row "Setup") specifies which test methods are recommended.

12

Id:0900d805807a33b4

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

2.2.2

Channel Models
Several factors of the radio link have influence on the E2E performance of application, or LTE network services. Their importance is dependent on the category of the KPI to be measured. The following parameters are checked for each scenario: UE category Antenna systems on UE and eNB Position and distribution of terminals in the cell, terminal movement Load conditions in the measured cell (UL/DL, data volume) Receive power level, interference and noise conditions, diversity and fading Cell area, or clutter type (urban, sub-urban, rural)

In order to reduce the number of possible combinations, mobile terminal speed and radio propagation conditions for different environments and distances were combined in "channel models". The relevant channel models for LTE are described in References 30. Annex B of [3GPP36.521-1]. The following three channel models are defined: EPA - Extended Pedestrian A EVA - Extended Vehicular A ETU - Extended Typical Urban

In addition, 3GPP proposes some combinations of channel models and Doppler frequencies, which are complemented in Table 1 Channel Models and Doppler Frequencies with scenarios for stationary UEs (EPA0, EVA0 and ETU0), and high speed train (ETU250)

The high speed train scenario with app. 250 km/h uses a carrier frequency of 900 MHz (former GSM band) since it is assumed that initial LTE deployments will use this frequency in rural areas. If the assumption is not true for a given customer trial, the Doppler frequency will change proportionally to the applied carrier frequency (500Hz @ 1.8 GHz, 583 Hz @ 2.1 GHz, 722 Hz @ 2.6 GHz). The current document uses only these models as reference models. Model EPA0 EVA0 ETU0 EPA5 EVA5 EVA70 ETU70 ETU250 ETU300 Table 1 3GPP36.521-1, Annex B; see 30. 3GPP36.521-1, Annex B; see 30. 3GPP36.521-1, Annex B; see 30. 3GPP36.521-1, Annex B; see 30. 3GPP36.521-1, Annex B; see 30. Source Maximum Doppler Frequency 0 Hz (stationary UE) 0 Hz (stationary UE) 0 Hz (stationary UE) 5Hz 5Hz 70Hz 70Hz 250 Hz (with 900 MHz carrier frequency) 300Hz

Channel Models and Doppler Frequencies

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a33b4

13

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

Channel model conditions can be set in lab trials with simulated radio links precisely, but they cannot be ensured in field network tests. In field trials, radio conditions with best suit to the channel model are used to calibrate trial lab results.

2.2.3

Template for the Description of Reference Conditions


Table 2 Description of Reference Conditions includes the channel model together with other parameters to summarize the essential conditions for measurement scenarios. The column "Reference Condition" gives textual descriptions and possible settings as abstract values only. The meaning of rows is as follows: "Antenna" identifies the number, type and modulation of antennas used on the UE and on the eNB side. The reference UE configuration comprises two receive antennas in DL and one antenna in UL. (However, it may not be freely selectable in reference clusters of operational networks.) "The "Application" parameter identifies the test application as a real application, or an equivalent replacement (test script), which allows the collection of information necessary to calculate / extrapolate the KPI figure. The "Setup" parameter identifies the environment, where the KPI can best be measured. It is not always the reference cluster of the live network. Some conditions, e.g. SINR are easier to set in the lab. The "Reference Cluster" denotes a selected area of the operators live network performance verification tests. In the cluster typical load conditions apply, as generated by real users -stationary or mobile - and the radio conditions are determined by cell location (rural, or city area, high, or low buildings, etc.). The conditions are given and cannot be modified during the trial. The "Trial Lab" is either a vendor or operator lab used for executing performance tests, where exact measurement conditions can be ensured. The lab allows the background traffic mix, load levels, and radio conditions to be precisely set. The simulation of mobile users of high velocity is also possible. "Simulation" results are applied where field and even lab trials are expensive or cannot be executed in the given time or budget. "Concurrent Load" describes the load which is to be generated in addition (i.e. background traffic) to the traffic produced by active test terminals in the measured cell. Load conditions in neighbor cells are the operational load in field network clusters, and are set to medium load in the lab. "Drive test "Routes" are defined for mobile terminals in live LTE environments. In trial labs, terminal movement and hand-over are simulated (by varying the channel conditions). "Backhaul Capacity" defines the necessary throughputs of transport links if their values deviate from default and are important for test execution, e.g. to prevent a transport link becoming bottleneck in user data rate, or cell throughput measurements. It also defines other parameters, like Packet Delay (PD), Packet Delay Variation (PDV), Packet Loss Rate (PLR) and their expected values. These parameters are observed in field trials, and are set to their maximal allowed values in the lab.

14

Id:0900d805807a33b4

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

Parameter Terminal (UE) Type Category Antenna Position Distribution Network Environment Channel Model Backhaul Capacity QoS Configuration Concurrent Load Application Traces Setup Route Table 2

Reference Condition

Mobile or smart phone, laptop with LTE card, or w/ modem UE categories 1-5 according to [3GPP36.306]; see 21. Type: SISO, MIMO1*2, MIMO2*2, MIMO4*2, etc. Modulation: QPSK, 16QAM, 64QAM Good, medium and cell edge (for each terminal) Equal, or non-equal distribution of "1 to n" UEs

EPA, EVA, ETU with different Doppler frequencies Capacities of X2, S1-U, S1-MME transport links. Link weight and priority assigned to the flow(s) Unloaded, or concurrent UL/DL data volumes Application service or traffic generation for LTE E2E, EPS, E-UTRAN, EPC bearers. Trace points & data to be collected Trial Lab, Reference Cluster, or Simulation In reference cluster, drive test route description.

Description of Reference Conditions

The number of test executions (sample size) is not defined as a parameter in the template, but it is to be defined for each measurement scenario so that the calculated KPI figure has the required statistical confidence.

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a33b4

15

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

2.3

Measurement Scenarios
Measurement Scenarios describes the measurement methods and recommended scenarios according to KPI categories summarized in Table 3 KPI Categories. The first column indicates the respective QoS terminology defined by the ITU [ITU-T.E800], and referenced by 3GPP in References 12. [3GPP32.410] and 17. [3GPP32.450] for LTE related KPIs. The third and fourth columns indicate the selected terminology and explanation for its usage in the current paper. ITU QoS Category Accessibility 3GPPKPI Category Accessibility Terminology Used Accessibility Meaningfor LTE Field Network Performance Service accessibility, e.g. Attach, or EPS Bearer Setup Request success rates. Once established, the reliability of the service, e.g. VoIP call, or EPS bearer drop rates. Promise to deliver the service with a certain quality, level of conformance. E.g. cell throughput, or user data rates, packet delay. Relationship between service up and down times, e.g. cell, or LTE network availability Different mobility related metrics, e.g. hand-over success or drop rates, latencies.

Reliability

Retainability

Reliability

Integrity

Throughput, Latency

Throughput, Latency

Availability

Availability

not used

not defined

Mobility

Mobility

Table 3

KPI Categories

Since the KPI category is part of all KPI definitions (see KPI Definition Template ), it is easy to lookup the corresponding measurement method to each KPI.

2.3.1

Throughput Measurements
Throughput is the primary metric for the characterization of the LTE radio technology (Orthogonal Frequency Division Multiple Access (OFDMA) in DL, Single-Carrier Frequency Division Multiple Access (SC-FDMA) in UL). It shows the data rate that can be provided to users among different radio conditions. The throughput of an LTE cell is expressed as a function of the number of concurrently active users and the amount of aggregate traffic that the cell can still support. The throughput the LTE system can offer depends on many factors: Channel environment (e.g. stationary or mobile, speed) and fading conditions. Reception conditions impaired by traffic load levels, and by interference between the cells, in short by the user's SINR. Network layout, type of antenna. Position of users in the cell (implies e.g. path loss and fading).

16

Id:0900d805807a33b6

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

Thus, it is obvious that the location of the mobile terminal in the cell will strongly influences the data rate that can be experienced by the user. For this reason, the capacity of an LTE cell is characterized by two distinguished metrics, one viewing capacity from the operator's point of view (How many users can be supported by the cell?), the other from the end-user perspective: Cell Throughput, and Peak User Data Rate

2.3.1.1

Cell Throughput
The metric "Cell Throughput" shows the sustainable aggregate cell capacity available to a number of "n" users. In order to approximate a typical operating environment, the users are distributed uniformly in the cell and use a typical mix of applications concurrently. The Figure 2 Distribution of Users for Cell Throughput Measurements shows how users are arranged for cell throughput measurements. Each user position is characterized by different combinations of path loss, receive signal, and neighbor cell interference strengths.

Figure 2

Distribution of Users for Cell Throughput Measurements

The measurement is done with stationary terminals to avoid the impact of movement and Hand-Over (HO). The latter implies that for those mobiles that are located at the cell edge, HO must be suppressed on the UE or by the network. The proposed reference conditions for cell throughput measurements are summarized in Table 4 Reference Conditions for Cell Throughput Measurements.

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a33b6

17

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

Parameter Terminal UE Type UE Category Antenna Position Distribution

Reference Condition

Laptop with LTE card Different categories, KPI specific Types: SISO, MIMO 1*2, 2*2, ; Good, medium, cell edge (see Distinguished Cell Position) "1-n" terminals, equal distribution in the cell

Network Environment Channel Model Backhaul Capacity QoS Configuration Concurrent Load Application Traces Setup Route Table 4 Stationary outdoor - EPA0, EVA0, ETU0 with different SINRs Capacities of X2, S1-U, S1-MME transport links. Non-real-time QoS, equal weights and priorities Unloaded (no concurrent load) Data upload / download (FTP); separate UL/DL and concurrent transmission. UE, Uu, eNB, S1-U, FTP Server Reference cluster in live network N.a. Reference Conditions for Cell Throughput Measurements

The default backhaul capacity of S1-U transport links are to be controlled and modified if necessary to prevent this link becoming a bottleneck during the measurement. The modified value should become default for subsequent measurements. Cell throughput measurements should be executed with an increasing number of UEs starting with 1. While the first test case corresponds to the peak user data rate measurement, the subsequent ones illustrate the dependency of this KPI on the number of active

18

Id:0900d805807a33b6

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

users. The results (cell throughput and peak user data rate) are to be illustrated together in a diagram like the example below:

2.3.1.2

Peak User Data Rate


The "Peak User Data Rate" metric describes the data capacity that is available to one user in a cell. Data rate is the most important single factor which influences the end-user experience of non-real-time application services, like web browsing, email, File Transfer Protocol (FTP) up/downloads, and interactive gaming, due to its impact on (signaling and data) transfer delays. The goal of field performance measurements is to investigate the sensitivity of this metric to LTE impairment factors. The realistically achievable data rates under concurrent load (i.e. with concurrently active users in the cell) in stationary case are shown with cell throughput measurements as described in Cell Throughput. Since the available peak user data rate is influenced by any or all of the following conditions, Channel environment (mobility, stationary) Radio conditions (signal power, path loss, SINR) Aggregate cell load (i.e. intra-cell and inter-cell traffic) Restriction of user data rates (e.g. by terminal category) Link sharing weights (Quality of Service (QoS) configuration) Backhaul capacity

the measurement scenarios include variations of these parameters. User data rates are also measured in cell edge conditions, even including hand-over. The reference condi-

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a33b6

19

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

tions for user data rate tests are summarized in Table 5 Reference Conditions for User Data Rate Measurements. Parameter Terminal UE Type UE Category Antenna Position Distribution Smart phone and laptop with LTE card Corresponding UE categories Types: SISO, MIMO 1*2, 2*2, etc. Good, medium, and cell edge (see Distinguished Cell Position) One single terminal Reference Condition

Network Environment Channel Model Backhaul Capacity QoS Configuration Concurrent Load Application Traces Setup Route Table 5 EPA, EVA stationary and mobile w/ 3, 50 km/h Capacities of X2, S1-U, S1-MME transport links. Non-real-time QoS, equal weights and priorities Unloaded FTP upload, download; separate UE, eNB, FTP Server; Uu, S1-U, S1-MME, X1, ... Reference cluster in live network Drive test route passing good and medium cell positions, and crossing cell borders. Reference Conditions for User Data Rate Measurements Ideally, the results of field or lab trial measurements of this KPI would be a full distribution of Peak User Data Rate values (a Cumulative Distribution Function (CDF)) across the cell. For practical reasons, however, it is sufficient to produce the values in selected cell positions as described in Distinguished Cell Position. Radio positions (good, average and poor SINR) that are best suited to the channel model of stationary users is selected. Mobility tests with up to 50 km/h can be executed in field reference clusters, but higher speeds, e.g. high speed train with ~350 km/h are to be executed in simulated a lab environment.

2.3.2 2.3.2.1

Latency Measurements Round Trip Time Measurement


The benchmark measurement for finding the E2E latency of LTE access networks for data applications is the Round Trip Time (RTT) measurement. RTT is measured with the Ping application of the UEs operating system or with a comparable measurement tool. It records the time difference between sending an Internet Control Message Protocol (ICMP) Echo Request to an IP host, and the reception of the corresponding ICMP Echo Reply message. The measurement can be executed between a mobile terminal and an IP host, which is configured to respond to ICMP Echo Requests. This way, round trip delays can be measured between UE - eNB (LTE Uu), UE - S/P-GW (E-UTRAN Radio Access

20

Id:0900d805807a33b6

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

Bearer(E-RAB), EPS) and UE - AS (LTE E2E), respectively. The measurement can (and should) be executed in both directions. The Ping application sends successive ICMP Echo Requests either upon reception of the ICMP Echo Reply to the previous request, or after a predefined period of time (in the range of ~100ms to 1s)

This is done to ensure that only one Ping request/response is under way at any given time. This way, the delay time will have no bias caused by queuing, or scheduling delays along the transport path. The total number of measurements per cycle should be chosen so that results have the required statistical confidence level. The recommended number of measurements per IP host is 100. If the measurement is executed over the radio link (LTE Uu), the first IP packet might suffer additional delay caused by the dynamic allocation of radio resources to non-realtime (non-GBR) bearers. For this reason, the round trip delay of the first packet should not be considered in statistic calculations. On the other hand, the first ping measurement can be used to determine UE state transition delays if the UE starts from non-registered or idle states (from non-registered to registered, or from idle to connected states). The periodicity of Pings needs to be selected so that the once allocated radio bearer is kept over the whole duration of the measurement. The test is to be done for stationary and mobile end users. The measurement with stationary UE should be performed under average radio link conditions with normal SINR. For moving UEs, two different positions are to be selected, one in average radio link conditions and normal SINR, and another in the cell overlapping area. It is important to make sure that handovers between cells are avoided. The reference conditions should be monitored during the test using UE performance measurements; also, relevant trace data can be collected from the network. In lab trials, the measurements need to be executed with and w/o concurrent load. From the measurement results (samples) mean figures, standard deviation and 95% delays need to be calculated.

2.3.2.2

Signaling and Media Delay Measurements


If the bandwidths of UL and DL bearers in RTT measurements are identical, the benchmark value of UL and DL latencies can be given as 1/2 RTT. However, latency KPIs of signaling procedures on the network or application level (e.g. Radio Access Bearer (RAB) service setup, or VoIP call setup flows) are often complex and include the exchange of more than one messages of different sizes. In addition, the link capacities used in UL and DL are often different, e.g. in case of the application services data download, gaming or video streaming. In these cases, the RTT or 1/2 RTT figures are not sufficient to characterize the latencies incurred by the mobile access network. Here, the time synchronization of network elements is the central problem, since the start and stop triggers of the measurement lie on different Network Elements (NE) and interfaces. In case of measuring latencies over the radio link, the time needed to attach the mobile to the network and to setup the necessary radio resources is a significant component. In these cases, the measurement should be made with different initial states of the mobile, e.g. power off, idle, connected and active states. LTE State Models lists the dif-

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a33b6

21

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

ferent states of Mobile Terminations (MT) in 3GPP LTE specifications, see References 3. [3GPP23.401]. In field tests, latencies should be measured with the typical initial state of the mobile. The initial state will depend on the measurement scenario, e.g. Idle for network attach, or Connected for EPS bearer setup. Depending on the location of trigger points, end-user visible delays, LTE network delays, different bearer (RB, E-RAB and EPS) delays, transport and NE processing delays can be described similarly. The following aspects need to be specified for latency measurements: Start and stop triggers of the media or signaling flow. The trigger point is defined by the message type, NE and interface; it identifies where time stamps are to be recorded. In case of measuring one-way delays, the method of time synchronization. Trace points and analysis tools. In addition to trace points in the network, traces may be needed on mobile terminals when measuring end-user visible latencies, or E2E LTE network delays.

2.3.2.3

Reference Conditions for Latency Measurements


Field measurements of latencies are requested under reference radio conditions (equal to cell medium), which are specified by the acceptable ranges of UL/DL SINR and of receive power levels. In addition, the following variations of conditions are planned (see also Table 6 Reference Conditions for Latency Measurements): Unloaded versus loaded cell conditions Stationary UE versus mobile on the move Drive route for measurements with velocity from medium to cell edge position including hand-over

22

Id:0900d805807a33b6

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

Parameter Terminal UE Type UE Category Antenna Position Distribution

Reference Condition

Mobile phone, smart phone, laptop with LTE card UE category 1-5 according to [3GPP36.306]; see References 21. One antenna: e.g. SISO Medium, overlapping area One terminal

Network Environment Channel Model EVA0, EVA70 (stationary, mobile w/ ~50 km/h)

Backhaul Capacity Capacities of X2, S1-U transport links. QoS Configuration Real-time and non-real-time QoS, the latter with equal weights and priorities Concurrent Load Application Traces Setup Route unloaded and loaded with concurrent UL/DL data of medium volume (below cell target capacity) Data upload download and real-time over EPS bearer. Trace points, send & receive packet counts Field Reference Cluster, Trial Lab From radio position with reference condition (medium) to overlapping area including hand-over. Reference Conditions for Latency Measurements

Table 6

Those samples that were taken in measurements during which the specified conditions (reference, or explicitly modified ones) were not met, have to be excluded from the statistic calculations.

2.3.3

Service Accessibility Measurements


Accessibility KPIs characterize the accessibility of a service, or service element to its users. They are expressed as a ratio of successful service requests to the total number of attempts. (This group of KPIs corresponds to the category of "service accessibility KPIs" in References 12. [3GPP32.410].) The KPI value is calculated as: Success Rate = (# successful service requests / # total attempts) * 100 [%] For this test to be meaningful, a significant number of samples need to be collected. E.g. to get the success rate with 0.2% granularity, at least 500 samples are needed. Service request attempts that are rejected due to insufficient access rights, bad user authentication, or errors in request parameters are not considered and should be taken out of the samples used for calculating the metric. The measurements should be executed with the following variations of reference conditions:

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a33b6

23

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

Application is selected according to the mobile network service procedure that is tested. Test scripts are used for the automated execution of service requests, for the evaluation of responses and for the calculation of the KPI figure. Mobile terminals in stationary and mobile use. The mobile use includes drive routes among good and medium receive conditions, as well as a route that crosses cell overlapping areas. In the latter case, hand-over is executed. The frequency of handovers follows the assumptions of the NSN reference traffic model for LTE networks. Unloaded and loaded cell conditions. Concurrent load is to be generated in lab trials with low and medium levels (as a certain percentage of the cell target load). In field network tests, where a certain background load is given, load levels are to be traced to allow for scaling (calibrating) of lab trial results.

The reference conditions are summarized in Table 7 Reference Conditions for Success Rate Measurements. Parameter Terminal UE Type UE Category Antenna Position Distribution Laptop with LTE card, or modem. UE category according to [3GPP36.306]; see References 21. One antenna: e.g. SISO Good, medium and cell edge area. n.a. (one terminal only) Reference Condition

Network Environment Channel Model EVA0, EVA70: stationary, mobile w/ ~50 km/h

Backhaul Capacity Capacities of X2, S1-U, S1-MME transport links. QoS Configuration n.a. Concurrent Load Application Traces Setup Route Unloaded or concurrent UL/DL data volumes of low and medium levels. Procedure dependent test script for automated execution of different applications. Data collected by test script. Monitoring of online statistic counters. Field Network Cluster and Trial Lab. Real drive test routes (in Cluster), or simulated with varying channel models (in Lab). Reference Conditions for Success Rate Measurements

Table 7

Due to the high number of test executions, the measurements can be executed in lab environment with simulated air links. At least the stationary tests should be executed in field network cluster. Automated test execution is needed in both environments. In addition to the KPI figures calculated by the test script, test results should include online performance measurement statistics, which are gained by monitoring the performance counters of LTE network elements (i.e. on the eNB, MME, S-GW, P-GW). Please

24

Id:0900d805807a33b6

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

note, however, that the success ratio figures calculated from eNB counters will not be identical to the success ratio measured by the test UE.

2.3.4

Reliability Measurements
Due to similarities in their measurements, the following types of KPIs are discussed together in this chapter. Service Drop Rates (Register, Attach, Service Request, etc.) Transport Error Rates (Block Error Rate (BLER), Packet Loss Ratio (PLR), Frame Error Rate (FER), etc.)

The service drop rate KPIs are categorized as Retainability by 3GPP, and Reliability by the ITU. The transport error rate KPIs are categorized as Integrity KPIs by 3GPP, and by the ITU.

2.3.4.1

Transport Error Rates


These KPIs put the number of erroneous or lost data units to the overall number of data units sent into relation. Depending on the payload of the transmitted data unit, the ratio is given as BLER for radio blocks, FER for voice and video frames, and Packet Loss Rate for IP packets. The chapter deals with the method of measuring PLR in the field. Measurements of BLER and FER are planned for lab tests only; their methods will be provided elsewhere. E2E IP PLR is measured with the IP flow of an application service over a single EPS bearer between the UE and the Application Server (AS) connected directly to the P-GW (first hop IP router). The RAB is selected with appropriate QoS handling (i.e. nGBR bearers for data applications and GBR bearers for real-time flows). The measurement equipment for PLR is placed on the UE side and on the AS side to measure PLR in both directions (UL and DL). It can either be a professional measurement tool, or a simple test script. PLR KPIs are measured across the following variations of conditions: Application service: data upload / download, and real-time (e.g. video streaming) Mobile terminal in stationary and mobile use The mobile use includes drive routes among good and medium reception conditions, as well as a route that crosses cell overlapping areas. In the latter case, a handover is executed. Operational load in source and destination cells of the field network cluster. Concurrent load generated in lab trials according to the reference traffic mix at low, medium and high levels.

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a33b6

25

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

Parameter Terminal UE Type UE Category Antenna Position Distribution

Reference Condition

Laptop with LTE card UE category tbd. according to [3GPP36.306]; see 21. One antenna: e.g. SISO Good and medium in originating cell, drive across hand-over area. n.a. (one terminal only)

Network Environment Channel Model EVA0, EVA70: stationary, mobile w/ ~50 km/h

Backhaul Capacity n.a. QoS Configuration n.a. Concurrent Load Application Traces Setup Route Loaded & unloaded in source & destination cells Data download (UDP) and video streaming. UE and eNB, mobility events Reference Cluster, Trial Lab. Drive test route passing best and medium cell positions, and crossing cell borders. Reference Conditions for Packet Loss Rate Measurements

Table 8

One KPI value (measured sample) is calculated for each scenario as: Packet Loss Rate = (# not received packets / # sent packets) * 100 [%] The PLR is given as mean value over all samples of the same measurement scenario. For this test to be meaningful, a significant number of samples needs to be collected. For this reason, the test should run for at least one hour in the lab. It is not necessary to make a detailed analysis of the reasons behind packet losses, as it is difficult to store extremely large amounts of trace data. However, if professional test equipment is available for this analysis, the measurement setup may include it. Online performance statistics on PLR can be collected during the measurement if the corresponding Performance Management (PM) counters are available on LTE network elements (i.e. on the P-GW). Please note, however, that the PLR figures collected may not be identical to the PLR measured on the test UE and on the AS.

26

Id:0900d805807a33b6

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

2.3.4.2

Service Drop Rates


Attach, Service Request, Register, etc. are examples of service requests on different service levels; service requests establish a relationship between the UE and the network or application. The relationship is normally terminated with a "Release Request" by the UE. In case of network errors (e.g. handover failures), or insufficient radio resources the relationship can be terminated unexpectedly. The ratio of such "abnormal" terminations to the number of all established relationships is given by "drop rate" KPIs. For certain services, like bearer creation or VoIP call, the service drop rate is also related to the duration of the session. This is motivated by the fact that the possibility of network errors, e.g. hand-over failures is higher if the service duration is longer. In such cases, the service usage time should also be measured, and the drop ratio expressed in [1/s] units instead of [%]. The current version of this document does not consider session time for service drop rates. It will be added in the next update of this document. In order to measure the ratio, the given service request has to be executed in a sufficiently large number of times to assume statistical significance. E.g. each test scenario should be executed 100 times. After the relationship has been established, common operations of the service are to be executed for a typical period of service usage time. The LTE traffic model can deliver input for this. Due to the complexity of the test setup and the long time needed to execute the tests, the measurements are better executed in a lab environment with simulated air links and automated service execution. The measurements are to be executed under the following varying conditions: Mobile terminal in stationary and mobile use The mobile use includes drive routes among good and medium reception conditions, as well as routes that cross cell overlapping areas. In the latter case, a handover is executed. If executed in the trial lab, these conditions are set in simulated radio links, i.e. the "drive route" is simulated. Unloaded and loaded cell conditions. Concurrent load is to be generated up to full (or target) load level of the cell.

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a33b6

27

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

Parameter Terminal UE Type UE Category Antenna Position Distribution

Reference Condition

Laptop with LTE card UE category tbd. according to [3GPP36.306]; see References 21. One antenna: e.g. SISO Good, medium and cell edge area. n.a. (one terminal only)

Network Environment Channel Model EVA0, EVA70: stationary, mobile w/ ~50 km/h

Backhaul Capacity Capacities of X2, S1-U, S1-MME transport links. QoS Configuration Link weight and priority assigned to the flow(s) Concurrent Load Application Traces Setup Route Unloaded or concurrent UL/DL data volumes up to full cell load. n.a. Trace points & data to be collected Trial Lab with simulated air link conditions. The drive test routes are simulated using different channel models and varying radio conditions (receive level, SINR). Reference Conditions for Service Drop Rate Measurements

Table 9

2.3.5

Mobility (Handover) Measurements


Mobility KPIs from a group because all of them are related to hand-over procedures. They could, however, be discussed in the Latency and Accessibility groups, as well, because HO-related KPIs are either service interrupt time, or success ratio type KPIs. A successful hand-over is understood to be any successfully performed make-beforebreak procedure. For its measurement, a DL User Datagram Protocol (UDP) data stream is generated as the only load during the whole measurement. The test is performed in an overlapping area between two cells. The overlapping region of a cell is defined by the level of radio conditions (DL SINR and receive power levels) which are identical for neighboring cells. It can be expressed as the difference between receive sensitivity levels of neighboring cells. I.e. at the border line of the overlapping region the relative receive sensitivity level is 0 dB. Drive routes need to be specified so that they begin in good radio conditions of the originating cell and proceed directly through the hand-off area and end in high signal strength and good SINR in the other cell. Drive routes should proceed directly through the hand-off area in order to prevent multiple handoffs caused by stop lights, stop signs, traffic congestion as well as to prevent extending of the time period in the hand-over region. Routes must be carefully selected to avoid areas where handoff is driven through lack of coverage, which leads to breaks in connection. This is the most common cause of so-called "false failures".

28

Id:0900d805807a33b6

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

In order to measure handover KPIs, the logging of mobility events is necessary in mobile terminals. A successful handover is marked by an entry in the log, which carries the identifier of the new cell (destination). The HO KPIs are measured under the following variations of conditions (Table 10 Reference Conditions for Handover Measurements): Unloaded versus loaded cell conditions for Intra RAT, ideal conditions in target cell for Inter Radio Access Technology (RAT) (GSM, UMTS) hand-over scenarios. Mobile terminals with vehicular speed. Drive routes from good reception condition (in the originating cell) over hand-over area to a good position on the destination cell. Parameter Terminal UE Type UE Category Antenna Position Distribution Smart phone, laptop with LTE card UE category tbd. according to [3GPP36.306]; see References 21. One antenna: e.g. SISO Good/medium in originating cell directly across hand-over area to good/medium position in new cell. n.a. Reference Condition

Network Environment Channel Model EVA0, EVA70: stationary, mobile w/ ~50 km/h

Backhaul Capacity n.a. QoS Configuration n.a. Concurrent Load Application Traces Setup Route Table 10 Unloaded and loaded destination cell in UL/DL. UDP data stream in DL with hand-offs acc. to reference traffic model. MT, Source and Target eNBs, mobility events Reference Cluster, Trial Lab In reference cluster, drive test route description. Reference Conditions for Handover Measurements

The drive test has to be repeated at least 20 times to get the required level of statistical confidence. All reasons for handover failures (no resources in target cell, service not supported, etc.) should be excluded.

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a33b6

29

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

2.4
2.4.1

Measurement Setup
Field Network Cluster
For field performance verification tests of KPIs to be executed in live network environments, the following LTE Reference Cluster is recommended (see also Figure 3 Reference Cluster for Field Performance Verification Tests): Sufficient number and positioning of UEs as specified by the measurement scenario. All UEs are inside of the cells of the reference cluster. eUTRAN including at least three eNBs, one MME, and an integrated S-GW/P-GW. Each eNB supporting three LTE cells, each with predefined minimum radio conditions at cell borders. Provisioning of network equipment and setting of configuration parameters according to the guidelines of the LTE network operator and agreed with the vendor.

The recommendation considers constraints of the NSN LTE product release. The Reference Cluster environment must not change during the test campaign (no changes to network configuration and parameter values). Traces are to be activated on the UE, and on the network elements MME and S/P-GW as required by the measurement scenario and reference conditions of the given KPI. Trace data that can be collected on the interfaces LTE-Uu, S1, X1 and S11 are described in References 14. [3GPP32.423]. Traces should be taken such that the investigated KPIs (e.g. latencies) are not impacted. If this cannot be avoided, the measurements should be repeated without traces.

Figure 3

Reference Cluster for Field Performance Verification Tests

In addition to Figure 3 Reference Cluster for Field Performance Verification Tests, the final diagram of a test specification should contain indications of: At least three eNBs with three cells each Trace points on all NE interfaces Monitoring station or test equipment with adequate protocol stack (e.g. K12xx) to decode IP and radio network layer protocol messages

30

Id:0900d805807a33b8

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

Test workstations (WS) to generate background load WS for post-processing of traces Operations, Administrations and Maintenance Performance Management (OAM PM) for monitoring counters, displaying of statistics

2.4.2

Trial Lab
A description of the Trial Lab should be given with the performance verification test specifications.

2.4.3

Measurement Data Collection


The following ways of performance data collection are possible in KPI measurements: Collection of traces from terminals, network elements and network interfaces Collection and evaluation of counters from network elements by OAM PM

The preferred way of collecting performance data in a live network is via PM counters. It does not add additional load to the network and is a permanent source of information that can be monitored all the times. However, PM counters do not exist for all categories and for all individual KPIs. Typically, no counters are defined, or can be implemented for latency KPIs. For this reason, terminal, server and network traces are also collected. Since tracing puts additional processing load on these elements, traces are activated only on operator demand and for the time of test execution. On the other hand, traces include very detailed information on call level, which allows performance analyses of individual mobiles and operations. With traces, it is possible to go further in monitoring and optimization operations. For more details on LTE MMS and E-UTRAN traces, see References 13. [3GPP32.421] and 14. [3GPP32.423].

2.4.3.1

Subscriber and Equipment Traces


Subscriber and UE Traces should be activated on the UE, and on the network elements MME and S/P-GW of the Reference Cluster, as required by the scenario description (see Reference Conditions) of the given KPI measurement. Collected measurement data is transferred to an external server for post-processing and for evaluation. This data is necessary to check if the test run was executed w/o failure (e.g. dropped call analysis) and if the measured figure can be considered for statistic calculations and statements. Moreover, it helps to verify if the reference conditions were met (e.g. RF coverage and capacity) during the whole measurement. Traces deliver additional information for tuning and optimizing the Reference Cluster itself. The capability to log data on any interface at call level for a specific user (e.g. International Mobile Subscriber Identity (IMSI)) or mobile type (e.g. International Mobile Equipment Identity (IMEI)), or service initiated by a user makes it possible to get information which cannot be deduced from Performance Measurements such as the end-user perception of Quality of Service (QoS) during a call (e.g. requested QoS vs. provided QoS), and correlation between protocol messages and RF measurements. Moreover, as opposed to OAM Performance Measurements, which provide values aggregated on an observation period (i.e. interval statistics), Subscriber and UE traces give instantaneous values for a specific event (e.g. call, location update, etc.).

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a33b8

31

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

2.4.3.2

PM Counters
Performance measurements are mandatory in all field test scenarios. This provides performance related information similar to operational environments. The PM counters of special interest are given with the KPI description (see Overview). This level of detail is provided in accordance with the availability of PM counters in the latest LTE product release and in 3GPP specifications.

32

Id:0900d805807a33b8

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

2.5

Measurement Reporting
Some measurement scenario descriptions already contain hints about the form of the expected measurement reports (e.g. in Peak User Data Rate). A complete description will be provided with the field verification test plan.

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a33ba

33

Definitions of Key Performance Indicators (KPIs)

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

3 Definitions of Key Performance Indicators (KPIs)


3.1 Overview
Key Performance Indicators (KPIs) are defined for application and for LTE network services (see Figure 1 LTE Bearer Service Architecture): a) b) c) d) e) Application Services LTE E2E Network Service Radio Bearer Services: E-RAB, RB Transport Bearer Services: X2, S1, S5/S8 and External LTE Network Elements Web Browsing Data Upload / Download Audio Video Streaming Conversational Voice (VoIP)

The Application KPIs are service specific. The following services are considered:

The current paper focuses on the application service Data Download/ Upload (FTP) Service KPIs", and on LTE E2E Network Service" KPIs in line with the feature scope of the LTE release. The performance indicators of the categories D and E are useful for the decomposition of E2E performance figures. Their output can help to locate potential performance bottlenecks in the field, and to optimize the overall performance of the LTE network. E.g. the KPI values of category D can be the basis for Service Level Objectives (SLOs) for IP backhaul networks if these are operated by third party ISPs. Because SLOs are very important, maximum acceptable values are specified for them as assumptions for the achievement of performance targets for application and LTE network service KPIs. KPIs are distinguished according to the measurement setup, which should be used for their verification. They are denoted "Lab" KPIs, if field network trials are not recommended with them due to high costs or technical limitations. These KPIs are measured in vendor or operator laboratories only. The other category of KPIs is called "Field" KPIs, because they should be measured in Reference Network Clusters of operational networks. However, also "Field" KPIs have to be verified in the laboratory before measurements are executed in the field. Lab results serve as benchmark figures and will be used to calibrate (scale) field network measurement results. After the overview of all KPI categories, detailed descriptions are given in Application Services and LTE Network Services.

34

Id:0900d805807a32b6

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

Definitions of Key Performance Indicators (KPIs)

3.1.1

Application Services
The full set of application service KPIs is summarized in Table 11 Overview of Application Service KPIs. The following IP based Packet Switched (PS) data services are characterized with a common set of KPIs: Data Download / Upload (FTP) Web browsing (HTTP) Email (POP/SMTP)

The Data Download / Upload service is selected for detailed investigations and verification in the LTE network, because of the Transmission Control Protocol (TCP) windowing mechanism. # KPI Name PS Data Services (FTP, HTTP, etc.) 1 2 3 4 5 Service Accessibility Ratio [%] Completed Session Ratio [%] Service Access Time [s] Session Time [s] User Data Rate [Mbps] Accessibility Field Reliability Latency Latency Throughput Field Field Field Field (FTP) Service Accessibility Ratio (FTP) Completed Session Ratio (FTP) Service Access Time (FTP) Session Time (FTP) User Data Rate KPI Category Trial Network Chapter

Table 11

Overview of Application Service KPIs

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a32b6

35

Definitions of Key Performance Indicators (KPIs)

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

3.1.2 3.1.2.1

LTE Network Services LTE E2E Network Service


The full set of LTE E2E network service KPIs is summarized in Table 12 Overview of LTE E2E Network Service KPIs. Some of the KPIs are defined as "Lab", which means that they are verified in in-house (or vendor) labs, but not in the field.

KPI Name LTE E2E Network Service, Control Plane

KPI Category

Trial Network

Chapter

6 7 8 9

Attach Time [ms] Detach Time [ms] Attach Success Rate [%] Service Request (EPS) Time [ms]

Latency Latency

Field Lab

Attach Time Detach Time Attach Success Rate Service Request Time (EPS), UE Initiated, Service Request (EPS) Time, Network Initiated Service Request (EPS) Success Rate Service (EPS Bearer) Drop Rate Handover Procedure Time Handover Success Rate Paging Time Paging Failure Rate

Accessibility Field Latency Field

10 Service Request (EPS) Success Rate [%] 11 Service (EPS) Drop Rate [%] 12 Handover Procedure Time [ms] 13 Handover Success Rate [%] 14 Paging Time [ms] 15 Paging Failure Rate [%] LTE E2E Network Service, User Plane 16 (LTE) Round Trip Time (RTT) [ms] 17 (LTE) User Data Rate [Mbps] 18 (LTE) Packet Loss Rate (PLR) [%] 19 (LTE) Service Interrupt Time (HO) [ms] Table 12

Accessibility Field Reliability Mobility Mobility Latency Field Lab Field Lab

Accessibility Lab

Latency Throughput Reliability Mobility

Field Field Field Lab

(LTE) Round Trip Time (LTE) User Data Rate (LTE) Packet Loss Rate (LTE) Service Interrupt Time (HO)

Overview of LTE E2E Network Service KPIs

36

Id:0900d805807a32b6

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

Definitions of Key Performance Indicators (KPIs)

3.1.2.2
#

Radio Bearer Services


KPI Category Trial Network Chapter

KPI Name Radio Bearer Service, User Plane

20 (RB) Packet Loss Rate UL / DL [%] 21 (RB) User Data Rate [Mbps] 22 Cell Throughput [Mbps] Table 13

Reliability Throughput Throughput

Lab Field Field

(RB) Packet Loss Rate (RB) User Data Rate Cell Throughput

Overview of LTE Radio Bearer Service KPIs

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a32b6

37

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

3.2

KPI Definition Template


For each KPI, relevant information is provided according to the structure shown in Table 14 KPI Definition Template. The fields in the KPI definition are marked as mandatory, or optional according to References 12. [3GPP32.410].

KPI Name KPI Type, Category, Object, Unit

M Mandatory long name of the KPI. M Type: Ratio, Mean, CDF, etc. [3GPP32.814]; see References 19. Category: Accessibility, Reliability, Latency, Mobility, etc. Object: Application Service, E2E, E-RAB, RB, X1, etc. Unit: %, ms, Mbps, Erlang, etc. M Description of the KPI. End-user or network view. Basic information about begin and end trigger points. O Test environment: field or lab test, or simulation.Measurement method: according to KPI Category, plus specifics for the KPI (if any). O Important information for measurement execution, i.e. scenario description including terminal category, channel model, load conditions, etc. M Provided for KPIs, e.g. Success Rate & Drop Rate which are calculated from some other metrics (measured) O Provided for latency KPIs, for monitoring of procedure correctness, and for success rate KPIs to indicate trigger points where PM counters are incremented. O Reference to related KPIs O Reference to related domain specific PM counters with indication of how to aggregate them to the KPI.

Definition Methodology of measuring Assumptions and pre-conditions

Formula (Logical) Message flow, trigger points

Related KPIs Related PM counters

Table 14

KPI Definition Template The "KPI Object" field identifies the service, or network element to be characterized: Services: AS or one of the 3GPP bearer services: E2E network (LTE), EPS, E-RAB, RB, X2, S1, S5/S8 and External. The bearer services are uniquely defined by their end-points, i.e. by the network interfaces of the LTE architecture. Network Elements: eNB, MME, S/P-GW.

The following "KPI Categories" are used: Accessibility, Reliability (Retainability), Latency, Throughput, Mobility and Quality. The PM measurements and KPIs of special interest are given with indication of their relationship to the field network KPI, e.g. if and how to calculate the field network KPI value out of them.

38

Id:0900d805807a3c4c

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

3.3
3.3.1

Application Services
Data Download / Upload (FTP) Service KPIs
Download of reference files with different sizes from the Internet, e.g. SW images, pictures, or video, as well as upload of the same files by FTP. For FTP protocol details, see References 33. [RFC979].

3.3.1.1

(FTP) Service Accessibility Ratio

(FTP) Service Accessibility Ratio KPI Category, Object, Type, Unit Definition M Category: Accessibility, Object: AS (UE, FTP server), Type: Ratio, Unit: [%] M The service accessibility ratio denotes the probability that the user can establish the necessary bearer (EPS) and access the FTP service successfully (see References 39. [ETSI102.250-2]). Service access covers starting the FTP client on the UE, setting up mobile access and creating a TCP connection to the FTP server. O Accessibility measurement, see Service Accessibility Measurements. Field and lab trial. O Conditions for accessibility measurements see Service Accessibility Measurements M
number_of(successful_ftp_commands) - 100% FtpCmdSR = -------------------------------------------------------------------------------------------------------number_of(total_ftp_commands)

Measurement method Assumptions, pre-conditions Formula (logical)

Related KPIs Related PM counters Message flow, trigger points

O (FTP) Completed Session Ratio O No PM counters exist for this KPI O Trigger points: Start: ftp get / put command issued on the UE Stop: first data byte sent / received by the UE, or ftp indicates network error (e.g. timeout expired)

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a3c21

39

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

3.3.1.2

(FTP) Completed Session Ratio

(FTP) Completed Session Ratio KPI Category, Object, Type, Unit Definition M Category: Availability, Object: AS (UE, FTP server), Type: Ratio, Unit: [%] M The completed session ratio is the proportion of completed FTP sessions and sessions that were started successfully (see References 39. [ETSI102.250-2]). The session is not completed if a predefined timer expires, or if a network failure has occurred that could not be repaired by automatic restarts. O Availability measurement, see Service Accessibility Measurements. Field and lab trial. O Conditions for availability measurements see Service Accessibility Measurements. The size of the file used for data upload / download, or the session duration time is to be specified for a concrete requirement on the target value of this KPI. M
number_of(completed_sessions) - 100% FtpSessionSR = -----------------------------------------------------------------------------------------------------------------number_of(successfully_started_sessions)

Measurement method Assumptions, pre-conditions

Formula (logical)

Related KPIs Related PM counters Message flow, trigger points

O (FTP) Service Accessibility Ratio O No PM counters exist for this KPI. O Trigger points: Start: ftp get / put command issued on the UE. Stop: after error free execution of the command, ftp returns with a prompt and indicates the number of bytes transmitted.

40

Id:0900d805807a3c21

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

3.3.1.3

(FTP) Session Time

(FTP) Completed Session Ratio KPI Category, Object, Type, Unit Definition M Category: Latency, Object: AS (UE, FTP server), Type: Mean and max (95%) values, Unit: [s] M It is the overall duration of the download or upload of reference files from / to the FTP server (ref. to [ETSI102.250-2]). The elapsed time is measured between start and end trigger points. O Latency measurement, see Latency Measurements O Conditions for latency measurements see Reference Conditions for Latency Measurements M
FtpSessionTime [s] = t sessionend t sessionstart

Measurement method Assumptions, pre-conditions Formula (logical)

Related KPIs Related PM counters Message flow, trigger points

O (FTP) Service Access Time, Service Request (EPS) Time O No PM counters exist for this KPI. O Trigger points UI: Start: ftp client started on the UI. Stop: after error free execution of the "ftp get / put" command, receiving prompt with the number of bytes transmitted. Start: First [SYN] sent to setup a TCP connection. Stop: Reception of the last data packet containing content. Start: First [SYN] sent to setup a TCP connection. Stop: Receive [FIN, ACK] to last data packet sent with file content.

Trigger points MT / R if (Download):

Trigger points MT / R if (Upload):

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a3c21

41

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

3.3.1.4

(FTP) Service Access Time


Figure 4 KPIs for Packet Switched Data Services shows the phases of a data download / upload (FTP) service session using GPRS access network as example (see References 39. [ETSI102.250-2]). In case of an LTE network, GPRS would be replaced by LTE and an EPS bearer would correspond to the "PDP-Context". The full data download / upload session is subdivided into the phases Service Access and Data Transfer. In this document, the Service Access Time is defined as independent KPI. It corresponds to the phase "IP Service Access" on the Figure 4 KPIs for Packet Switched Data Services. (The EPS bearer setup time is defined as LTE E2E network KPI in Service Request Time (EPS), UE Initiated.) The trigger points for the Service Access Time KPI are defined on two levels: end-user interface (UI), i.e. on command line level, and TE - MT interface, i.e. on UE R interface level.

On the command line level, the end of the mobile network access phase is recognized by the ftp> prompt. By this time, the FTP client is initialized, the terminal is attached to the LTE network and the EPS bearer has been created. The Session is started by issuing the "ftp get / put" command and terminates with the display of the results of the data transfer (e.g. the # of bytes transmitted.). After the "get / put" command has been be issued, a TCP connection to the FTP server is created and the data transfer phase begins. Trigger events of the Service Access Time KPI cannot be seen on the terminal UI. In order to find them, it is necessary to analyze the message flows on the R interface level. The Session (and the IP Service Access phase, see Figure 4 KPIs for Packet Switched Data Services) begins with setting up the TCP connection to the FTP server and terminates with the transmission (send, or receive) of the last IP packet with file content payload. The phases Service Access and IP Service Access end, and the Data Transfer phase begins with the transmission (send, or receive) of the first IP packet with file content payload.

42

Id:0900d805807a3c21

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

Figure 4

KPIs for Packet Switched Data Services

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a3c21

43

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

(FTP) Service Access Time KPI Category, Object, Type, Unit Definition M Category: Latency, Object: AS (UE, FTP server), Type: Mean and max (95%) values, Unit: [s] M It is the time period needed to access the FTP service successfully, from starting the ftp client to the point of time when the first data packet is sent or received (see References 39. [ETSI102.250-2]). It is assumed that the UE is already attached to the mobile network and an EPS bearer exists. O Latency measurement, see Latency Measurements. O Conditions for latency measurements see Reference Conditions for Latency Measurements. M
FtpServiceAccessTime [s] = t content sent or received t ft p command started

Measurement method Assumptions, pre-conditions Formula (logical) Related KPIs Related PM counters Message flow, trigger points

O (FTP) Session Time, Service Request (EPS) Time O No PM counters exist for this KPI. O Trigger points DL/UL: Start: "ftp get / put" command issued on the UI Stop: first data packet containing file content received / sent on the R interface.

44

Id:0900d805807a3c21

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

3.3.1.5

(FTP) User Data Rate

(FTP) User Data Rate UL / DL KPI Category, Object, Type, Unit Definition M Category: Throughput, Object: AS (UE, FTP server), Type: Min (CDF 5%), Mean (CDF 50%), Max (CDF 95%) values, Unit: [Mbps] M After the connection to the FTP server has been successfully established, the parameter describes the average data transfer rate measured over the data transfer phase (see Figure 4 KPIs for Packet Switched Data Services). This means that successful Service Access (availability of an EPS bearer and connection to the FTP service) is a prerequisite to the data transfer. The data transfer phase should also conclude successfully. The data rate is measured on application level in UL/DL direction. O Throughput measurement, see Peak User Data Rate. Single user measurement both in field and lab trials. In order to eliminate the impact of service startup times, the measurement begins with the transmission of first IP packets with file content payload. Since the data rates can vary strongly from session to session, a sufficiently large number of measurement executions is needed (>10) to get statistical confidence. The size of the large reference file should be selected such that the data transfer phase does not take less then thirty seconds. Optionally, a small reference file can be measured in addition, but the number of executions should be the same. The measured User Data Rates should be displayed in diagrams as a function of time to show the impact of FTP slow start. Conditions for capacity measurements see Peak User Data Rate. The FTP server will be connected directly to the SGi interface. Additional delays or bottlenecks between the S/P-GW and the server should be excluded. The initial status of UE before starting the ftp application: EMM-REGISTERED, ECM-CONNECTED. The FTP application will be executed in binary mode, and the file to be transferred will also be binary.
transferred_data_volume_UL/DL[bytes] 8 FtpMeanDataRateUL/DL = ------------------------------------------------------------------------------------------------------------------transfer_time [s]

Measurement method

Assumptions, pre-conditions

Formula (logical)

Related KPIs Related PM counters Message flow, trigger points

O O O

(FTP) Service Access Time, (FTP) Session Time No PM counters exist for this KPI. The average throughput is measured from the opening of the data connection to the end of the successful transfer of the content (file, e-mail or web page). The trigger points for the Data Transfer phase can be seen in Figure 4 KPIs for Packet Switched Data Services. For more details see References 39. [ETSI102.250-2], chapters 4.6.1.7 and 4.6.2.7.

It is to be noted that the measurement of the User Data Rate (FTP) KPI (named after [ETSI102.250-2]; see References 39.) is executed as a single user measurement without concurrent load in the cell, since the goal is to find the maximum available FTP data rate for one user.

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a3c21

45

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

3.4

LTE Network Services


This chapter includes detailed KPI definitions for LTE Network Services.

3.4.1 3.4.1.1

LTE E2E Network KPIs Attach Time

Attach Time, Control Plane KPI Category, Object, Type, Unit Definition M Category: Latency, Object: End-user, LTE (UE R interface), Type: Mean and max (95%) value, Unit: [ms] M With Attach, the mobile terminal registers at the LTE network. At the end of the procedure the UE is authenticated, and a default (nGBR) bearer is established. The Attach Time is the interval between the connection request and the acknowledgement of the positive response by the UE. O Latency measurement, see Latency Measurements. Field and lab trial. Stationary users only. O Conditions for latency measurements, see Reference Conditions for Latency Measurements. UE status before measurement: EMMDEREGISTEREDMAP inter-working according to [3GPP29.002], see References 10., is assumed between MME and HSS (no Diameter). M tbd.: mean value and 95% from all measured samples
Attach Time [ms] = tAt tach Complete tAt tach Request

Measurement method Assumptions, pre-conditions

Formula (logical)

Message flow, trigger points

O Trigger points on UE R interface (see Figure 5 Attach Procedure w/ Initial EPS Bearer Establishment (3GPP 23.401), References 3. [3GPP23.401]): Begin: 1. RRC CONNECTION REQUEST sent by UE (see also References 26. [3GPP36.331]) carrying NAS: Attach Request in its body. End: 21. RRC DIRECT TRANSFER message with NAS: Attach Complete in its body sent by UE (see References 4. [3GPP24.008]).

Related KPIs

O a) Attach Success Rate. b) No 3GPP defined KPI for Attach Time found (exists). c) Attach Setup Time KPI in [ETSI102.250-2], see References39. chapter 4.6.3.2. O Attach Time related measurements are not defined in 3GPP Release 8 documents. However, the same trigger events for counting Attach requests and successful attachment completions can be used as Begin and End triggers for the measurement of elapsed time on the eNB.

Related PM counters

46

Id:0900d805807a3c48

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

For initial Attach, the following specifics of the message flow in Figure 5 Attach Procedure w/ Initial EPS Bearer Establishment (3GPP 23.401), have to be considered: A signaling bearer (SRB1) is to be established, before an Attach Request can be sent, i.e. the UE is expected execute a contention based Random Access (RA) procedure. Step 3. is not executed, since the IMSI is queried from the UE directly using Step 4. The optional Identity Request / Response messages in step 5.b do not have to be exchanged if the MEI is already returned by the UE using the Security Mode Complete message as part of step 5.a. Steps 7-11 are not executed, no EPS bearers are to be deleted.

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a3c48

47

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

Figure 5

Attach Procedure w/ Initial EPS Bearer Establishment (3GPP 23.401)

48

Id:0900d805807a3c48

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

3.4.1.2

Detach Time

Detach Time, Control Plane KPI Category, Object, Type, Unit Definition M Category: Latency, Object: LTE (UE R interface) Type: Mean and max (95%-ile) values, Unit: [ms] M With an explicit Detach request the UE informs the LTE network that it does not want to access the EPS any longer. At the end of the procedure all EPS bearers of the UE are released. The Detach Time is the interval between the Detach Request and the reception of a Detach Accept message by the UE. No Detach Accept is sent by the network if the cause for Detach is switching the UE off. O Latency measurement, see Latency Measurements Lab trial. Stationary users only.

Measurement method

Assumptions, O Conditions for latency measurements see Reference Conditions for Latency Measurements. pre-conditions UE status before measurement: EMM-REGISTERED. Formula (logical) M
Detach Time [ms] = t D etachRequest t DetachAccept

Message flow, O Trigger points on UE R interface (see Figure 6 UE-Initiated Detach Procedure, trigger points [3GPP23.401]): Related KPIs Related PM counters Begin: 1. RRC UL Information Transfer sent by UE carrying NAS: Detach Request in its body. End: 11. RRC DL Information Transfer message with NAS: Detach Accept in its body received by the UE.

O a) Attach Time b) No 3GPP defined KPI for Detach Time found (exists). O Detach Time related measurements are not defined in 3GPP Release 8 documents.

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a3c48

49

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

Figure 6

UE-Initiated Detach Procedure

50

Id:0900d805807a3c48

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

3.4.1.3

Attach Success Rate

Attach Success Rate, Control Planee KPI Category, Object, Type, Unit Definition M Category: Accessibility, Object: LTE (UE R, eNB Uu interface), Type: Ratio, Unit: [%] M The Attach Success Rate is defined as the ratio between the number of successful registrations and the number of all requests. This is the probability that a user can attach to the LTE network at any moment of time. The calculated success ratio figure excludes attach requests, which are rejected by authentication failures. On the other hand, network attach requests which are terminated by timer expiry (due to the unavailability of some LTE resource) are considered as unsuccessful registrations. If the success rate is calculated on the eNB by counting incoming RRC requests, RRC CONNECTION REQUEST retries are to be excluded, since they would increase the overall number of establishment attempts, and thus reduce the success ratio. O Accessibility measurement, see . Field and lab trial. Stationary users only. For defining the success ratio, the same series of "Attach Time" measurements can be used. The KPI cannot be measured as end-user perceived ratio on certain mobiles, which do not indicate network attachment status to the end-user.

Measurement method

Assumptions, O Conditions for accessibility measurements see Service Accessibility Measurements. pre-conditions See also conditions inAttach Time. Formula (logical) M
number_of_successful_attachments - 100% LTENwAttSR = ------------------------------------------------------------------------------------------------number_of_all_attempts

Message flow, O Trigger events on UE R / eNB Uu interface (see Figure 5 Attach Procedure w/ Initial EPS trigger points Bearer Establishment (3GPP 23.401)): Related KPIs Success: 21. RRC DIRECT TRANSFER message with L3 NAS: Attach Complete in its body (see References 4. [3GPP24.008]) sent by the UE / received by eNB Attempts: 1. RRC CONNECTION REQUEST messages (with cause Attach Request) sent by UE / received by eNB.

O a) Attach Time, Service Request (ESP) Success Rate. b) E-RAB Accessibility" KPI in [3GPP32.450], see References17. chapter 6.1.1. Since an SRB and S1 connection, as well as initial EPS bearer(s) have to be created during the Attach procedure (see Figure 5 Attach Procedure w/ Initial EPS Bearer Establishment (3GPP 23.401)), its success rate is dependent of the success rates of those component bearers. The success rate of initial EPS bearers is given by the following expression (for additional EPS bearers, refer to the Service Request (EPS) Success Rate):

c) See "Attach Failure Ratio" KPI in References39. [ETSI102.250-2], Chapter 4.6.3.1 defined as the inverse probability of successful network attachment.

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a3c48

51

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

Related PM counters

O a) Related 3GPP measurements are defined in [3GPP32.425],see References15., for eNB: Signaling Radio Bearer (SRB) creation: Attempts: RRC.ConnEstabAtt.Cause = Attach Success: RRC.ConnEstabSucc.Cause = Attach Failures: RRC.ConnEstabFail.Cause = Attach UE associated logical S1 connection (S1-AP): Attempts: S1SIG.ConnEstabAtt Success: S1SIG.ConnEstabSucc Initial EPS setups during Attach: Attempts: SAEB.EstabInitAttNbr.QCI Success: SAEB.EstabInitSuccNbr.QCI where QCI identifies the SAE Bearer level quality of service class. Failures: SAEB.EstabInitFailNbr.Cause where Cause identifies the cause resulting in the setup failure. b) The corresponding eNB counters are: Attempts: SIG_CON_EST_ATT due to Attach Successful attachments: SIG_CON_EST_SUCC Failures: SIG_CON_EST_FAIL For the calculation of success, or failure ratios, it is necessary to have identical Causes for _ATT, _SUCC and _FAIL types of counter groups. Since the EPS_SETUP_ATT/_SUCC counters do not differentiate between initial and subsequent EPS bearer setups, they cannot be set in direct relation to the Attach SR KPI.

52

Id:0900d805807a3c48

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

3.4.1.4

Service Request Time (EPS), UE Initiated

Service Request Time (EPS Bearer Setup), UE Initiated, Control Plane KPI Category, Object, Type, Unit Definition M Category: Latency, Object: EPS (UE R, eNB Uu interface), Type: Mean and max (95%) values, Unit: [ms] M It is the time taken by the LTE network to setup an EPS bearer on request by the UE. The EPS bearer can be new (dedicated), or an existing one (e.g. the default EPS bearer). The latter is needed to reassign Uu radio and S1 bearer resources to the existing EPS bearer of a previously Idle UE. The EPS bearer has to be created or activated before IP packets can be exchanged, i.e. the Service Request creates the IP link dynamically over the mobile access (LTE) network. The UE Initiated EPS Bearer Setup Time is the interval between the submission of the message RRC CONNECTION REQUEST (see References 26. [3GPP36.331]) carrying a NAS: Service Request message in its body, and the reception of the RRC CONNECTION RECONFIGURATION response by the UE. The Bearer Setup procedure is based on the Service Request procedure, see References 3. [3GPP23.401] and 20. [3GPP36.300]. O Latency measurement, see Latency Measurements. Field and lab trial. Stationary users only. Dedicated, or default EPS bearer setup. The response time can be measured on the UE R interface, or on the eNB Uu interface. The latter metric shows the delay component of the E-UTRA network only. End-user delays cannot be measured, since terminals do not indicate the corresponding state change toward the end-user. The setup times of the RB and S1 bearers are part of the EPS setup time and can be determined using the same sequence of measurements. O Conditions for latency measurements see Reference Conditions for Latency Measurements. UE status before measurement: EMM-REGISTERED , but otherwise different initial states (idle, dormant or active). M Mean value and 95% from all measured samples.Service Request
Service Request Time [ms] = t RR C_Reconfig t RR C_Request

Measurement method

Assumptions, pre-conditions

Formula (logical)

Message flow, trigger points

O The message flow diagram is shown in Figure 7 EPS Bearer Setup Procedure, Part 1 and Figure 8 EPS Bearer Setup Procedure, Part 2. The user initiated EPS bearer setup time is measured between the trigger points BUE and EUE: Begin: RRC CONNECTION REQUEST sent by the UE carrying a NAS: Service Request message in its body. End: corresponding RRC CONNECTION RECONFIGURATION COMPLETE message or first PDU in UL sent by the UE (see References 4. [3GPP24.008]).

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a3c48

53

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

Related KPIs

O a) Attach Time, network initiated Service Request (EPS Bearer Setup) Time. b) No 3GPP defined KPI for EPS bearer setup found (exists). O The eNB delay budget of the overall Service Request Time can be measured using two counters of [3GPP32.425], see References 15. Mean EPS Setup Time: SAEB.EstabTimeMean.QCI Max EPS Setup Time: SAEB.EstabTimeMax.QCI

Related PM counters

The latency is measured between the trigger points T1, T2 as can be seen in Figure 8 EPS Bearer Setup Procedure, Part 2.

54

Id:0900d805807a3c48

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

LTE-Uu UE eNodeB

S1- MME MME

S11 S-GW / P-GW D DL data BN

S6a HSS

UE EMM-REGISTERED and ECM-IDLE Paging Procedure RRC CONNECTION REQUEST RRC CONNECTION SETUP RRC CONNECTION SETUP COMPLETE (L3 NAS: SERVICE REQUEST) S1-AP: INITIAL UE MESSAGE (L3 NAS: SERVICE REQUEST) Setup SRB1 L2 SCTP: DL DATA NOTIFICATION

BUE

Begin measurement

L2 SCTP: Create Bearer Request L2 SCTP: Create Bearer Response

Activate EPS bearer setup in ePC

If authentication vectors are not available in MME, authenticate UE S1-AP: UL NAS TRANSPORT RRC: DL INFORMATION TRANSFER (L3: AUTHENTICATION AND CIPHERING REQUEST) RRC: UL INFORMATION TRANSFER (L3: AUTHENTICATION AND CIPHERING RESPONSE) S1-AP: DL NAS TRANSPORT (L3: AUTHENTICATION AND CIPHERING RESPONSE) (L3: AUTHENTICATION AND CIPHERING REQUEST)

MAP: Send Authentication Info MAP: Send Authentication Info Ack

If authentication timer has expired, setup security association for user plane RB

Figure 7

EPS Bearer Setup Procedure, Part 1

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a3c48

55

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

LTE-Uu UE eNodeB

S1- MME MME


S1-AP: INITIAL CONTEXT SETUP REQUEST S1-AP: INITIAL CONTEXT SETUP FAILURE

S11 S-GW / P-GW

S6a HSS

T1

PM Counter: EPS_SETUP_ATT

RRC SECURITY MODE COMMAND RRC CONNECTION RECONFIGURATION

Setup user plane RB

RRC SECURITY MODE COMMAND COMPLETE


EUE

RRC CONNECTION RECONFIGURATION COMPLETE

End measurement

First UL PDU S1-AP: INITIAL CONTEXT SETUP COMPLETE L2 SCTP: Update Bearer Request L2 SCTP: Update Bearer Response First DL PDU
EN

T2

PM Counter: EPS_SETUP_SUCC

Means Delay DL Packet Notification Request to stop/ enable DL data transfer

Figure 8

EPS Bearer Setup Procedure, Part 2

The service request times for UE and for network initiated EPS bearer setups are measured between the trigger points BUE and EUE, and BN and EN, respectively. These trigger points are colored green in the diagram. The trigger points T1 and T2 denote events, which increment the PM counters EPS_BEARER_ATT and EPS_BEARER_SUCC on the eNB. They are colored yellow. These counters can be used on the PM server to calculate EPS bearer setup success rates as seen by the eNB. Please note that this value will be different from that of the Service Request Success Rate KPI defined in this document (see Service Request (EPS) Success Rate). The eNB counters calculate the RB (and not the EPS) setup rate.

56

Id:0900d805807a3c48

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

3.4.1.5

Service Request (EPS) Time, Network Initiated

Service Request (EPS Bearer Setup) Time, Network Initiated, Control Plane KPI Category, Object, Type, Unit Definition M Category: Latency, Object: EPS (S-GW S11 interface), Type: Mean and max (95%) values, Unit: [ms] M It is the time taken by the LTE network to set up an EPS bearer on request by the P-GW. The EPS bearer has to be created before IP packets can be sent (DL) to the UE if the UE has no proper EPS bearer for the given IP packet flow. The network initiated Service Request Time includes a Paging Time (ref. to Paging Time) if the UE is idle. The EPS Bearer Setup procedure is based on the Paging and Service Request procedures according to References 26. [3GPP36.331] and 3. [3GPP23.401]. O Latency measurement, see Latency Measurements. Lab trial and stationary users only. Default EPS bearer setup. The setup times of ERAB and RB bearers, as well as the Paging time are part of the EPS setup time and can be calculated from the same sequence of measurements by taking appropriate traces. O Conditions for latency measurements see Reference Conditions for Latency Measurements. UE status before measurement: EMM-REGISTERED, but otherwise different initial states (idle, dormant or active). M Mean value and 95% from all measured samples.Service Request
Service Request Time [s] = t RR C_Reconfig t RR C_Request

Measurement method

Assumptions, pre-conditions

Formula (logical)

Message flow, trigger points

O The message flow of the Service Request procedure is shown in Figure 7 EPS Bearer Setup Procedure, Part 1 and Figure 8 EPS Bearer Setup Procedure, Part 2. The network initiated service request time is measured between the trigger points BN and EN on the S-GW S11 interface: Begin: L2 SCTP: DL DATA NOTIFICATION sent to MME End: L2 SCTP: Update Bearer Response sent to MME

If the service request is triggered by the arrival of a DL packet, the interrupt time of the packet stream is measured on the S-GW as the latency between: Related KPIs Related PM counters Begin: arrival of the DL packet at the S-GW toward the idle UE, End: forwarding of the same packet to the UE connected.

O Attach Time, Paging Time, E-RAB and RB Setup Times. O The eNB delay budget of the overall Service Request (ESP) Time is identical to that of the UE initiated procedures. See related PM counters and measurements there. The network initiated Service Request Time is not described from the UE point of view, because the delay component seen on the UE is not the complete time the downlink data packet is delayed before it can be forwarded from the P-GW to the UE. The network initiated EPS bearer setup time is dependent on the mobility state of the mobile terminal. If the terminal is idle, paging becomes necessary.

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a3c48

57

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

3.4.1.6

Service Request (EPS) Success Rate

Service Request (EPS) Success Rate, Control Plane KPI Category,Object, M Category: Accessibility, Object: EPS (UE R, eNB Uu), Type, Unit Type: Ratio, Unit: [%] Definition M This KPI is defined as the ratio between successfully established EPS bearers compared to the overall number of EPS bearer establishment attempts. It corresponds to the probability that a user or the LTE network can establish an EPS bearer at any moment in time. Requests that are terminated by timer expiry (due to the unaccessibility of some LTE resource) are considered as unsuccessful attempts. Authentication errors (requests rejected by the MME) are included in the total number of failures. Only the first RRC CONNECTION REQUEST is to be considered, since counting retries of the same message would increase the overall number of bearer establishment attempts, and thus reduce the success ratio. O Accessibility measurement, see Service Accessibility Measurements. Field and lab trial. Stationary users only. For calculating the success ratio, the same series of "Service Request (EPS) Time" measurements can be used. O Conditions for accessibility measurements see Service Accessibility Measurements. See additional conditions in Service Request (EPS) Time, Network Initiated. M
number_of(RRC_CONN_RECONFIGURATION_COMPLETE) - 100% EPSSR = ---------------------------------------------------------------------------------------------------------------------------------------------------------------------number_of(RRC_CONNECTION_REQUEST)

Measurement method Assumptions, pre-conditions Formula (logical)

Message flow, trigger points

O The message flow diagram is shown in Figure 7 EPS Bearer Setup Procedure, Part 1 and Figure 8 EPS Bearer Setup Procedure, Part 2. The trigger points for UE initiated EPS bearer setup success ratio on the R interface are: Success: RRC CONNECTION RECONFIGURATION COMPLETE message sent by the UE, see EUE in Figure 8 EPS Bearer Setup Procedure, Part 2. Total: an attempt is made to send an RRC CONNECTION REQUEST with appropriate establishment cause by the UE, see BUE in Figure 7 EPS Bearer Setup Procedure, Part 1. Trigger points on the eNB Uu interface: Success: RRC CONNECTION RECONFIGURATION COMPLETE message received by the eNB. Total: an attempt is made to receive an RRC CONNECTION SETUP COMPLETE MESSAGE by the eNB with L3 NAS: SERVICE REQUEST in its body.

The measurement on the eNB side is an approximation of the success ratio as perceived by the end-user, since it does not include SRB setup failures. The UE side delivers more precise results.The exact calculation of network initiated EPS bearer setup success ratios uses the triggers BN and EN, which are also shown in Figure 7 EPS Bearer Setup Procedure, Part 1 and Figure 8 EPS Bearer Setup Procedure, Part 2.

58

Id:0900d805807a3c48

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

Related KPIs

Service Request (EPS) Time, user and network initiated.The "E-RAB Accessibility" KPI of [3GPP32.450], see References 17. chapter 6.1.1 corresponds well with this KPI, considering explicitly created "additional" EPS bearers:

For initial EPS bearers, see KPI Attach Success Rate. Related PM counters O a) 3GPP defines the following measurements for additional EPS bearers: h. Additional EPS setup attempts: SAEB.NbrAttEstabAdd.QCI i. Successful setups: SAEB.EstabAddSuccNbr.QCI. j. Failures: SAEB.EstabAddFailNbr.Cause For initial EPS bearer setup measurements, see Attach procedure in Attach Success Rate. b) The corresponding eNB counters are: EPS_SETUP_ATT, EPS_SETUP_SUCC, EPS_SETUP_FAIL each counts both initial and additional EPS setups (i.e. no differentiation is made). See also trigger points T1 & T2 in Figure 8 EPS Bearer Setup Procedure, Part 2.

Using online PM measurements on the eNB to count the total number of EPS bearer setup requests (EPS_SETUP_ATT) and the successfully terminated ones (EPS_SETUP_SUCC), an EPS setup success rate can be calculated (see trigger points T1 and T2). This ratio is, however, not identical to the user perceived success rate, since the latter includes additional failure causes, like failures in setting up the SRB signaling bearer and the ePC part of the EPS bearer, as well as UE authentication errors.

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a3c48

59

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

3.4.1.7

Service (EPS Bearer) Drop Rate

Service (EPS Bearer) Drop Rate, Control Plane KPI Category, Object, Type, Unit Definition M Category: Availability, Object: EPS (UE R, eNB Uu), Type: Ratio, Unit: [%] M It is the ratio between abnormally released bearers and the overall number of established EPS bearers. An abnormal release is defined as any EPS bearer termination that was not triggered by the mobile user (from UE side). Thus, it reflects the probability that an established bearer is aborted due to insufficient network resources. Dropping the bearer becomes visible to the end-user if an application service is actively using it. If the application automatically re-establishes the bearer, it remains unnoticed by the user. O Availability measurement, see Service Accessibility Measurements. Field and lab trial. O Conditions for availability measurements see Service Accessibility Measurements.The ratio is calculated on the same series of Service Request (EPS) Time measurements (see Service Request Time (EPS), UE Initiated and Service Request (EPS) Time, Network Initiated). M
number_of(dropped_calls) - 100% EPSBearerD R = ----------------------------------------------------------------------------number_of(successful_calls)

Measurement method Assumptions, pre-conditions

Formula (logical)

Message flow, trigger points

O Trigger points on UE R / eNB Uu interfaces: Success: submission of the message RRC CONNECTION RECONFIGURATION COMPLETE by the UE, see References26. [3GPP36.331], or reception by the eNB. Drop: successful establishments minus terminations by the user, i.e. the UE submitting a L3DETACH REQUEST (see References 4. [3GPP24.008]) carried within the RRC message UL NAS TRANSPORT (see References 26. [3GPP36.331]), or the eNB receiving this message.

Measurements on the eNB side produce equivalent results to measurements on the UE side. The drop ratio calculated this way is valid for both UE initiated EPS bearers. For the calculation of a network initiated EPS bearer drop rate, ePC initiated L3 DETACH REQUEST-s needs to be considered.

60

Id:0900d805807a3c48

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

Related KPIs

O a) Service Request (EPS) Success Rate. b) The "E-RAB Retainability" KPI of [3GPP32.450], see References 17. in chapter 6.2.1, is an abnormal EPS release rate, which is related to the above Drop Rate definition, but also considers the session duration (its unit is [1/s]):

SAEB.RelAct.[QCI]
R2 = QCI ------------------------------------------------------------SAEB.SessionTimeUE

The necessary measurements are defined by 3GPP (see below). Related PM counters O a) The following online PM counters of the eNB can be used to calculate the Service Drop Rate: EPS_SETUP_SUCC, EPS_REL_EPC_INI, EPS_REL_ENB_INI. b) 3GPP defines the following measurements for an abnormal EPS bearer release: SAEB.RelAct.QCI, SAEB.SessionTimeUE and SAEB.SessionTimeQCI.QCI The EPS bearer release rate determined by online PM counters on the eNB is equivalent to the value of the Service (EPS) Drop Rate KPI.

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a3c48

61

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

3.4.1.8

Handover Procedure Time

Handover Procedure Time, Control Plane KPI Category, Object, Type, Unit Definition M Category: Mobility (Latency), Object: LTE (UE R interface), Type: Mean and max (95%) value, Unit: [ms] M It denotes the total time needed for the hand-over procedure as seen by the UE. It begins by receiving a Handover Command from the SeNB and ends by sending the Handover Confirm response to the TeNB by the UE. Its relevance is the discontinuity of the IP packet flow in the user plane that is implied by it (also called service interruption). The value of the HO Procedure Time KPI depends on the hand-over scenario. The following HO scenarios are distinguished (though not directly seen by the UE): intra LTE intra- and inter-frequency mobility inter RAT mobility (LTE 2G/3G) intra vs. inter eNB, the latter via X2, or S1 interface intra vs. inter MME/S-GW

Remark: The value of the HO Procedure Time itself does not depend on the direction of an ongoing data transfer (UL, or DL). It is the LTE Service Interrupt Time (HO) which will be different. Measurement method O Mobility (HO) measurement, see Mobility (Handover) Measurements. Lab and field trials, mobile user scenarios only, crossing of cell borders included. The handover time should be measured on the UE R interface. However, measurements on the source eNBs also provide good approximations of the handover time as perceived by the UE. They even have some advantage, since the SeNB sees the full handover procedure, the handover preparation phase included. End-user delays can be measured with test equipment only, since terminals do not indicate cell changes toward the end-user.

Assumptions, pre-conditions O Conditions for handover measurements see Mobility (Handover) Measurements. UE status before measurement: registered and connected. Formula (logical) M Mean value and 95% from all measured samples.
Handover Procedure Time [ms] = t HO_Confirm tHO_Command

Message flow, trigger points O The HO procedure time is seen by the UE as the elapsed time between (see message flow in Figure 9 Handover Procedure: Intra MME/S-GW, Inter eNB via X2): Related KPIs BUE: UE receives Handover Command in the body of an RRC Connection Reconfiguration request from source eNB (SeNB). EUE: UE sends Handover Confirm to the target eNB (TeNB) encoded in an RRC Connection Reconfiguration Complete message.

O a) (LTE) Service Interrupt Time, (RB) Service Interrupt Time, HO Success Rate. b) No 3GPP defined KPI for HO procedure time found. O HO Procedure Time related measurements are not defined in 3GPP Release 8 documents.

Related PM counters

62

Id:0900d805807a3c48

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

Figure 9

Handover Procedure: Intra MME/S-GW, Inter eNB via X2

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a3c48

63

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

Figure 10

Handover Procedure: Intra MME/S-GW, Inter eNB via S1

64

Id:0900d805807a3c48

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

3.4.1.9

Handover Success Rate

Handover Procedure Time, Control Plane KPI Category, Object, Type, Unit Definition Measurement method M Category: Mobility (Accessibility), Object: eNB (X2 interfaces), Type: Ratio, Unit: [%] M The Handover Success Rate is the ratio between successfully executed (committed) HO procedures and the number of all Handover attempts. O Mobility (HO) measurement, see Mobility (Handover) Measurements. Field and lab trial. For calculating the success ratio, the same series of "Service Interrupt Time" measurements can be used. For restrictions in measurement scenarios, refer to the description of the KPI "(LTE) Service Interrupt Time". The measurement is to be executed for different HO scenarios (see description with the KPI Handover Procedure Time). If practical limitations make it difficult carry out a sufficient number of performance tests with terminals, eNB online statistics can be used as approximations (as described below). In case of Inter-RAT scenarios, the eNB statistics cover the LTE leg of the hand-off only. HO failures in the 2G/3G leg have to be collected with corresponding statistics of the BTS/NodeB. O Conditions for mobility measurements see Mobility (Handover) Measurements. See additional conditions at the KPI (LTE) Service Interrupt Time". M
number_of(Handover_Confirm) - 100% HOSR = ------------------------------------------------------------------------------------number_of(Handover_Request)

Assumptions, pre-conditions Formula (logical)

Message flow, trigger points

O The success rate perceived by end-users is approximated with the success rate as seen on the source eNB. Trigger points on the eNB X2 interface: Total #: Handover Request sent by SeNB to TeNB (X2). Success #: UE Context Release message received by SeNB from TeNB.

In case of Intra eNB HO procedures, no Handover Request is sent, both trigger points are to be counted by the (Source) eNB internally. Related KPIs O a) Handover Procedure Time b) E-UTRAN Mobility Success Rate" in [3GPP32.450], see References17. Chapter 6.5.1, is a success rate, which applies to all Intra- and Inter-RAT mobility scenarios:

The KPI is calculated by multiplying the success rates for HO preparation and execution phases. The relationship of variables in this expression and the HO measurements defined in [3GPP32.425],see References 15., is still to be clarified.

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a3c48

65

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

Related PM counters

O a) 3GPP defines measurements / counters on eNB to support HO success ratio calculations (see References 15. [3GPP32.425] chapter 4.3). Examples for selected scenarios: Intra-RAT, intra-eNB handovers: # HO attempts: HO.IntraENBOutAtt.Cause # HO success: HO.IntraENBOutSucc.Cause where Cause identifies the cause for handover Intra-RAT, inter-eNB handovers: # HO attempts: HO.InterENBOutPrepAtt # HO attempts: HO.InterENBOutAtt.Cause # HO success: HO.InterENBOutSucc.Cause Handover measurements on neighbor cell basis: # HO attempts: HO.OutAttTarget.Cause # HO success: HO.OutSuccTarget.Cause Inter-RAT handovers (LTE -> 2G/3G): # HO attempts: HO.IartOutAtt.Cause # HO success: HO.IartOutSucc.Cause b) LTE product related counters on eNB which support the calculation of handover success rates: Intra-RAT, intra-eNB handovers: # HO attempts: INTRA_HO_PREP # HO success: INTRA_HO_SUCC Intra-RAT, iner-eNB handovers: # HO attempts: INTER_HO_PREP # HO success: INTER_HO_SUCC Inter-RAT handovers (LTE -> 2G/3G): # HO attempts: INTER_RAT_HO_PREP # HO success: INTER-RAT_HO_SUCC The calculation of HO Success Rate does not differentiate between different causes of a negative hand-over decision. All rejected HO attempts are considered as unsuccessful handovers.

66

Id:0900d805807a3c48

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

3.4.1.10

Paging Time

Paging Time, Control Plane KPI Category, Object, Type, Unit Definition M Category: Latency, Object: LTE (MME, S-GW; S11, S1-U), Type: Mean and max (95%) value, Unit: [s] M Paging is initiated : By the arrival of a DL packet at the S-GW for an UE whose location is not known, i.e. its location information does not exist in the S-GW, or By network initiated EPS bearer setup requests (ISR flag is set)

It denotes the total procedure time from starting the paging request in DL and terminating it with the subsequent service request (EPS bearer setup) of the UE after it has been located. Thus, the paging time is per definition the difference between network and UE initiated Service Request Times. Trigger events can be defined on the MME and S-GW as initiators of the paging request, either on the S11 (MME - S-GW) or on the S1-U (UE - S-GW) interfaces, see "Trigger points" below. Measurement method O Latency measurement, see Latency Measurements. Lab trial with mobile user scenario only, crossing of cell borders included. The paging time can be measured on the S11 (MME - S-GW), or S1-U (UE - S-GW) interfaces. End-user delays cannot be measured, since paging is always initiated by the network. O Conditions for latency measurements see Reference Conditions for Latency Measurements. UE status before measurement: registered, but idle (not connected). M The paging time is calculated as the difference between network initiated service request and UE initiated service request times.
Paging Time [s] = t SR T network initiated tSR T UE initiated

Assumptions, pre-conditions

Formula (logical)

Message flow, trigger points

O See message flow in Figure 11 Paging Procedure.The paging time can be approximated on the MME S11 with: Start: L2 SCTP: DL Data Notification received for an UE Stop: L3 NAS: SERVICE REQUEST received from the same UE Start: DL packet arrives for an idle UE Stop: DL packet is forwarded to the connected UE

On S-GW S1-U, latency is measured between:

The latter latency is a good metric of DL "service interrupt time" for an IP packet stream caused by paging. The paging time has to be calculated as the measured latency minus the UE initiated Service Request Time (EPS). Related KPIs O a) Service Request Time (EPS), UE and network initiated. b) No 3GPP Release 8 defined KPIs for Paging Time found.

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a3c48

67

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

Related PM counters

O Paging Time related measurements are neither defined in 3GPP Release 8, nor in LTE reference documents.

LTE-Uu UE eNodeB

S1- MME MME

S11 S-GW / P-GW


BN

S6a HSS

DL Packet
D e l a y

UE EMM-REGISTERED and ECM-IDLE

Paging Procedure [3GPP TS 36.413]

Paging Attempt T1
S1-AP: DL NAS TRANSPORT

L2 SCTP: DL Data Notification L2 SCTP: DL Data Notification Ack

RRC: DL INFORMATION TRANSFER (L3 NAS: PAGING) Retry (optionally)

(L3 NAS: PAGING)

To all cells of all TAs

T i m e r

Paging Failure T2

L2 SCTP: DL Data Notification Reject

UE Triggered Service Request Procedure [3GPP TS 23.401]

UE in EMM-REGISTERED and ECM-CONNECTED

L2 SCTP: Stop Paging Forward DL Packet to UE

EN

Figure 11

Paging Procedure

68

Id:0900d805807a3c48

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

3.4.1.11

Paging Failure Rate

Paging Failure Rate, Control Plane KPI Category, Object, Type, Unit Definition M Category: Accessibility, Object: MME (S1-AP, S11), Type: Ratio, Unit: [%] M The Paging Failure Rate is the ratio between unsuccessful paging requests and the number of all paging attempts initiated by the MME. Retries of the same paging request by the MME are not counted as new attempts. Similarly, multicasting the same request to more than one eNBs (in the UEs tracking areas) is considered as one attempt. O Accessibility measurement, see Service Accessibility Measurements. Field and lab trials. For calculating the failure ratio, the same series of "Service Request (EPS) Success Rate" measurements can be used, however, restricted to the network initiated bearer setup scenario. For restrictions in measurement scenarios, see also the KPI Service Request Success Rate". The existence of online performance counters for paging on the MME can help to avoid the execution of performance tests repeatedly. Suitable counters are indicated in the message flow in Figure 11 Paging Procedure (trigger points T1 and T2). O Conditions for accessibility measurements, see Service Accessibility Measurements. See additional conditions at the KPI "Service Request Success Rate". M
number_of(Paging_Failures) - 100% PagingFR = ------------------------------------------------------------------------------number_of(Paging_Attempts)

Measurement method

Assumptions, pre-conditions

Formula (logical)

Message flow, trigger points

O Trigger points on MME (see T1: Attempt and T2: Failure on the Figure 11 Paging Procedure): Total #: different L3 NAS Paging Requests sent to an eNB (S1-AP). Failure #: DL Data Notification Reject messages sent to S-GW (S11)

Related KPIs Related PM counters

O a) Paging Time, Service Request Success Rate - network initiated. b) No Paging related KPI is defined in 3GPP Release 8 documents. O Directly applicable trigger events are T1 and T2 as shown in Figure 11 Paging Procedure. a) 3GPP defined counter (see References15. [3GPP32.425] Chapter 4.7) on eNB is: Discarded Paging Records: PAG.DiscardedNbr b) LTE product related counters defined for eNB in: Requests: RRC_PAGING Responses: SIG_CON_EST_ATT.Cause = PagingResponse Using these counters, the Paging Failure Rate can be calculated as: Paging FR =( 1 - SIG_CON_EST_ATT.Cause = PagingResponse / RRC_PAGING *100%)

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a3c48

69

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

3.4.1.12

(LTE) Round Trip Time

Round Trip Time (RTT), User Plane KPI Category, Object, Type, Unit M Category: Latency, Object: LTE (UE R, P-GW Gi interfaces), Type: min (CDF 5%), mean (CDF 50%), max (CDF 95%) values, Unit: [ms] M RTT in UL is the interval between sending a datagram by the UE & receiving the corresponding reply from an IP peer entity connected to the Gi interface of the P-GW. RTT in DL is the interval between sending a datagram to the UE & receiving the corresponding reply by the IP host (peer entity). O Latency measurement, Latency Measurements. Field and lab trials. Stationary users in different cell positions, which are uniformly distributed across the cell. RTT is measured with the Ping application between the UE and an IP host (peer entity). As average RTT figure, the output of the Ping application is used (e.g. "MS-DOS >ping -n count -l size host_name"). Size of ICMP packets: 32, , MTU, 2x MTU. Ping should be executed at least 100 times to reduce the impact of the first ICMP message, which triggers the setup of radio bearer establishments in UL and DL directions. O Conditions for latency measurements see Reference Conditions for Latency Measurements. UE status before measurement: registered and connected. The IP peer entity should be located as close as possible to the SGi interface. Additional delays between the P-GW and the IP host should be avoided or minimized. M Mean value and 95% from all measured samples.
Round Trip Time [ms] = t ICMP Echo Reply t IC MP Echo Request

Definition

Measurement method

Assumptions, pre-conditions

Formula (logical)

Message flow, trigger points

O Trigger points both from UE, and from IP peer entity view: Start: ICMP ECHO REQUEST Stop: ICMP ECHO REPLY

Related KPIs

O a) Network Delay UL / DL KPI for LTE, RB and IP based transport bearers X2, S1, S5/S8) b) No 3GPP defined KPI for LTE RTT found. c) See "{Service} Round Trip Time" KPI definition for Ping service in [ETSI102.250-2], see References 39. Chapter 4.6.3.6. O Measurements of round trip times are neither defined in 3GPP nor in LTE reference documents.

Related PM counters

70

Id:0900d805807a3c48

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

3.4.1.13

(LTE) User Data Rate

User Data Rate UL / DL, User Plane KPI Category, Object, Type, Unit Definition M Category: Throughput, Object: LTE (UE R if, IP peer entity) Type: Min (CDF 5%), Mean (CDF 50%) Max (CDF 95%) values, Unit: [Mbps] M The metric describes the data speed available to one user of the LTE network on UDP/IP level. It is given as the maximum (95%-ile) value that can be observed over a short period of time (e.g. of 1s) and as a mean value that characterizes longer data transfer periods (minutes). Its value distribution over the radio cell is given as a function of the SINR. The maximum value is often referred to in the literature as instantaneous "Peak Throughput" that is achieved in optimal radio conditions. The user data rate can be given for a single user active in the cell (single user data rate), or to one of several concurrently active users. O Throughput measurement, see Throughput Measurements. Stationary and mobile user. Cell positions: best, medium, cell edge, including different HO scenarios. The data rate is measured in UL/DL direction with UDP/IP traffic over a time period of several minutes. Signaling delays, e.g. initial radio bearer setup has no impact on the result, because transient times at the beginning and end of the data transfer are not considered. Conditions for throughput measurements see Peak User Data Rate. The IP host used as traffic generator (e.g. with Iperf) should be connected directly to the SGi interface. Additional delays and bandwidth restrictions between the S/P-GW and the server need to be excluded.
transferred_data_volume [bytes] 8 UserDataRate = -------------------------------------------------------------------------------------------------transfer_time [s] 1000

Measurement method

Assumptions, pre-conditions

Formula (logical)

Message flow, trigger points

Trigger points on the UE R interface: UL: UE sending UDP/IP packets to the IP peer entity. DL: IP peer entity sending UDP/IP packets to the UE.

Related KPIs Related PM counters

O O

(RB) User Data Rate, (FTP) User Data Rate. Measurements of user data rate on LTE level (R <-> Gi) are neither defined in 3GPP, nor LTE reference documents.

The User Data Rate KPI has been defined on E2E network level to verify if the capacity of large non-GBR EPS bearers (QoS profile with large maximum bit rates) can in fact be exploited by IP based services. The same holds for the E-RAB and RB bearers being part of the EPS. Especially, the maximum achievable data rate of the RB bearer is interesting, since it is the critical resource of the LTE network. For this reason, the Peak Data Rate is defined as a separate KPI on the RB level (see (RB) Packet Loss Rate), too. Because of this relationship (containment) between EPS, E-RAB and RB, user data rate KPIs can be verified with the same series of measurements. The KPIs are interesting as values calculated over the full duration of long data transfers (~minutes). Of equal interest are peak values observed for a short time period only (~seconds). Please note that data rate KPIs have also been defined for application services, e.g. User Data Rate KPI of the Data Download / Upload service, but these do not characterize the capability of the LTE network alone, but also depend on the application service and its protocols.

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a3c48

71

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

3.4.1.14

(LTE) Packet Loss Rate

(LTE) Packet Loss Rate UL / DL, User Plane KPI Category, Object, Type, Unit M Category: Availability, Object: LTE (UE R interface, IP peer entity) Type: Ratio, Min (CDF 5%), Mean (CDF 50%) Max (CDF 95%) values Unit: [%] M This is the ratio between the numbers of lost or corrupted IP packets, and of all IP packets sent. Corrupted IP packets are those that contain bit errors in their headers or in their payload.Packets with "residual", i.e. undetected errors are not counted as lost. O Availability measurement, see Service Accessibility Measurements. The EPS bearer is to be configured such that corrupted IP packets are not delivered to the application. Measured in loaded and unloaded network, under different radio conditions. Stationary and mobile users. The packet loss rate should be measured in test scenarios, where handovers occur. The number of handovers should be oriented toward the NSN reference traffic model, or, in operator trials, toward the operators traffic model. O Conditions for availability measurements, see Service Accessibility Measurements. Measured in loaded and unloaded network, under different radio conditions. Stationary and mobile user including HO scenarios. The ratio can be calculated on a series of User Data Rate (see Peak User Data Rate) measurements. M
number_of(lost_corrupted_packets) PLR = ------------------------------------------------------------------------------------------------ 100% number_of(all_packet_sent)

Definition

Measurement method

Assumptions, pre-conditions

Formula (logical)

Message flow, trigger points

O UL / DL: Total: packages sent at UE R if / IP peer entity Lost: Total minus packets received at IP peer entity / UE R if.

Related KPIs Related PM counters

O a) (RB) Initial Radio Block Error Rate (BLER) b) No PLR KPI (LTE level) is defined in 3GPP and ETSI documents. O Measurements of packet loss rate on LTE level (R <-> Gi) are neither defined in 3GPP nor in LTE reference documents.

72

Id:0900d805807a3c48

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

3.4.1.15

(LTE) Service Interrupt Time (HO)

(LTE) Service Interrupt Time (HO), User Plane KPI Category, Object, Type, Unit Definition M Category: Latency, Object: LTE (UE R if) Type: Mean and max (95%-ile) value, Unit: [ms] M The Service Interrupt Time is the interval between the last sent/received IP packet of a continuous UL/DL data stream in the old cell and the first sent/received user IP packet in the new cell measured on the UE (also called "user plane break"). The value of the KPI depends of the handover scenario. See HO scenarios at the KPI Handover Procedure Time. O Mobility (HO) measurement, see Mobility (Handover) Measurements. Only mobile user scenarios including cell border crossing. The test application is an FTP upload/download service, or an isochronous UP/IP flow (iperf) maintained during handover. O Conditions for availability measurements, see Mobility (Handover) Measurements. In inter eNB "HO via X2" scenariosthe X2 interface shall be enabled between the source and target eNBs. M
Service Interrupt Time [ms] = t first packet to/from TeNB tlast packet to/from SeNB

Measurement method

Assumptions, pre-conditions

Formula (logical)

Mean value and 95% from all measured samples. Message flow, trigger points O The trigger points for service interrupt time (from UE point of view): Related KPIs Related PM counters BUE: UE sends/receives last packet to/from source eNB EUE: UE sends/receives first packet to/from target eNB.

See Figure 9 Handover Procedure: Intra MME/S-GW, Inter eNB via X2 O a) HO Procedure Time, Voice Interrupt Time (HO) b) No related KPI definitions found in 3GPP Rel. 8 documents. O Measurements of service interrupt times caused by HO are neither defined in 3GPP nor in LTE reference documents.

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a3c48

73

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

3.4.2 3.4.2.1

Radio Bearer KPIs (RB) Packet Loss Rate

(RB) Packet Loss Rate UL / DL, User Plane KPI Category, Object, Type, Unit M Category: Availability, Object: RB (UE R interface, eNB) Type: Ratio, Min (CDF 5%), Mean (CDF 50%) Max (CDF 95%) values Unit: [%] M This is the ratio between the numbers of lost or corrupted IP packets, and of all IP packets sent. Corrupted IP packets are those that contain bit errors in their headers or in their payload. Packets with "residual", i.e. undetected errors are not counted as lost. O The KPI is not measured on its own. Traces are to be taken on the UE and eNB during (LTE) Packet Loss Rate measurements, see (RB) Packet Loss Rate. Lab trials with results for internal use. O See comment above. M
number_of(lost_corrupted_packets) RBPLR = ------------------------------------------------------------------------------------------------ 100% number_of(all_packet_sent)

Definition

Measurement method

Assumptions, pre-conditions Formula (logical)

Message flow, trigger points

O UL / DL: Total #: packages sent on UE R / eNB Lost #: Total minus packets received on eNB / UE R

Measured at the upper SAPs of the L2 PDCP. Related KPIs Related PM counters O a) Initial Radio Block Error Rate (BLER) b) No PLR KPI (RB level) is defined in 3GPP and ETSI documents. O a) The following measurements are defined in LTE for the number of discarded (dropped) PDUs on the eNB in DL: Discarded PDCP SDUs: PDCP_SDU_DISCARD Discarded RLC blocks: SDU_DISCARD_DL_DTCH b) The following measurements are defined by 3GPP in [3GPP32.425], see References 15. chapters 4.4.3-4, for packet (PDCP SDU) loss and drop rates: Drop rate on eNB in DL: DRB.PdcpSduDropRateDl.QCI Loss rate on Uu in DL: DRB.PdcpSduAirLossRateDl.QCI Loss rate on Uu in UL: DRB.PdcpSduLossRateUl.QCI A packet drop rate on eNB in UL is not defined.The loss rate values may refer to the L2 protocols PDCP, RLC or MAC (see [3GPP32.314] chapter 4.1.5.1-3)

74

Id:0900d805807a3c48

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

3.4.2.2

(RB) User Data Rate

(RB) User Data Rate UL / DL, User Plane KPI Category, Object, Type, Unit M Category: Throughput, Object: RB, eNB, Type: Min (CDF 5%), Mean (CDF 50%), Max (CDF 95%) values, Unit: [Mbps] M The metric describes the UDP/IP data rate achievable by one user. It can be given as single user data rate if only one user is active in the cell, or as multi-user data rate for a given number of concurrently active users. The user data rate distribution over the cell is given as function (CDF) of the radio conditions characterized by the SINR. The time variation of the KPI value in a given cell position is given with its maximum (95%) (also referred to as instantaneous "Peak User Data Rate"), and with its average (mean) value. The Peak User Data Rate corresponds to the Cell Throughput KPI on UDP/IP level, when the number of active users in the cell is n=1 and best radio conditions are observed. The User Data Rate is also called "User Throughput" in the literature. O Throughput measurement, see Throughput Measurements. Stationary and mobile user. Cell positions: best, medium, cell edge, including different HO scenarios. Field and lab trials. The eNB assigns all resources (radio blocks) to this user, except for the radio blocks used for signaling and controlling. Measured on UDP/IP level for all UE categories (1-5). With the help of trace analysis, PDCP, RLC, MAC and PHY layer throughputs can also be defined. With and w/o concurrency in UL / DL traffic. The mean user data rate is calculated over a few minutes, peak data rate values are averaged over short periods of time (e.g. 1s). Assumptions, pre-conditions O Conditions for capacity measurements, see Peak User Data Rate. All transport bearers between the UE and IP peer entity should have higher capacity than the radio link in order to avoid bandwidth bottlenecks. M
transferred_data_volume [bytes] 8 RBUserDataRate = ------------------------------------------------------------------------------------------------6 transfer_time [s] 10

Definition

Measurement method

Formula (logical)

Message flow, trigger points

O Trigger events on the eNB: UL: eNB receiving PDUs DL: eNB sending PDUs

Measured at the upper SAPs of the protocols PDCP, RLC, MAC or PHY. Related KPIs Related PM counters O a) (LTE) User Data Rate, Cell Throughput. b) See 3GPP defined KPIs in Cell Throughput. O See 3GPP defined measurements with the KPI in Cell Throughput.

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a3c48

75

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

3.4.2.3

Cell Throughput

Cell Throughput UL / DL, User Plane KPI Category, Object, Type, Unit Definition M Category: Throughput, Object: UE, RB Uu, Type: Min (CDF 5%), Mean (CDF 50%), Max (CDF 95%) values, Unit: [Mbps] M The metric shows the sustainable aggregate throughput of the cell (in UL/DL) available to "n" stationary users distributed uniformly in the cell and running a typical mix of applications. The "cell throughput" is the sum of all bits transported in all radio blocks carrying PDUs (i.e. bits in UL-SCH / DL-SCH transport blocks) during one second. The cell capacity is also given as peak value (called peak cell capacity, or throughput), which is defined as the aggregate throughput of "n" users all located in best radio conditions. The cell throughput value is defined here on PHY level, but could be given for other protocol levels (UDP/IP, PDCP, RLC, MAC), too. When the (peak, average) cell throughput is expressed on UDP/IP level, it corresponds to the (peak, mean) user data rate value at comparable radio conditions. O Throughput measurement, Throughput Measurements. Stationary users uniformly distributed in the cell. UE categories according to application mix. Peak value measured with one user in LoS cell position using UDP/IP load to approach the full buffer condition. Field trial without target value (only to learn the available capacity). Measured with UDP/IP traffic on PHY layer (UL-SCH / DL-SCH transport block bits) over several minutes. With the help of trace analysis PDCP, RLC and MAC layer values can also be given. Conditions for throughput measurements see Attach Success Rate. All IP transport bearers (S1, S5/S8, SGi) between the UE and IP peer entity should have higher bandwidth than the cell capacity to avoid bandwidth bottlenecks.
transferred_data_volume [bytes] 8 CellThroughput = ------------------------------------------------------------------------------------------------6 transfer_time [s] 10

Measurement method

Assumptions, pre-conditions O

Formula (logical)

Message flow, trigger points O

Trigger points on the eNB Uu interface: UL: eNB receiving PDUs / RLC blocks / transport data blocks DL: eNB sending PDUs / RLC blocks / transport data blocks

Related KPIs

a) (RB) User Data Rate b) 3GPP defines the KPI "E-UTRAN IP Throughput" in [3GPP32.450], see References 17. chapter 6.3.1.

76

Id:0900d805807a3c48

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

Related PM counters

a) The following measurements are available on eNB in LTE reference document, which are related to the Cell Throughput KPI: CELL_THROUGHPUT_PDCP (separate counters for min, mean and peak values, both for UL and DL on Uu interface) PDCP_SDU (same for one cell) b) The following measurements are defined by 3GPP in [3GPP32.425] Chapters 4.4.1 for cell data (PDCP SDU) rates: Average data rate in DL: DRB.PdcpSduBitrateDl.QCI Maximal data rate in DL: DRB.PdcpSduBitrateDlMax Average data rate in UL: DRB.PdcpSduBitrateUl.QCI Maximal data rate in UL: DRB.PdcpSduBitrateUlMax The data rate values are measured on the eNB at the upper SAP of L2 PDCP and refer to SDU bits. SDUs that are forwarded over X2/S1 to another eNB are subtracted from the overall bit count.

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a3c48

77

References

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

4 References
1. [3GPP21.905] 3GPP TR 21.905; v 8.x.x; Vocabulary for 3GPP Specification 2. [3GPP23.203] 3GPP TS 23.203; V8.7.0 (2009-09); Policy and charging control architecture 3. [3GPP23.401] 3GPP TR 23.401; v.8.x.x; General Packet Radio Service (GPRS) enhancements for Long Term Evolution (LTE) access 4. [3GPP24.008] 3GPP TR 24.008; V8.5.0 (2009-03); Mobile Radio Interface Layer 3 Specification; Core network protocols; Stage 3 5. [3GPP24.228] 3GPP TR 24.228; V5.15.0 (2006-10); Signaling flows for the IP multimedia call control based on Session Initiation Protocol (SIP) and Session Description Protocol (SDP);Stage 3 (Release 5) 6. [3GPP24.229] 3GPP TR 24.229; V8.8.0 (2009-06);IP multimedia call control protocol based on Session Initiation Protocol (SIP) and Session Description Protocol (SDP);Stage 3 (Release 8) 7. [3GPP25.913] 3GPP TR 25.913; V8.0.0 (2008-12); Requirements for Evolved UTRA (E-UTRA) and Evolved UTRAN (E-UTRAN) 8. [3GPP25.943] 3GPP TR 25.943; V8.0.0 (2008-12); Deployment Aspects (Release 8) 9. [3GPP26.114] 3GPP TS 26.114 V8.2.1 (2009-03); IP Multimedia Subsystem (IMS); Multimedia Telephony; Media handling and interaction (Release 8) 10. [3GPP29.002] 3GPP TS 29.002, V8.x.x; Mobile Application Part (MAP) 11. [3GPP29.060] 3GPP TS 29.060; v 8.x.x; General Packet Radio Service (GPRS); GPRS Tunneling Protocol (GTP) across the Gn and Gp interface 12. [3GPP32.410] 3GPP TS 32.410; V8.0.0 (2009-03); Telecommunication management; Key Performance Indicators for UMTS and GSM (Release 8) 13. [3GPP32.421] 3GPP TS 32.421, V8.4.0 (2008-12); Telecommunication management; Subscriber and equipment trace; Trace concepts and requirements 14. [3GPP32.423] 3GPP TS 32.423, V8.0.0 (2009-03); Telecommunication management; Subscriber and equipment trace, Trace data definition & management 15. [3GPP32.425] 3GPP TS 32.425, V8.1.0 (2009-06); Telecommunication management; PM Performance Measurements, E-UTRAN (Release 8) 16. [3GPP32.426] 3GPP TS 32.426, V8.0.0 (2009-03); Telecommunication management; Performance Management (PM); Performance measurements, Evolved Packet Core network (EPC) 17. [3GPP32.450] 3GPP TS 32.450, V8.1.0 (2009-06); Telecommunication management; Key Performance Indicators for E-UTRAN: Definitions (Release 8) 18. [3GPP32.451] 3GPP TS 32.451, V8.0.0 (2009-03); Telecommunication management; Key Performance Indicators for E-UTRAN: Requirements (Release 8) 19. [3GPP32.814] 3GPP TS 32.814, V7.0.0 (2007-03); Telecommunication management; UTRAN and GERAN Key Performance Indicators (KPI), (Release 7) 20. [3GPP36.300] TS 36.300; v 8.x.x; Evolved Universal Terrestrial Radio Access (EUTRA) and Evolved Universal Terrestrial Radio Access Network (E-UTRAN); Overall description; Stage 2 21. [3GPP36.306] TS 36.306; V8.3.0 (2009-03); Evolved Universal Terrestrial Radio Access (E-UTRA); User Equipment (UE) Radio Access Capabilities 22. [3GPP36.211] TS 36.211; V8.7.0 (2009-05); Evolved Universal Terrestrial Radio Access (E-UTRA); Physical Channels and Modulation

78

Id:0900d805807a32b8

DN0972412 Issue 01 DRAFT APPROVED

LTE E2E Field Network Performance - Definitions of Key Performance Indicators

References

23. [3GPP36.212] TS 36.212; V8.7.0 (2009-05); Evolved Universal Terrestrial Radio Access (E-UTRA); Multiplexing and channel coding 24. [3GPP36.214] TS 36.214; V8.6.0 (2009-03); Evolved Universal Terrestrial Radio Access (E-UTRA); Physical layer - Measurements 25. [3GPP36.314] TS 36.314; V8.1.0 (2009-03); Evolved Universal Terrestrial Radio Access (E-UTRA); Layer 2 - Measurements 26. [3GPP36.331] TS 36.331; V8.6.0 (2009-06); Evolved Universal Terrestrial Radio Access (E-UTRA); Radio Resource Control (RRC) protocol specification 27. [3GPP36.401] TS 36.401; v 8.x.x; Evolved Universal Terrestrial Radio Access Network (E-UTRAN); Architecture description 28. [3GPP36.413] TS 36.413; V8.6.1 (2009-06); Evolved Universal Terrestrial Access Network (E-UTRAN) ; S1 Application Protocol (S1 AP) 29. [3GPP36.423] TS 36.423; v 8.x.x; Evolved Universal Terrestrial Radio Access Network (E-UTRAN); X2 Application Protocol (X2AP) 30. [3GPP36.521-1] 3GPP TS 36.521-1 V8.2.1 (2009-06); Evolved Universal Terrestrial Radio Access (E-UTRA) User Equipment (UE) conformance specification, Radio transmission and reception, Part 1: Conformance Testing 31. [3GPP36.942] TS 36.942; V8.2.0 (2009-05); Evolved Universal Terrestrial Radio Access (E-UTRA); Radio Frequency (RF) system scenarios 32. [RFC792] IETF RFC 792: Internet Control Message Protocol 33. [RFC979] IETF RCF 979: File Transfer Protocol 34. [RFC2679] IETF RCF 2679: A One-way Delay Metric for IPPM, September 1999 35. [RFC2680] IETF RCF 2680: A One-way Packet Loss Metric for IPPM, September 1999 36. [RFC2681] IETF RCF 2681, A Round-trip Delay Metric for IPPM, September 1999 37. [RFC3261] IETF RCF 3261, SIP - Session Initiation Protocol, June 2002 38. [RFC4960] IETF RFC 4960, Stream Control Transmission Protocol 39. [ETSI102.250-2] ETSI TS 102 250-2 V1.3.1 (2005-07); Speech Processing, Transmission and Quality Aspects (STQ); Part 2: Definition of Quality of Service parameters and their computation 40. [ITUE500] ITU-T Recommendation E.500, Traffic intensity measurement principles, 11/98 41. [ITUE800] ITU-T Recommendation E.800, Definitions of terms related to quality of service, 09/2008 42. [ITUG107] ITU-T Recommendation G.107 (03/2003), The E-model, a computational model for use in transmission planning

DN0972412 Issue 01 DRAFT APPROVED

Id:0900d805807a32b8

79