Beruflich Dokumente
Kultur Dokumente
Contents
1
Introduction
2.1
KPI Reporting.................................................................................................................... 1
2.2
2.3
2.4
KPIs by Service
3.1
3.2
3.3
3.1.2
3.2.2
3.4
3.6
3.5
4.1
Email ................................................................................................................................. 6
4.2
Iperf................................................................................................................................... 7
4.3
4.4
4.2.1
TCP ............................................................................................................................... 7
4.2.2
UDP .............................................................................................................................. 7
4.3.2
4.3.3
CS Fallback..................................................................................................................... 13
Introduction
This document describes the implementation of ETSI circuit-switched and packetswitched service KPIs, as well as some closely related non-ETSI KPIs, in TEMS
products.
The document is valid for the following TEMS product versions:
2.1
KPI Reporting
For PS data services, current TEMS products report KPI events, each of which
contains either
These in turn underlie the aggregation of KPI statistics in TEMS Discovery or some
other post-processing tool. The probability KPIs obviously need to be based on a fair
number of sessions for a good estimate of the probability in question to be obtained.
The data-collecting TEMS products do not in themselves output values of these latter
KPIs; rather, they need to be calculated during post-processing.
TEMS Symphony users should note that earlier versions of TEMS Symphony (version
6.x and older) did not report data in this fashion, but only trigger points on which to
base the KPI computation: for example, session start and end times.
Regarding CS voice, see section 3.6.
2.2
ETSI Compliance
In TEMS products, timestamps for KPI computation are captured as close to the source
as possible: that is, from the packet capture driver within the operating system kernel.
Calculations adhere to the specification ETSI TS 102 250-2 V2.2.1 (2011-04) unless
otherwise stated. This specification is hereafter referred to as ETSI 102 250-2.
2.3
The standardized KPIs have been defined with complete transactions in mind.
Transactions that are aborted after a fixed duration specified by the TEMS product user
(timed measurements) have not been considered. In case of timed measurements, a
best-effort solution is provided. Where triggers do not fully match ETSI 102 250-2, they
have been aligned as closely as possible to their ETSI counterparts in a manner
NT13-25359, ver 2.0, 9/25/2013
compliant with ETSI TS 102 678 V1.2.1, except that the following item stated in
section 4.1 of that specification has not been implemented:
The connection should be checked to be still alive at the end of the transfer period td. The
reception of any data packet sent by the server on the data connection after the end of the
transfer period is a valid indicator that the data connection is still alive. An appropriate
timeout of e.g. 3 to 5 multiples of typical RTT in the measured network might be used while
waiting for the desired packet.
Session Time KPIs, of course, make sense only for sessions that have been allowed
to run to completion.
For details, see the various subsections of chapter 3.
2.4
IP Capture Options
The default IP data capture that is done by TEMS products during PS data service
testing is designed to obtain sufficient input for KPIs. No special activity needs to be
used in Service Control scripts for this purpose.
It is however also possible to conduct a more thoroughgoing IP sniffing by means of
the Start IP Sniffing script activity. This activity has a Filter parameter whose
possible settings are given here for convenience.
Filter: Type of filtering to apply to IP packets.
o
None: No filtering of IP packets. Note: This means that all packets belonging to any
service used by the PC will be captured; packets are filtered with respect to IP address
only. Be aware that full packet capturing can cause crashes on high speed networks
where sustained high throughput is achieved.
KPIs by Service
This chapter deals with TEMS product KPIs for each service.
3.1
Sect.
Service Non-Accessibility
6.1.1
Setup Time
6.1.2
6.1.3
6.1.4
Session Time
6.1.6
Sect.
6.1.7
6.1.8
3.1.1
Timed Measurements
For timed measurements, the end trigger is defined as Last incoming packet
containing content. This trigger is valid for both completed and aborted transfers.
Since this definition conforms to the ETSI standard, timed FTP downloads will have
ETSI compliant KPIs.
3.1.2
If SFTP is used for the file transfer rather than FTP proper, no KPIs are obtained. ETSI
102 250-2 does not define any KPIs for SFTP, and some metrics would not be possible
to obtain owing to the SSH encryption.
3.2
Sect.
Service Non-Accessibility
6.1.1
Setup Time
6.1.2
6.1.3
6.1.4
Session Time
6.1.6
6.1.7
6.1.8
3.2.1
Timed Measurements
For timed FTP uploads, the end trigger is defined as Reception of ACK for last packet
containing content. This trigger is not ETSI compliant: ETSI 102 250-2
(section 6.1.6.3) defines the trigger as Stop: Reception of the [FIN, ACK] for the last
data packet containing content. That is, the FIN indicator has been removed from the
condition. This is because the FIN indicator is sent during connection teardown, a
procedure which does not necessarily take place during a timed measurement.
3.2.2
3.3
This section deals primarily with HTTP downloads conducted with the TEMS browser.
The same KPI values apply also for Internet Explorer browser usage.
ETSI-to-TEMS Mapping Table
ETSI KPI HTTP
Sect.
Service Non-Accessibility
6.8.1
Setup Time
6.8.2
6.8.3
6.8.4
Session Time
6.8.6
6.8.7
6.8.8
3.3.1
Timed Measurements
There is no difference between timed and complete measurements; the same triggers
can be used in both scenarios.
3.4
ETSI does not define KPIs for HTTP Post/Upload. The TEMS product KPIs have been
designed to closely resemble the ETSI KPIs for other services, such as HTTP Get and
FTP (compare sections 3.13.3).
ETSI-to-TEMS Mapping Table
ETSI KPI HTTP (Get)
Sect.
Service Non-Accessibility
6.8.1
Setup Time
6.8.2
6.8.3
6.8.4
Session Time
6.8.6
6.8.7
Sect.
6.8.8
3.4.1
Timed Measurements
Since ETSI does not define any KPIs for HTTP Post, the session end trigger was
defined in such a way as to work for both timed and complete measurements.
3.5
Sect.
6.3.1
The round-trip time is that calculated by the Windows API function IcmpSendEcho2Ex.
This is the lowest-latency way available in Windows to send Ping requests and capture
replies. Reference:
msdn.microsoft.com/en-us/library/windows/desktop/aa366050(v=vs.85).aspx
The difference between the round-trip time measured by Windows and that derived
from the network trace is in the range 0.5 ms. Since the round-trip time is reported in
milliseconds in the KPI, the difference after rounding is not noticeable.
TEMS products also define a timeout error message Ping Timeout signifying a failed
Ping request.
3.5.1
Timed Measurements
Not applicable.
3.6
Service Non-Accessibility
Sect.
6.6.1
Sect.
Setup Time
6.6.2
6.6.4
6.6.5
For CS voice, no special KPI events are generated. However, call setup time is carried
as extra information by the Call Setup event, and speech quality scores are reported in
information elements belonging to the Media Quality category. Supported speech
quality measures are PESQ, POLQA NB and POLQA SWB.
TEMS products also have an additional, non-ETSI speech quality measure called
Speech Quality Index (SQI).
CS voice KPIs are computed for mobile-originated (MO) as well as mobile-terminated
(MT) calls.
For the following services, TEMS products do not have any KPIs defined. However,
they do exhibit certain information elements and/or events with similar content.
Average throughputs and all percentages apply to the period following the latest
network connect (triggering of Network Connect event) and are reset at network
disconnect (when the Network Disconnect activity has completed).
4.1
Email
IE Name
Range/Unit
Description
Email Receive
Average
Throughput
0 ... 350000
kbit/s
Email Receive
Transfer Time
0 ... 172800
s
(= 48 h)
Email Send
Average
Throughput
0 ... 350000
kbit/s
Email Send
Transfer Time
0 ... 172800
s
4.2
Iperf
Iperf testing is conducted with the Network Bandwidth activity in Service Control
scripts. The testing can be done over either TCP or UDP.
4.2.1
TCP
IE Name
Range/Unit
Description
TCP Download
Average
Throughput
(kbit/s)
0 ... 350000
kbit/s
TCP Download
Transfer Time
0 ... 172800
s
0 ... 100
%
TCP Upload
Average
Throughput
(kbit/s)
0 ... 350000
kbit/s
TCP Upload
Transfer Time
0 ... 172800
s
4.2.2
UDP
IE Name
Range/Unit
Description
UDP Download
Average
Throughput
(kbit/s)
0 ... 350000
kbit/s
UDP Download
Jitter
Text
UDP Download
Packet Loss
0 ... 100
%
IE Name
Range/Unit
Description
UDP Download
Transfer Time
0 ... 172800
s
UDP Upload
Average
Throughput
(kbit/s)
0 ... 350000
kbit/s
UDP Upload
Transfer Time
0 ... 172800
s
4.3
4.3.1
These information elements relate to the task of downloading the streaming player.
Please note that not all of the Streaming Player elements are obtained if the
streaming server is an HTTPS server, since the relevant packets are then encrypted.
IE Name
Range/Unit
Description
Streaming Player
Service IP Access
Failure Ratio
0 ... 100
%
Streaming Player
Service IP Access
Time
0 ... 172800
s
Streaming Player
Download Data
Transfer Failure
Ratio
0 ... 100
%
Streaming Player
Download Data
Transfer Time
0 ... 172800
s
Streaming Player
Session Failure
Ratio
0 ... 100
%
Streaming Player
Session Time
0 ... 172800
s
4.3.2
Range/Unit
Description
Streaming Service
Access Time
0 ... 172800
s
Streaming Service
Non Accessibility
0 ... 100
%
Streaming
Reproduction
Start Delay
0 ... 100
%
Streaming
Reproduction
Start Failure
Text
Streaming Video
Play Start Failure
Ratio
0 ... 100
%
Streaming Video
Play Start Time
0 ... 172800
s
Streaming
Reproduction Cut
off Ratio
0 ... 100
%
4.3.3
Overall
IE Name
Range/Unit
Streaming Video
Session Failure
Ratio
0 ... 100
%
Streaming Video
Session Time
0 ... 172800
s
Description
Percentage of streaming video replay
session that failed at some point before
the last video packet was received.
Total time of streaming video replay
session from user request to receipt of last
video packet.
Range/Unit
Description
IE Name
Range/Unit
Description
Streaming
Absolute MOS-V
0 ... 5
MOS
Streaming
MOS-A
0 ... 5
MOS
Streaming
MOS-AV
0 ... 5
MOS
Streaming
Relative MOS-V
0 ... 5
MOS
These average scores are also reported in a Streaming Quality MOS event at the end
of a streaming session.
Bandwidth
IE Name
Range/Unit
Description
Streaming
Average Audio
Received
Bandwidth
0 ... 350000
kbit/s
Streaming
Average Video
Received
Bandwidth
0 ... 350000
kbit/s
Range/Unit
Description
Streaming
Average Audio
Effective Packet
Loss Rate
0 ... 100
%
Streaming
Average Video
Effective Packet
Loss Rate
0 ... 100
%
10
IE Name
Range/Unit
Description
Streaming Audio
Packet Corrected
Rate
0 ... 100
%
Streaming Audio
Packet Discarded
Rate
0 ... 100
%
Streaming Audio
Packet Out Of
Sequence Rate
0 ... 100
%
Streaming Video
Packet Corrected
Rate
0 ... 100
%
Streaming Video
Packet Discarded
Rate
0 ... 100
%
Streaming Video
Packet Out Of
Sequence Rate
0 ... 100
%
Range/Unit
Streaming Video
Interruption Count
Text
Description
Number of times the video stream replay
was interrupted for rebuffering.
Streaming Video
Interruption
Duration
0 ... 172800
s
Streaming Playout
Buffer Delta Rate
0 ... 350000
kbit/s
Streaming Playout
Buffer Empty
Proportion
0 ... 100
%
Streaming Playout
Buffer Empty Rate
0 ... 350000
kbit/s
Streaming Playout
Buffer Fill Rate
0 ... 350000
kbit/s
11
IE Name
Range/Unit
Streaming Playout
Buffer Rebuffering
Proportion
0 ... 100
%
Description
Percentage of the total session time that
was spent rebuffering.
12
4.4
CS Fallback
Description
A CS fallback call was blocked. This can happen in
several ways:
Extra information:
Block type
EUTRAN Reselection
Time After CSFB Call
13
Event Name
PS Data Interruption
Time Due To CSFB
Description
This event reports the IP interruption time during RAT
change to UTRAN/GERAN due to initiation of a CS
fallback call.
Extra information: Interruption time in ms. Measured
from last received IP packet in EUTRAN to first
received IP packet in UTRAN/GERAN.
14