Domenico Papaleo, Stefano Salsano

The Linux Traffic Generator

INFOCOM Department Report 003-004-1999 University of Rome La Sapienza

ABSTRACT
This report deals with the specification and realization of a synthetic traffic generator for IP traffic, called LTG Linux Traffic Generator. The LTG can generate multiple independent UDP flows with given traffic profiles (i.e. CBR or VBR), with millisecond resolution. The LTG works on a common PC with Linux Operating System. It is possible to evaluate a set of performance metrics related to throughput, loss and delay. The LTG is a key component for the experiments on IP QoS in the context of MURST Project Tecniche per la garanzia di qualit in reti di telecomunicazioni multiservizi. In fact, the LTG is used to evaluate the performance of IP traffic control mechanisms supporting QoS.

1. Introduction ......................................................................................................................... 2 2. Metric definition................................................................................................................... 2 3. Generation of traffic flows ................................................................................................... 5 4. Performance metric collection ............................................................................................. 8 5. LTG: List of command line options ................................................................................... 11 6. References.......................................................................................................................... 12

D. Papaleo, S. Salsano: The Linux Traffic Gnerator

1. Introduction
In order to evaluate the performance of the proposed QoS mechanisms for IP networks (considering either the IntServ or the DiffServ model) one needs to develop powerful and flexible tools for traffic generation and performance monitoring. The requirements for such tools are: to operate on a set of independent flow in parallel to control packet generation with a granularity in the order of few milliseconds, for data rates up to 100Mb/s The LTG (Linux Traffic Generator) tool derives from the TTCP tool [1]. The LTG allows to generate multiple independent flows of UDP traffic. For each flow one can define the pattern of packet emission with millisecond resolution. The LTG evaluates the average throughput and can log into a file the instantaneous throughput. By use LTG in combination with tcpdump [4] and using additional post-processing tools, it is also possible to evaluate the variable part of the end-to-end delay and the istantaneous packet delay variation (ipdv) [5].

2. Metric definition
The IP Performance Metric (IPPM) working group of the IETF [1] is defining a set of standard metrics to measure the Internet performance. These metrics aims at evaluating the quality, performance, and reliability of Internet data delivery services, so they should provide generic unbiased measures of the performance of the network. For example the one-way or round-trip delay between two end-points, or the loss across an internet path are considered. The goal is to define metrics which refer to the network and are not related to the different single flows. For example the proposed methodology to measure the one-way delay is based on the poisson sampling: a set of packets are sent as a probe, with random (poisson) interdeparture times and their delay is evaluated. While our work is related to the IPPM activities, we follow a slightly different approach. We want to evaluate the performances (throughput, loss, delay) specifically experienced by one or more test flows. This is needed to evaluate how the traffic control mechanisms affect the quality perceived by the different flows. Therefore we will generate a syntetic flow with given traffic characteristic and we will measure the performace perceived by the flow itself. Only UDP flows are being considered, as the target applications are real time ones.

D. Papaleo, S. Salsano: The Linux Traffic Gnerator

Two kinds of metrics have been considered, respectively related to troughput and to delay. The metrics related to throughput are: average throughput, loss, instantaneous throughput, while the metrics related to delay are: variable part of the delay, ipdv. 2.1 Throughput related metrics The definition of the metrics related to throughput is straightforward. Let Pt(j) and Pr(j) respectively be the volume of payload data (bit) sent by the traffic source and received by the sink related to flow j. The transmission time is Tt(j) and the reception time is Tr(j). The transmitted and received average throughput at the UDP payload level are: tUDPpayload ( j ) = Pr ( j ) Tr ( j )

[bit s]

rUDPpayload ( j ) =

Pr ( j ) Tr ( j )

[bit s ]

Taking into account the length lj of the UDP payload and the IP and UDP header, the average throughput at the IP level becomes: tIP ( j ) = Pt ( j ) l j + 28 Tt ( j ) l j Pr ( j ) l j + 28 Tr ( j ) l j

[bit s]

rIP ( j ) =

[bit s ]

The loss percentage is: L( j ) = Pt ( j ) Pr ( j ) 100% Pt ( j )

For a given test the LTG on the sending side evaluates the transmitted bytes and the transmitting time for each flow. The same happens on the receiving side with the transmitted bytes and the receiving time. Hence the loss and average throughput can be evaluated. The istantaneous throughput is defined as the received throughput for each flow averaged over a small time interval (in the order of tens of milliseconds). If Pr ( j ) t 2 and Pr ( j ) t + 2 represents the volume of received payload data (bit) at time t 2 and t + 2 , we define the received instantaneous throughput ir for UDP payload as: irUDPpayload ( j ) = Pr ( j ) t + 2 Pr ( j ) l j + 28 lj
t 2

[bit s ]

and the received instantaneous throughput at the IP level as: irIP ( j ) = irUDPpayload ( j )

[bit s ]

D. Papaleo, S. Salsano: The Linux Traffic Gnerator

As an option, the LTG on the receiving side can record on a log file the instantaneous throughput. The duration of the averaging interval can be defined as a command line option. 2.2 Delay related metrics In order to define and evaluate the metrics related to packet transfer delay, one should take properly into account the problem of clock synchronization. For example the one-way delay metric defined by the IPPM WG [2] assumes that the clocks on the transmitting and receiving side are synchronous and provides the corrections to take into account the synchronization errors which appear in the real measurements. In order to achieve good result, hardware devices to synchronize the clocks are needed (typically based on GPS receivers). On the contrary, we assume that clocks on the transmitting and receiving side are not synchronous and we consider two metrics that can be evaluated under this hypothesis. These metrics are the variable part of the delay and the instantaneous packet delay variation. Let tsk be the departure time of k-th packet of a flow (time read on the source clock) and trk the arrival time of the packet on the receiver host (time read on the receiver clock). The measured one-way delay D'k includes the effective one-way delay Dk and the error terms Ck which takes into account clock syncronization and Ek which accounts for all other errors: D 'k = trk tik = Dk + Ck + Ek A detailed discussion on these error terms is provided in [2]. As we even do not try to synchronize the clocks, the synchronization error makes the D' k completely meaningless. But if we want to consider the variable part of the delay we can subtract the minimum observed delay. Let Dm = min D j = D'm +Cm + Em , then we define the variable part of the delay:
j

VPDk = D 'k D 'm = Dk Dm + (Ck Cm ) + ( Ek Em ) If we assume that the synchronization error is constant during the experiment Ck = Cm , we have that the variable part of the delay is not sensible to this error. According to [5], the instantaneous packet delay variation for a couple of packets (k+1, k) is defined as the difference between the one way delay of packet k+1 and packet k. Also in this case we can use the measured one way delay, assuming that the synchronization error is constant: IPDVk = D'k +1 D 'k = Dk +1 Dk + (Ck +1 Ck ) + ( Ek +1 Ek )

D. Papaleo, S. Salsano: The Linux Traffic Gnerator

As we will describe later, in order to evaluate delay related metrics we need an additional tool along with LTG. The LTG on the sender side will mark packets with a sequence number in the UDP payload. The tcpdump tool will be used on both the sender and the receiver side to capture packets with their time-stamp onto log files. Then a set of post processing tool will be used to post-process the log files.

3. Generation of traffic flows

3.1 Internal behavior The LTG supports both CBR and VBR flows. There are different mechanisms to specify the traffic pattern of a source. The basic principle is that a tick of 1024 Hz regulates the packet emission. For each tick the LTG can schedule a number of packets for each flow. The size of the packet is fixed per each flow (different flows can have different packet size as specified in the data_len file). A practical example of the generation of multiple CBR flows is depicted in Figure 1. A set of j CBR flows are generated. For each kernel tick bj UDP packet are sent. When the LTG has transmitted the packets for a tick, it asks the operating system to sleep for a very short delay (D) in the order of few micorseconds. The operating system sets a timer and it will check the expiration of the timer only on the next kernel tick. This allow to have a precise control of the emission time of the packets. Obviously there should be no other process in the PC hosting the traffic generator which compete for processing time. Let lj be the length in bytes of the UDP payload for packets of flow j, than the bit rate at the IP level of flow j is given by: IP ( j ) = b j (l j + 28) 8 T

It is possible to notice that the 20 bytes IP header and the 8 bytes UDP header are added to the UDP payload.
T = 1 / 1024 sec T = 1 / 1024 sec

b1 1

b2

bj

b1 1

b2

bj

Figure 1: Packet generation for multiple CBR flows

D. Papaleo, S. Salsano: The Linux Traffic Gnerator

In order to easily generate lower bit-rate CBR flows one can use a different mechanisms where a packet of a given flow j is generated each Nj kernel ticks, as depicted in Figure 2. In this case the the IP level bit rate at the of flow j is given by: IP ( j ) =

(l

+ 28) 8 N jT

T = 1 / 1024

T = 1 / 1024

T = 1 / 1024

T = 1 / 1024

1 2

D N1 N2

..

Figure 2: Packet generation for low rate CBR flows The mechanisms described in Figure 1 and Figure 2 are used to generate CBR flow. A more generic mechanism allows to generate VBR, based on the same principle that we can define how many packets to send for each flow in each tick. The simplest mechanism is to read the emission profile (number of packets per tick) of a given flow from an external data file, and then to play out this emission profile periodically. The maximum duration of the period is in the order of 100 seconds. For VBR flows it can happen that the scheduled packets exceed the link capacity in one or more consecutive ticks. In this case the excess packets are logically buffered and reported to the following ticks. During this overflow period the available rate is roughly shared among the active flows with a sort of round robin. It is not the purpose of the LTG to handle this congestion period with sophisticated mechanisms (e.g. Weighted Round Robin). 3.2 Available mechanisms for traffic generation and pratical usage The first available mechanism is to use a file to describe the number of packets per tick (one separate datainxx file for each source xx). The values are read into an array and then played out periodically. The maximum size of the array is 100.000 elements, therefore one can play out a periodic sequence with a period up to 97 seconds for each flow. This mechanism of course can be used to produce CBR and VBR flows. Example 1.1 and 1.2 - receiver side command line: ./ltg5 -r -u z5 l1500 i100

output_file

D. Papaleo, S. Salsano: The Linux Traffic Gnerator

Example 1.1 - sender side command line: ./ltg5 -t -u s5 e10 dest_host (in this case the packet length for each flow is read in the data_len file)

Example 1.2 - sender side command line: ./ltg5 -t -u s5 l1000 e10 dest_host (in this case the packet length for each flow is 1000 bytes UDP payload)

The second mechanism is useful to easily produce a set of CBR flows without providing a separate datain file for each flow. This applies to CBR sources that emit one or more packets (N) each tick. A single file called data_sock specifies the N value for a set of flows. Example 2 - receiver side command line: ./ltg5 -r -u z5 l1500 i100

output_file

Example 2 - sender side command line: ./ltg5 -t -u s5 f l100 e10 dest_host

The third mechanism is useful to easily produce a set of CBR flows without providing a separate datain file for each flow. This applies to CBR sources that emit a packet each F ticks. A single file called data_freq specifies the F value for a set of flows. Example 3 - receiver side command line: ./ltg5 -r -u z5 l1500 i100

output_file

Example 3 - sender side command line: ./ltg5 -t -u v5 l100 e10 dest_host

D. Papaleo, S. Salsano: The Linux Traffic Gnerator

A fourth mechanism is to specify with a command line parameters the number of packets to be sent by all the sources per each tick. This implies that we will have a set of identical CBR sources. Example 4 - receiver side command line: ./ltg5 -r -u z5 l1500 i100

output_file

Example 4 - sender side command line: ./ltg5 -t -u s5 b3 l100 e10

dest_host

The fifth mechanism is to have a statistical generator for example to produce a markovian ON/OFF flow that emulates a voice flow. This mechanism is still under development. To a certain extent, the different mechanisms can coexist in the same measurement

session. The duration of the test can be specified in a couple of ways using command line options on the sending side. The simplest way is to give the number of seconds (-e options). The second possibility is to give the number of packets that must be transmitted by each flow (this is not very flexible because there is a single value common for all the flows).

4. Performance metric collection

At the end of the test the LTG on the sending/receiving side prints the results on the standard output. These results include for each flow the transmitted/received bytes, the transmitting/receiving time and the throughput. Both throughput for UDP payload and IP level throughput are recorded. The outputs are typical redirected to log files. As an option, the LTG on the receiving side can record on a log file the instantaneous throughput only at the UDP level for the moment. An example of the transmit and receive side output is reported hereafter: Transmit side:
ltg-t: port 2000 UDP payload packet len: 1000 ltg-t: 76627000 UDP payload bytes processed ltg-t: UDP stats Kbits/sec ltg-t: IP stats Kbits/sec 20.0008 real sec = 3938.47 KB/real sec, 31507.8 20.0008 real sec = 3831.2 KB/real sec, 30649.6

ltg-t: port 2001

UDP payload packet len: 1000

ltg-t: 36100000 UDP payload bytes processed

D. Papaleo, S. Salsano: The Linux Traffic Gnerator

ltg-t: UDP stats Kbits/sec ltg-t: IP stats Kbits/sec

20.006 real sec =

1804.46 KB/real sec,

14435.7

20.006 real sec =

1854.98 KB/real sec,

14839.9

Receive side:
ltg-r: port 2000 UDP payload packet len: 1000 ltg-r: 76627000 UDP payload bytes processed ltg-r: UDP stats Kbits/sec ltg-r: IP stats Kbits/sec 20.001 real sec = 3938.43 KB/real sec, 31507.4 20.001 real sec = 3831.16 KB/real sec, 30649.3

ltg-r: port 2001

UDP payload packet len: 1000

ltg-r: 36100000 UDP payload bytes processed ltg-r: UDP stats Kbits/sec ltg-r: IP stats Kbits/sec 20.0023 real sec = 1855.32 KB/real sec, 14842.6 20.0023 real sec = 1804.79 KB/real sec, 14438.3

4.1

Delay measures: tools for post-processing of dump files The LTG tool in itself only allows the evaluation of the loss, average and

instantaneous throughput. Additional tools are needed to get information on the delay. The idea is to use tcpdump on the sending and on the receiving side to record the transmitted and received packets with their time stamps. Note that the packets are produced by LTG with an increasing sequence number in the UDP payload. The post processing tools correlate the dump files on the transmitting and receiving side to extract delay information. The lost packets are neglected, and for all the received packets the variable part of the end-to-end delay and the instantaneous packet delay variation are evaluated. The final phase of the post processing calculates the distribution of the variable part of the delay and of the ipdv. Assume that you want to evaluate delay metrics of one flow with destination UDP port 2000. The following steps must be accomplished: 1) Run the tcpdump on tx and rx side, listening on the specific UDP port of the flow that you want to analyze: Transmit side:
#tcpdump x s46 i eth0 'port 2000' w 'dump_tx_2000'

Receive side:
#tcpdump x s46 i eth0 'port 2000' w 'dump_rx_2000'

D. Papaleo, S. Salsano: The Linux Traffic Gnerator

10

(packets on eth0 interfaces with UDP port 2000 will be dumped on the binary files dump_tx_2000 on transmit side and dump_rx_2000 on receive side)

2) Run the ltg first on receiver side and then on transmitter side

3) Process the output files with tcpdump again to produce readable dump files Transmit side:
#tcpdump x s46 i eth0 'port 2000' r 'dump_tx_2000' > asc_dump_tx_2000

Receive side:
#tcpdump x s46 i eth0 'port 2000' r 'dump_rx_2000'> asc_dump_rx_2000

(the readable text dump files asc_dump_tx_2000 on transmit side and asc_dump_rx_2000 on receive side will be created)

4) Copy the asc_dump_tx_2000 and asc_dump_rx_2000 into a common post-processing directory.

5) In the post-processing directory clean the readable dump files removing the unneeded information
#clean asc_dump_tx_2000 ok_source_2000 #clean asc_dump_rx_2000 ok_dest_2000

6) Evaluate the variable part of the delay and then its distribution
#vp_delay ok_dest_2000 ok_source_2000 vp_delay_2000

(the file vp_delay_2000 contains the samples of the variable part of the delay)

#marg_positive 50 vp_delay_2000 distr_vp_delay_2000

(The first parameter represents the size (s) of each interval of the distribution. In the file distr_vp_delay_2000 the first column represents the center of each interval, the second column represents the number of sample in that interval divided by the total number of samples)

D. Papaleo, S. Salsano: The Linux Traffic Gnerator

11

7) Evaluate the ipdv and then its distribution:

#calc_delta ok_dest_2000 ok_delta_dest_2000 #calc_delta ok_source_2000 ok_delta_source_2000 #calc_ipdv ok_delta_dest_2000 ok_delta_source_2000 ipdv_2000

(the file ipdv_2000 contains the samples of the variable part of the delay)

#marg_gener 50 ipdv_2000 distr_ipdv_2000

(The first parameter represents the size (s) of each interval of the distribution. In the file distr_ipdv_2000 the first column represents the center of each interval, the second column represents the number of sample in that interval divided by the total number of samples)

5. LTG: List of command line options

Lato client -t indica il client. -u -p## indica uso di socket UDP, deve essere sempre presente indica la porta UDP di destinazione del server verso cui indirizzare il primo flusso di dati da trasmettere; per n flussi di dati le porte saranno la p, p+1, , p+(n-1); per default, ove non precisata diversamente fissata su entrambi pari alla 2000 -x## sul client indica la porta sorgente UDP da cui trasmettere il primo flusso di dati verso il server; per n flussi di dati le porte saranno la x, x+1, , x+(n-1); per default, ove non precisata diversamente fissata su entrambi pari alla 3000 -n## indica il numero di pacchetti di dati che il client intende trasmettere per ogni flusso; per default fissato a 1024. -e## -s## sul client indica il tempo di trasmissione ed usato in alternativa a n##. indica il numero di socket che si intende aprire per effettuare una trasmissione cbr o vbr; per default fissato a 1. -v## indica il numero di socket che si intendono aprire per effettuare una trasmissione cbr con trasmissione di un pacchetto ogni intervallo temporale letto da file. -l## indica la lunghezza del payload in bytes del singolo pacchetto trasmesso dal client. Se non presente nella riga di comando la lunghezza del singolo pacchetto per ogni flusso letta da un file di nome data_len.

D. Papaleo, S. Salsano: The Linux Traffic Gnerator

12

-b##

sul client indica il numero di pacchetti trasmessi per ogni tick per ogni flusso. utilizzato solo per trasmettere un insieme di flussi CBR tutti uguali. Se non presente nella riga di comando indica trasmissione vbr.

-f

sul client indica che il numero di pacchetti di ogni blocco di dati per ogni flusso letto da un file di nome data_sock (esempio).(tale opzione va utilizzando con il comando b, ossia trasmissione CBR).

Sul client possibile oltre a specificare il nome o lindirizzo ip di un unico server, specificare gli indirizzi ip di diversi server digitando read_file al posto del nome o dellindirizzo IP dellunico server, permettendo cos la trasmissione tra un client e pi server. In questo caso la lettura degli indirizzi avviene dal file data_addr Lato server -r indica il server -u -p## indica uso di socket UDP, deve sempre essere presente indica la porta UDP sulla quale mettersi ad ascoltare i dati in arrivo del primo flusso; per n flussi di dati le porte saranno la p, p+1, , p+(n-1); per default, ove non precisata diversamente fissata su entrambi pari alla 2000. -z## indica il numero di socket che si intendono aprire e deve essere pari al numero complessivo di sockets aperto sul client. -l## corrisponde alla dimensione del buffer che allocato sul server (che viene riempito con i dati del singolo blocco letto); -i## sul server indica in ms lintervallo temporale trascorso il quale lapplicazione provvede a memorizzare in un array il troughput relativo sul link e il tempo in cui effettuata la misura; per default fissato a 100 ms; tali valori sono, alla fine della esecuzione dellapplicazione, scritti su file il cui nome specificato dallutente come ultimo parametro nella riga di comando.

6. References
[1] IETF IP performance Metric (IPPM) Working Group http://www.ietf.org/html.charters/ippmcharter.html [2] RFC 2679 A one-way delay metric for IPPM September 1999 [3] TTCP Test TCP connection T.C. Slattery, USNA, see http://ftp2.sunet.se/ftp/pub/benchmark/ttcp/ [4] V.Jacobson, C.Leres S.McCanne Lawrence Berkeley Laboratory, University of California, Berkeley, CA tcpdump ftp://ftp.ee.lbl.gov/tcpdump.tar.Z

D. Papaleo, S. Salsano: The Linux Traffic Gnerator

13

[5] C. Demichelis, P.Chimento Instantaneous Packet Delay Variation for IPPM draft-ietf-ippmipdv-03.txt