A Measurement Study of Cache Rejection in P2P Live Streaming System
Yishuai Chen, Changjia Chen, Chunxi Li
School of Electrical and Information Engineering, Beijing Jiaotong University
chenyishuai@gmail.com, changjiachen@sina.com, cxl@telecom.njtu.edu.cn
Abstract
In P2P system, each peer caches the received data
for accessing of other peers. P2P live streaming
system audience’s TV-like watching pattern brings
unique characteristics for its cache management
algorithm. By measuring and studying the cache
rejection algorithm used in PPLive with our PPLive
crawler, we found a simple but interesting cache
rejection algorithm used in PPLive, i.e. the cache
rejection of peers is synchronized with the chunk
upload of media server on chunk offset. We modeled
this algorithm as a fixed-duration virtual buffer which
includes the P2P network and peer’s local buffer and
then verified it in our trace. We finally gave some
analysis and explanations for this algorithm. What we
found is benefit to the future design and modeling of
P2P live streaming system .1
1. Introduction
P2P live streaming system is showing its potential
to change people’s way of watching TV. For example,
PPLive [1], a popular P2P online video broadcasting
and advertising network, has had 75 million global
installed base, 20 million monthly active users and
more than 600 channels [2].
The advantage of P2P technology in live streaming
system is impressing. It is reported that PPLive
supported 1,480,000 audiences viewing a live play at
the same time with 1 PC server and 10M bandwidth in
Q2, 2007 [2].
The essence of P2P technology is peer’s caching of
received content and sharing it with other peers. Then
other peers can obtain higher download speed and the
total system becomes more scalable.
In some P2P applications, e.g. P2P file sharing
system, all received data are stored in disk and cache is
This work was supported by China 973 2007CB307101, China
NSFC 60672069 and China NSFC 60772043
not required. However, in P2P live streaming system,
people usually watch program online and don’t expect
local disk is affected. In this case, cache is required
and the corresponding cache management algorithm
becomes an important factor to affect the performance
of total system. In this paper, we focus on the cache
rejection algorithm.
In P2P live streaming system, as all audience watch
in same forward pace, the oldest piece of content
usually is rejected firstly. Because the oldest piece of
content usually is at the head of cache, this cache
rejection behavior can be looked as the forward of the
cache head. We measured the cache rejection behavior
of PPLive by actively crawling in PPLive’s network.
We tracked media server’s chunk upload and peers’
cache rejection. We found all peers reject a chunk with
same rejection rate and this rate is equal to the media
server’s upload rate of this chunk. This is an
interesting finding because it means the cache rejection
in peers is synchronized with the chunk upload of
server. We then modeled this behavior and found it
meant the latency between peer’s rejection of a chunk
and media server’s upload of this chunk always keeps
constant, even if the media rate changes. We then
verified this characteristic in our trace. Finally, we
analyzed the possible reasons and advantages of this
algorithm from the view of design of P2P live
streaming system.
This paper is organized as follows. In Section 2 we
introduce related work. In Section 3, we briefly
analyze the characteristic of caching in P2P live
streaming system and then give the peer buffer
structure and the data can be obtained by measurement
in PPLive. In Section 4, we present the detail of our
measurement and the result. Section 5 is the model and
its verification in trace. In Section 6, we discuss the
possible meanings of this cache rejection algorithm for
P2P live streaming system. Section 7 concludes the
paper.
2. Related work
 Peer buffer size is a frequently mentioned system
parameter in design of P2P live streaming systems. In
Coolstreaming [3], 120s is used as all peers’ buffer
size because experiment results show that the time lag
between nodes is unlikely higher than 1 minute. In
Anysee [4], different buffer sizes are used for peers in
different layers, e.g. 20ms for peers in 1st layer, 41.6ms
for peers in 2nd layer, etc. [5] usually uses 1 minute as
buffer size. In [6], it is found that buffering can
dramatically improve performance of P2P streaming
system and different buffer sizes, e.g. 0, 30, 60 and
120 seconds, are simulated and evaluated. It is
observed that a 30-second lag is sufficient to obtain
almost all potential gains from buffering. All these
work focus on the size of peer’s local buffer. In this
paper, what we discussed and modeled is a fixed-
duration virtual buffer which includes the P2P network
and peers’ local buffer.
The traffic characteristics of controlled peers of
PPLive were measured in Oct 2005 and Dec 2005 [7].
The high rate of change of parents was found at that
time. The problem of remotely monitoring network-
wide quality was examined in [8] by analyzing the
crawled peer buffer maps. The overlay-based
characteristics were studied in [9] by analyzing the
crawled peer partner lists. In [10], it was observed that
buffer size in PPLive varied from 7.8 MBytes to
17.1Mbytes by downloading media file from its local
streaming server after physically disconnecting the PC
from network. The authors inferred that PPLive
adaptively allocated buffer size according to the
streaming rate and the buffering time period specified
by the media source but did not explore the buffer
management algorithm. This paper provides the
answer.
3. PPLive peer buffer structure
In P2P live streaming system, people usually watch
as they watch TV. There are two characteristics for
people to watch TV: (1) The content before audiences’
selected start point is not cared by audiences. (2) All
audiences in one channel watch the same scene at the
same time and also keep same forwarding pace.
Therefore, the played content becomes out-of-date
immediately and is not required by any audience any
more. We call such a watching pattern as instant
watching pattern. Such a pattern is also a requirement
of P2P live streaming system and should be satisfied as
much as possible. That is to say, the content in P2P
live streaming system only has limited lifetime and can
be rejected from peer’s cache safely after some time.
The media data in P2P live streaming system is
usually organized, transmitted and cached in unit
called chunk. Trunks are uploaded to network by a
media server one by one, each with a sequence number.
Peer uses this sequence number to assemble the
received trunks and then send them to the local media
player for playback. The sequence number is usually
incremented continuously and can be looked as the
trunk’s offset to the start point of program, so it is
usually called offset of the chunk. The chunk with
smaller offset has earlier playback time than the chunk
with bigger offset has.
The peer’s local buffer in P2P live streaming
system has two functionalities. One is buffer, i.e. it
holds chunks for reordering and assembling. The other
is cache, i.e. it caches data for accessing of partner
peers. For simplicity, we just call it peer buffer.
Buffer window
Cache part Buffer/Cache part
1111111111111…………1111111111 00101101010001
trunk
rejected
Media
Assembled new
buffer Ro buffer
head Point trunk end
arrived
Figure 1. Structure of peer‘s local buffer
The structure of peer’s local buffer is shown in
figure 1. In this figure, the Media Assembled Point
(MAP) points to the last chunk that has been sent to
media player. Peer’s local buffer is separated by MAP
into two parts. One is the cache part. It includes the
space allocated for chunks with offset smaller than or
equal to MAP. The chunks in this part have been
assembled and sent to media player. Therefore, they
are only cached for accessing of other peers. The other
is the buffer/cache part. It includes the space allocated
for chunks with offset bigger than MAP. Chunks in
this part are waiting for assembling so they are
buffered. On the other side, they can also be accessed
by other peers, so they are also cached.
The ‘1’ and ‘0’ in Figure 1 indicate if the chunk has
been received by peers. Since chunks in cache part
have been received and sent to media player, they are
always marked with ‘1’. On the other hand, the
received chunks in buffer/cache part are buffered
because they are waiting for earlier chunks to finish
assembling, so only the received chunks are marked
with ‘1’. Such a ‘0/1’ bit pattern is usually called
buffer bitmap.
When new trunk is found and peer prepares to
request it, buffer space is allocated for it and the
corresponding bit in buffer bitmap is initialized to ‘0’.
The buffer end then moves forwards if required. When
a new trunk is received, the corresponding ‘0’ becomes 4.2. Basic observation
‘1’. When continuous ‘1’ block beginning from MAP
appears and can be assembled, chunks are assembled We firstly compared the basic trend of Rs and Ro.
into media block and sent to media player for local To calculate Rs at ti, we used following equation based
playback. MAP move forwards. on our observation in [12], i.e.
As chunk becomes out-of-date, it can be rejected e − hi
from the cache. With Belady's minimum cache R s ,i = i (chunk / s ) (1)
120
rejection principle, i.e. the most efficient caching To obtain Ro, we used N samples {Pj, j ∈
algorithm would be to always discard the information
[ k − N2 , k + N2 − 1 ]} to calculate Pj’s slope on tk. After
that will not be needed for the longest time in the
future, the chunk at the head of the buffer should be We selected N = 12. We also did a running average
rejected because it is the oldest chunk in the buffer and with 24 as window size to smooth the curve.
has the least possibility for future request of other Figure 2 shows the Rs obtained from one tracker
peers. Therefore, it is rejected from the buffer and the and the Ro of a random selected peer.
buffer head move forwards.
When peer receives partner peer’s query for its
available chunks list, it returns its buffer bitmap and
the offset of the chunk at the head of buffer. With this
information, partner peer can schedule its requests for
chunks. Peer can also query the tracker for the status of
media server’s upload buffer. Tracker returns the
information of media server’s buffer window, which
includes the offset of the buffer head and the buffer
end, i.e. all trunks between these two offset are
available.

4. PPLive measurement and result

In this section, we give the detail of our
measurement of caching rejection algorithm in PPLive
and the obtained result.
4.1. Measurement
We executed our measurement and collected data
from 1 May to 2 May 2007 on a PPLive channel. The
duration was about 5 hours. 4 trackers and 376 peers
were queried frequently for media server’s or peers’
buffer status.
To query media server’s buffer status, we randomly
selected one tracker from total 4 trackers and queried it
each 10s. When a response was received, we recorded
a sample point Si = (ti, hi, ei), ti was the local time, hi
and ei were the offsets of the buffer head and the
buffer end. We then used {Si} to obtain media server’s
chunk upload rate, Rs.
On peer side, we collected 376 active peers in the
first 20 minutes of measurement and then periodically
queried each peer for its buffer bitmap information
each 5s. For each received response, we recorded a
sample point Pk = (tk, nk), tk was the local time and nk
was the offset of the buffer head. We then use {Pk} to
obtain each peer’s buffer rejection rate, Ro.
Figure 2. Comparison of Rs and Ro
We had following observations from Figure 2: (1)
Rs changes in the range 7-10chunks/s during the
measurement. (2) When Rs changes, Ro also changes to
match the newest Rs. (3) There is latency between the
change of Ro and that of Rs. The latency seems fixed
for one peer, even if there is Rs change.
The first observation is consistent to the
measurement result of PPLive in [11]. In [11], it was
observed that the video play rate varied in range 400k-
800kbps. With uniformed chunk size used in PPLive,
this result is consistent to our observation. For the 2nd
and 3rd observations, to our knowledge, this is the first
time that they are reported. We explore these two
observations below.
4.3. Rs and Ro synchronization on trunk offset
We found peers change their Ro at the same chunk
where media server changes its Rs, i.e. if media server
changes its upload rate from r1 to r2 when it uploads
chunk with offset ni, then peers also change their
buffer rejection rate from r1 to r2 when they reject
chunk with offset ni. We call this behavior
synchronization of Ro and Rs on trunk offset. Below is
the detail of our measurement.
We firstly compared the Ro rate change points of all
peers and found different peers changed their Ro at
different time but at the same chunk.
To obtain the concise rate change point of Ro, we
did not use the slope method mentioned in section 4.2.
The first reason is tk in Pk is not the accurate remote
partner peer’s time for the corresponding nk but the
local time of our measurement machine, which makes
the obtained slope inaccurate. The second reason is
using N samples to fit the slope for a sample point and
then doing running average to smooth the rate change
curve fuzzes the concise rate change point.
Instead, with the known 5s bitmap request period,
we directly used 5 to divide the values of increased
offset in the sampling intervals to get Ro. In detail, for
adjacent two samples Pk=(tk, nk) and Pk+1 =(tk+1, nk+1),
Ro is obtained for sample Pk by following equation:
n − nk
Ro ,k = k +1 (chunk / s ) (2)
5
A difficulty of using this method is the loss of
sampling point. It is very possible that a query has not
a response because either partner peer does not
response or the request or response is lost in the
network. When responses are not received, the
calculated Ro will be multiplied several times. For
example, when Pk and Pk+2 are received adjacently but
Pk+1 is not received, Ro shall be doubled of the correct
rate because nk+2 - nk = (nk+2 - nk+1) + (nk+1 - nk).
Therefore, we firstly used slope method to get the
possible rate change ranges in the total measurement
duration. As result, 6 ranges were obtained. Then for
each range, we picked up samples peers that covered
this range and did not lose sample points in this range.
As result, we obtained 70, 50, 31, 16, 14 and 9 valid
samples peers for the 6 ranges respectively. It is noted
that the decrease of amount of valid samples peers as
time goes on is in anticipation because peers kept
leaving during the experiment. Then we calculated Ro
using equation (2) and got the rate change curve for all
valid peers on corresponding rate change range.
By comparing the rate change point of different
peers, we found different peers changed Ro nearly at
the same chunk although they did it at different time.
Figure 3 show the rate change point of 4 random
selected peers on the 4th rate change range. Figure (a)
shows they changed Ro at obviously different time and
figure (b) shows they changed Ro nearly at the same
chunk. Such a pattern is applicable to all valid samples
peers on 6 rate change ranges.
We then obtained the statistic characteristic of the
chunk offsets of rate change points. In detail, on each
rate change range D, we did linearly interpolation for
each valid peer to obtain its Ro(n) curve, n∈Z and n∈
D. We then got the offset nm on the interpolated result
whose corresponding Ro,m is nearest to 9 and used it as
the offset of the rate change point of the peer. As result,
for j peers, we obtained {nm,l, l = 1…j}. We then
calculated the standard diversion of {nm,l} and the
maximum difference among different nm,l. The result is
shown in Table 1.
Figure 3. Ro(t) and Ro(n)
Table 1. Distribution of rate change offset
Range 1 2 3 4 5 6
Std. Dev. 4.9 7.2 11.1 4.8 7.1 12.6
Max Diff 13 20 35 10 22 33
Considering 35-50 chunks in one measurement
interval, i.e. 7-10 chunks/s in 5s interval, the standard
diversion and the maximum difference shown in Table
1 are very small. Therefore, we concluded that all
peers changed their buffer rejection rate Ro on the
same chunk offset.
We then used following method to calculate Rs. For
each sample point Si = {ti, hi, ei}, we used equation (1)
to calculate a Rs value for all chunks in the reported
buffer window, i.e. for any chunk j∈[hi,ei], we have
e −h
one Rs,j value for it. Rs , j = i i (chunk / s) . We then
120
averaged all obtained Rs,j values of one chunk to obtain
this chunk’s average Rs. Finally, we did a running
average with 128 as windows size to obtain the Rs
curve. The obtained Rs curve slopped gently and we
used the offset with its rate nearest to 9 as the Rs rate
change offset.
We then compared the obtained Rs rate change
offset with the mean of peers’ Ro rate change offsets on
all rate change ranges. The differences are shown in
Table 2.
To explain the meaning of Table 2, let us use its
first column as an example. The first column is for the
1st rate change range in our measured data. As
mentioned before, we have 70 valid sample peers
covered this rate change range. Therefore, the first
column shows the difference between the average Ro
rate change offset of these 70 peers and the media
server’s Rs rate change offset is only 3 chunks.
Table 2. Difference of Ro rate change offset
and Rs rate change offset
Range 1 2 3 4 5 6
Diff 3 -72 -42 186 56 -126
For other rate change ranges, the differences are
bigger than that of the 1st rate change range, but they
are still acceptable considering the transition slope in
the obtained Rs curves usually covers 1.5K-2K offsets.
Therefore, we concluded that peers’ Ro rate change
offset is equal to media server’s Rs rate change offset.
Finally, we concluded all peers changed their buffer
rejection rate Ro on the same chunk offset where media
server changed its upload rate Rs.
5. Modeling and verification
What’s above finding means? We model it as below.
We model the P2P network and peer’s local buffer
together as a virtual buffer. The end of this virtual
buffer is at the media server and the head is at the head
of peer’s local buffer. When a chunk is uploaded into
the network by the media server, it is injected into the
virtual buffer. The injection rate is Rs. When a chunk is
rejected from the head of peer’s local buffer, it is
rejected from the virtual buffer. The rejection rate is Ro.
The model is shown in Figure 4.
Ro Rs
Virtual Buffer
Figure 4. Virtual buffer consists of P2P
network and peer local buffer
With above model, we abstract the P2P streaming
network and peers’ local buffer as a FIFO buffer.
There are lags among different peers’ buffer heads, but
all peers have same buffer end, i.e. at the media server.
The injection and rejection of this virtual buffer follow
the rule we found in Section 4, i.e. Ro and Rs
synchronized on trunk offset. Below we prove that this
virtual buffer is a fixed-duration buffer.
Denote the duration of media content included in
trunk i as Di. At t=0, chunk k is rejected from the
buffer and chunk j is injected into the buffer. Because
the chunk is injected into the buffer and rejected from
the buffer in sequence, chunks [i,j] are included in the
buffer. So the buffer duration is：
j
Lo = ∑ Di .
i =k
Assume at time t, chunk l is rejected and chunk n is
injected, then the buffer duration at time t can be
written as：
n n l −1
L(t ) = ∑ Di = L0 + ∑ Ds − ∑ Do (3)
i =l s = j +1 o=k
Assume media server uploads chunk with chunk’s
playback rate, i.e. it always uploads 1s’ content in 1s,
n
so ∑ Ds is always equal to t. And because Rs and Ro
s = j +1
always keep synchronization on the chunk offset,
chunk’s rejection rate is also equal to chunk’s playback
rate, i.e. peers also always reject 1s’ content in 1s.
l −1
Therefore, ∑ Do is also always equal to t. Then
o =k
equation (3) becomes：
L(t ) = L0 + t − t = L0 (4)
That is to say, the buffer is a fixed duration buffer.
We then verified this property in our PPLive trace.
We obtained the buffer duration by measuring the
buffer time of chunks. To obtain media server’s upload
time of chunk k, for the media server buffer status
sample returned from tracker j, we did linear
interpolation on the (ti, ei) and used the ti,j with
corresponding ei = k as the upload time. We then used
the mean of {ti,j, j=1..4} as the upload time of chunk k.
We then deducted it from this chunk’s buffer rejection
time tk to obtain the buffer duration. We especially
paid attention to the variance of buffer duration when
chunks’ upload and rejection rate changed. If it is a
fixed duration buffer, the buffer duration shall keep
fixed. We successfully observed this behavior in our
trace. Figure 5 shows the buffer duration of chunks in
three randomly selected peers’ virtual buffers.
Figure 5. Buffer duration of chunks
The upper subfigure in Figure 5 shows chunks’
upload and rejection rate changed from 10chunks/s to
8chunks/s and then changed back again. However, as
shown in the lower subfigure, although different peer
has different buffer duration, the buffer duration of one
peer keeps stable during this time, i.e. the virtual buffer
is a fixed duration buffer.
Then we can prove the observation (3) we made in
Section 4.2, i.e. the latency between Ro and Rs is fixed,
even if there is Rs change.
6. Why this behavior?
What we found is a simple but interesting cache
rejection algorithm used in P2P live streaming system.
It can be looked as an approximate implementation of
the fixed-duration peer local buffer method mentioned
in miscellaneous P2P documents [3-6] but this is not
the whole story. We have following possible
explanations for this algorithm.
The first explanation is the local media player
directly uses the rejected chunks as input, so the
synchronization of buffer rejection with media server’s
chunk upload makes the local media player obtain
stable media streaming input.
The second explanation is it help peer’s buffer
rejection synchronize with the local media playback.
Then peer can infer the status of local media playback
from the status of its local buffer. For example, when
the head of buffer is approaching a chunk position but
this chunk is still not received, peer can infer its media
player is also approaching this chunk position and shall
have to stop because of the unavailability of this chunk.
Then peer can start the aggressive download procedure
as observed in [10].
The third explanation is it may improve the
performance of P2P sharing. This algorithm makes the
cache rejection of peers is as stable as media server’s
chunk upload, rather than random or burst-like,
therefore, it brings some stability to the P2P sharing,
which may help peer’s children peers keep stable
connection with it and then improve the stability of
total system.
The final explanation is it adaptively adjusts buffer
size according to the streaming rate, which is inferred
but not explored in [10], i.e. when chunk rate increases,
because the duration of content included in each chunk
decreases, buffer size is increased accordingly,
Conversely, when rate decreases, the buffer size is
decreased to include lesser chunks. Such an adaptive
adjustment of chunk size helps save the system
resource used for buffer.
7. Conclusion
We found the cache rejection in PPLive peers is
synchronized with media server’s chunk upload on
chunk offset, which makes a fixed-duration virtual
buffer for each peer. We analyzed the possible
explanations for this algorithm from the view of design
of P2P live streaming system. What we found also
inspires us to observe and model P2P streaming
network in a new way, i.e. look it as a set of fixed-
duration virtual buffers and look its working
mechanism as the communication among these virtual
buffers to finish the propagation of media streaming.
We are exploring towards this way.
8. References
[1] www.pplive.com
[2] G. Huang “Experiences with PPLive”, [Online document],
P2P-TV Workshop in conjunction with SIGCOMM 2007,
Available HTTP: http://www.sigcomm.org/sigcomm2007
/p2p-tv/Keynote-P2PTV-GALE_PPLIVE.ppt,
[3] X. Zhang, J. Liu, B. Li, and T.-S. P. Yum, “DONet/
CoolStreaming: A Data-driven Overlay Network for Peer-to-
Peer Live Media Streaming,” IEEE INFOCOM’05, 2005.
[4] X. Liao, H. Jin, Y. Liu, L. M. Ni, and D. Deng, “Anysee:
Scalable Live Streaming”, IEEE INFOCOM’06, 2006.
[5] M. Zhang, Q. Zhang, L. Sun and S. Yang "Understanding
the Power of Pull-based Streaming Protocol: Can We Do
Better?" P2P-TV Workshop in conjunction with SIGCOMM
2007
[6] R. Kumar, Y. Liu, and K. W. Ross, "Stochastic fluid
theory for P2P streaming systems," in Proceedings of
INFOCOM, 2007
[7] S. Ali, A. Mathur and H. Zhang "Measurement of
Commercial Peer-To-Peer Live Video Streaming",
Workshop in Recent Advances in Peer-to-Peer Streaming,
August, 2006.
[8] X. Hei, Y. Liu, and K. W. Ross, “Inferring Network-
Wide Quality in P2P Live Streaming Systems”, [Online
document], 2007, Available HTTP: http://eeweb.poly.edu/
faculty/yongliu/docs/index-buffermap.pdf,
[9] L. Vu, I. Gupta, J. Liang, K. Nahrstedt. "Measurement of
a Large-scale Overlay for Multimedia Streaming", High
Performance Distributed Computing (Poster - HPDC 2007)
[10] X. Hei, C. Liang, J. Liang, Y. Liu and K. W. Ross,
“Insights into PPLive: A Measurement Study of a Large
Scale P2P IPTV System”, IPTV workshop in conjunction
with WWW2006, May 2006.
[11] X. Hei, C. Liang, J. Liang, Y. Liu and K. W. Ross, “A
Measurement Study of a Large Scale P2P IPTV System”,
[Online document], Nov 2006, Available HTTP: http://cis.
poly.edu/~ross/papers/ P2PliveStreamingMeasurement.pdf
[12] C. Li and C. Chen, “Inferring Playback Rate and Rate
Resets on PPLive By Piecewise Line Envelop
Approximation”, submitted to TOB, IEEE, Spt, 2007